How to Use Exit Handlers with Argo Workflows to Trigger Notifications Part 2
May 25, 2022
12 min read
Learn how to send notifications from Argo Workflows to Slack, PagerDuty, Discord, Twilio, SMS, and email using exit handlers.
Argo Workflows is a tool for designing pipelines that run on Kubernetes. These pipelines are composed of multiple tasks and each task runs in a Kubernetes pod. The defined tasks range from downloading data from a storage bucket to running data analytics (Spark or Dask) workloads.
The previous article addressed the basics of Argo workflows and the various ways you can add exit handlers to a workflow. In this article, you’ll learn how to trigger notifications from exit handlers to tools such as email, Slack, PagerDuty, Twilio, and more.
Triggering Notifications in Argo Workflows with Exit Handlers
Exit handlers are ideal for triggering notifications in Argo Workflows. Before you proceed with this tutorial, you’ll need the following prerequisites:
- A Kubernetes cluster deployed to a cloud environment
- A Slack account
- A PagerDuty account
- A Twilio account
- A Discord account
You’ll be using the same workflow to set up notification handlers for the different tools that will be addressed:
Setting Up Argo Workflows
Argo Workflows is a custom Kubernetes resource, so you need to apply the Argo Workflows Kubernetes manifest. You can set up Argo Workflows in any Kubernetes environment. The easiest setup is a local setup using minikube.
Here are some other ways you can deploy Kubernetes to your local or cloud environment:
You can also automatically provision Argo Workflows using a fully managed service like Pipekit.
Installing Argo Workflows
Argo is a Kubernetes resource that can be applied globally or to a specific namespace. Run the command below in a UNIX shell within your Kubernetes environment (in the shell that has access to Kubernetes). The command creates a namespace and applies the Argo Workflows configurations to it:
Run the command below in the same shell as above to port forward the Argo Workflows user interface HTTP address to your local environment:
This will serve the UI on http://localhost:2746.
Install Argo CLI
You can install the Argo CLI on Linux and macOS machines. Visit the releases page for the latest CLI release for macOS and Linux. This article makes use of version 3.3.5. You can use the bash snippet below to install version 3.3.5 on a Linux machine:
Setting Up Notifications Using Exit Handlers
You’re now ready to set up notifications using exit handlers. You’ll be setting up email, webhook, Slack, PagerDuty, Twilio, and Discord notifications.
Sending Email Notifications from Exit Handlers
You can send email notifications using providers such as SendGrid, Mailgun, and Amazon SES. First you need to get the cURL configuration and add it to your workflow exit handler. This article uses SenGrid, but any other mail sending tool should also work.
Follow the steps below to configure and send email notifications from exit handlers:
- Navigate to app.sendgrid.com to log in or create an account.
- Navigate to the SendGrid integration guide and choose Web API.
- On the Web API integration page, choose cURL as the language you want to use.
- On the cURL integration page, choose a descriptive API key, then copy the environment variable export and the cURL command.
- You’ll add the shell code snippet below to your Argo workflow. Replace <SendGrid_API_KEY> with the API that was generated in the previous step. Also, replace <YOUR_VERIFIED_EMAIL> and <YOUR_VERIFIED_SENDER> with your configured verified email and sender.
YOUR_VERIFIED_EMAIL is the email you set that can receive emails you send. YOUR_VERIFIED_SENDER is the email you configured that can send emails through your SendGrid account. This only applies for a free account. You can also modify the subject and content in the following snippet:
- Create a new dummy workflow in your Kubernetes environment where Argo Workflows is running that sends an email when the run completes. You can name this workflow cowsay-email.yml. Add the following YAML snippet to the workflow file:
- Submit the workflow using argo submit -n argo --watch cowsay-email.yml. An email will be sent to your verified email when the workflow run completes. Run argo -n argo logs @latest to see the logs from the cowsay-email.yml workflow. You should get the following logs on success:
Sending Slack Notifications from Exit Handlers (using a webhook)
Slack provides a webhook endpoint for sending notifications to channels via HTTP POST requests. Here are the steps to send notifications to Slack:
- Create a channel in a workspace where you have channel creation rights. You can create a new workspace if you don’t have channel creation in any existing workspace.
- Navigate to the Slack app API dashboard to create an app. Choose the From scratch option.
- After choosing From scratch, pick a name and select the workspace in which you created the channel in the previous step, then click Create App. You’ll be navigated to the app settings page.
- Your app needs to be configured to receive incoming webhooks in order to receive notifications. On the basic information page, choose the Incoming Webhooks option and turn on incoming webhooks. A section will appear with the button Add New Webhook to Workspace. Click on the button. You’ll be redirected to a page to set the channel that will receive the webhook notifications:
- Choose the channel you created in the first step, then click Allow to allow webhook notifications to be sent to the channel. You’ll be navigated to a page that contains the webhook URL of your Slack app. You’ll use this URL in your workflow, so copy it and keep it handy.
- Create a new file called cowsay-slack.yml in the cluster and namespacemachine where Argo Workflowsyour Kubernetes cluster is running, and add in the content below. Replace <YOUR_SLACK_WEBHOOK_URL> with the webhook URL you copied in the previous step:
- Run cowsay-slack.yml using argo submit -n argo --watch cowsay-slack.yml. This should send a Slack notification to the channel you created.
Sending PagerDuty Notifications from Exit Handlers
PagerDuty is an incident reporting platform that collects signals from different systems (like Argo Workflows), assigns priorities to them, and determines the right people to fix them based on an on-call rotation.
PagerDuty exposes an API that can serve as a webhook endpoint that you can trigger using cURL. Follow the steps below to send notifications from Argo Workflows to PagerDuty:
- Create a PagerDuty account.
- Navigate to Service Directory under the Services tab.
- Click on New Service to create a new service.
- Give your service a name like “Argo Workflows Alerts” and a description. Then click Next.
- Leave the default options in Assign and Reduce Noise, by clicking Next on each page.
- Select Events API V2 from the integrations list on the Integrations page, then click Create Service. You’ll be redirected to a page with your integration key and a webhook URL.
- On the integrations details page, make a note of the Integration Key and Integration URL for Alert Events. You’ll use these details in your workflow template.
- Create a new file called cowsay-pager-duty.yml in the cluster and namespace where Argo Workflows is running, and add in the content below:
Replace <YOUR_INTEGRATION_KEY> with the Integration Key shown on your integration details page.
- Run cowsay-pager-duty.yml using argo submit -n argo --watch cowsay-pager-duty.yml. This should send an incident to PagerDuty.
- Navigate to your incidents page to see the incident that was created from the workflow run.
Sending Twilio Notifications from Exit Handlers
Twilio is a service that provides communications APIs for platforms like SMS, phone calls, WhatsApp, and chatbots. It lets you send notifications through these platforms. Follow the steps below to send SMS via exit handlers:
- Create a Twilio account. You’ll be navigated to an email verification page, and after verifying your email, you’ll land on a phone number verification page.
- After completing your email and phone number verification, you’ll need to set up a messaging service. Head over to the sidebar, then navigate to Messaging > Try it out > Get Set Up to start the setup process. Click on Start setup to get started.
- First choose a messaging service name, then click Create Messaging Service.
- Next, select the phone number from your list of phone numbers. Twilio provides you with a default phone number for sending text messages. Click Add this number to proceed.
- Navigate to Messaging > Try it out > Send an SMS, to get the cURL command for sending SMS.
- Select the messaging service you previously created and enter some body text. Copy the cURL command and keep it handy. The AuthToken in the cURL command can be obtained by navigating to your Twilio console page.
- Create a new file called cowsay-sms.yml in the cluster and namespace where Argo Workflows is running and add in the content below. Replace <YOUR_TWILIO_API_URL>, <YOUR_VERIFIED_PHONE_NUMBER>, <YOUR_MESSAGING_SERVICE_ID>, <YOUR_ACCOUNT_SID>, and <YOUR_AUTH_TOKEN> with your credentials:
- Run cowsay-sms.yml using argo submit -n argo --watch cowsay-sms.yml. This should send an SMS to the phone number you provided.
Sending Discord Notifications from Exit Handlers
You can send notifications to Discord the same way you can send notifications to Slack. Discord has fewer steps to send notifications:
- Create a new Discord channel in a Discord server where you have channel-creation permission. You can find the Create Channel button when you click on the server name located at the top left corner of the sidebar.
- Select Text as the channel type and input a descriptive channel name.
- After the channel has been created, click Edit Channel. You’ll be navigated to the channel settings page.
- In the channel settings page, go to Integrations and then click Create Webhook.
- On the Webhooks page, choose a descriptive name for your webhook like “Argo Workflow Bot”, click Copy Webhook URL to copy the webhook URL, and finally, click Save Changes at the bottom. Note that the Save Changes button might not appear if you’re on the Discord web app (where saving happens automatically).
- Now, create a file called cowsay-discord.yml in your Kubernetes cluster and namespace where Argo Workflows is running, and add in the content below. Replace the <YOUR_DISCORD_WEBHOOK_URL> with the webhook URL you copied in the previous step (step 5).
- Run cowsay-discord.yml using argo submit -n argo --watch cowsay-discord.yml. This should send a Discord message to the channel you created.
In this article, you learned how to trigger notifications in Argo Workflows. You also learned how to send notifications to Slack, PagerDuty, Discord, Twilio, SMS, and email from exit handlers.
Setting up Argo workflows across different clusters in your infrastructure means you have to manage your infrastructure alongside your workflows. An easier way to set up workflows is through a fully managed service like Pipekit.
Pipekit lets you set up production-ready Argo workflows across multiple Kubernetes clusters. With Pipekit, you can trigger workflows, collect logs, and manage secrets without managing your infrastructure.
Subscribe for Pipekit updates.
Get the latest articles on all things Pipekit & data orchestration delivered straight to your inbox.