0
0
Apache Airflowdevops~5 mins

Integration with PagerDuty and Slack in Apache Airflow - Commands & Configuration

Choose your learning style9 modes available
Introduction
Sometimes workflows fail or need attention. Integrating Airflow with PagerDuty and Slack helps you get alerts quickly so you can fix problems fast.
When a critical Airflow task fails and you want to get an immediate alert on Slack.
When you want to create an incident in PagerDuty automatically from Airflow failures.
When your team uses Slack for communication and PagerDuty for incident management.
When you want to reduce downtime by getting notified instantly about workflow issues.
When you want to keep track of Airflow alerts in one place without checking logs manually.
Config File - my_dag.py
my_dag.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator
from airflow.providers.pagerduty.operators.pagerduty import PagerDutyIncidentOperator

def task_that_fails():
    raise Exception('This task fails to trigger alerts')

with DAG(
    dag_id='pagerduty_slack_integration',
    start_date=days_ago(1),
    schedule_interval='@daily',
    catchup=False
) as dag:

    fail_task = PythonOperator(
        task_id='fail_task',
        python_callable=task_that_fails
    )

    slack_alert = SlackWebhookOperator(
        task_id='slack_alert',
        http_conn_id='slack_connection',
        message=':warning: Task failed in DAG pagerduty_slack_integration',
        channel='#alerts'
    )

    pagerduty_alert = PagerDutyIncidentOperator(
        task_id='pagerduty_alert',
        routing_key='your_pagerduty_integration_key',
        summary='Airflow task failure in pagerduty_slack_integration',
        source='airflow',
        severity='error'
    )

    fail_task >> [slack_alert, pagerduty_alert]

This DAG defines a failing task to simulate an error.

The SlackWebhookOperator sends a message to a Slack channel using a configured connection.

The PagerDutyIncidentOperator creates an incident in PagerDuty using your integration key.

The failing task triggers both alerts in parallel when it fails.

Commands
This command adds a connection in Airflow to Slack's webhook URL so the SlackWebhookOperator can send messages.
Terminal
airflow connections add 'slack_connection' --conn-type 'http' --conn-host 'hooks.slack.com' --conn-extra '{"webhook_token": "/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"}'
Expected OutputExpected
Connection `slack_connection` added successfully
--conn-type - Specifies the connection type, here 'http' for Slack webhook
--conn-host - Sets the Slack webhook host
--conn-extra - Sets the Slack webhook token path
This command manually triggers the DAG to run and test the integration with PagerDuty and Slack.
Terminal
airflow dags trigger pagerduty_slack_integration
Expected OutputExpected
Created <DagRun pagerduty_slack_integration @ 2024-06-01T00:00:00+00:00: manual__2024-06-01T00:00:00+00:00, externally triggered: True>
This command lists all tasks in the DAG to verify the task names and operators.
Terminal
airflow tasks list pagerduty_slack_integration
Expected OutputExpected
fail_task slack_alert pagerduty_alert
This command tests the failing task locally to see the error and trigger alerts manually.
Terminal
airflow tasks test pagerduty_slack_integration fail_task 2024-06-01
Expected OutputExpected
Traceback (most recent call last): File "/path/to/airflow/cli/commands/task_command.py", line 123, in test result = task.execute(context=context) File "/path/to/airflow/operators/python.py", line 114, in execute return self.execute_callable() File "/path/to/airflow/operators/python.py", line 130, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/path/to/my_dag.py", line 6, in task_that_fails raise Exception('This task fails to trigger alerts') Exception: This task fails to trigger alerts
Key Concept

If you remember nothing else from this pattern, remember: configure Airflow connections and use operators to send alerts automatically on task failures.

Common Mistakes
Not setting the Slack webhook URL correctly in Airflow connections.
Without the correct webhook URL, Slack messages will not be sent and you won't get alerts.
Use the exact Slack webhook URL when adding the connection with airflow connections add.
Using the wrong PagerDuty integration key or missing it in the PagerDutyIncidentOperator.
PagerDuty will reject the incident creation without a valid routing key, so no alert is created.
Copy the correct integration key from PagerDuty and set it in the operator's routing_key parameter.
Not linking the failing task to alert tasks with >> operator.
If the alert tasks are not downstream of the failing task, they won't run on failure.
Use fail_task >> [slack_alert, pagerduty_alert] to trigger alerts after failure.
Summary
Add Airflow connections for Slack webhook with the correct URL.
Create a DAG with a failing task and alert tasks using SlackWebhookOperator and PagerDutyIncidentOperator.
Trigger the DAG and test tasks to verify alerts are sent on failure.