Airflow metrics with Prometheus - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to enable Prometheus metrics in Airflow's configuration.

Apache Airflow

[metrics]
enabled = [1]

Drag options to blanks, or click blank then click option'

AYes

BFalse

CNo

DTrue

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'Yes' or 'No' instead of boolean values.

Leaving the value as 'False' which disables metrics.

Setting enabled = True activates Prometheus metrics in Airflow.

2fill in blank

medium

Complete the code to set the Prometheus metrics port in Airflow's configuration.

Apache Airflow

[metrics]
port = [1]

Drag options to blanks, or click blank then click option'

A8080

B9090

C5000

D1234

Attempts:

3 left

💡 Hint

Common Mistakes

Using ports like 8080 or 5000 which are not default for Prometheus.

Choosing a random port number.

Port 9090 is the default port for Prometheus metrics in Airflow.

3fill in blank

hard

Fix the error in the command to start the Prometheus metrics server for Airflow.

Apache Airflow

airflow metrics [1]

Drag options to blanks, or click blank then click option'

Aserve

Blaunch

Cstart

Dexecute

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'start' or 'launch' which are not valid Airflow CLI commands.

Typing 'execute' instead of 'serve'.

The correct Airflow CLI command to start the metrics server is airflow metrics serve.

4fill in blank

hard

Fill both blanks to create a Prometheus scrape configuration for Airflow metrics.

Apache Airflow

- job_name: 'airflow'
  static_configs:
    - targets: ['[1]:[2]']

Drag options to blanks, or click blank then click option'

Alocalhost

B9090

C8080

Dairflow-metrics

Attempts:

3 left

💡 Hint

Common Mistakes

Using the wrong port like 8080.

Using a hostname that does not resolve to the metrics server.

The scrape target should be localhost:9090 to collect Airflow metrics locally on the default port.

5fill in blank

hard

Fill all three blanks to define a Prometheus alert rule for Airflow task failures.

Apache Airflow

groups:
- name: airflow_alerts
  rules:
  - alert: AirflowTaskFailure
    expr: airflow_task_failed_total[1] > [2]
    for: [3]m

Drag options to blanks, or click blank then click option'

A{job="airflow"}

B5

C10

D{job="scheduler"}

Attempts:

3 left

💡 Hint

Common Mistakes

Using the wrong job label like 'scheduler'.

Setting too low or too high thresholds without reason.

The alert triggers if the total failed tasks metric for the Airflow job exceeds 5 for 10 minutes.

Practice

1. What is the main purpose of integrating Prometheus with Airflow metrics?

easy

A. To schedule Airflow DAGs automatically

B. To store Airflow logs permanently

C. To monitor Airflow task and system health in real-time

D. To send emails from Airflow tasks

Solution

1. Step 1: Understand Airflow metrics role

   Airflow metrics provide data about task execution and system performance.

2. Step 2: Identify Prometheus integration purpose

   Prometheus collects and visualizes these metrics for monitoring and alerting.

3. Final Answer:

   To monitor Airflow task and system health in real-time -> Option C

4. Quick Check:

   Monitoring = B [OK]

Hint: Prometheus is for monitoring, not scheduling or logging [OK]

Common Mistakes:

• Confusing monitoring with scheduling
• Thinking Prometheus stores logs
• Assuming Prometheus sends emails

2. Which configuration option must be enabled in Airflow to expose metrics for Prometheus scraping?

easy

A. metrics_enabled = False

B. statsd_on = True

C. prometheus_exporter = False

D. logging_level = DEBUG

Solution

1. Step 1: Identify Airflow metric exposure setting

   Airflow uses StatsD to collect metrics, so enabling StatsD is required.

2. Step 2: Match correct config option

   The correct option to enable StatsD is statsd_on = True.

3. Final Answer:

   statsd_on = True -> Option B

4. Quick Check:

   Enable StatsD = A [OK]

Hint: Enable StatsD to expose metrics for Prometheus [OK]

Common Mistakes:

• Disabling metrics with false flags
• Confusing logging level with metrics
• Assuming Prometheus exporter is off by default

3. Given this Airflow config snippet:

[metrics]
statsd_on = True
statsd_host = "localhost"
statsd_port = 8125

What will Prometheus scrape from Airflow?

medium

A. Airflow task execution metrics via StatsD exporter

B. Airflow DAG definitions

C. Airflow database contents

D. Airflow log files

Solution

1. Step 1: Analyze config snippet

   The snippet enables StatsD metrics collection on localhost port 8125.

2. Step 2: Understand Prometheus scraping

   Prometheus scrapes metrics exposed by StatsD exporter, which includes task execution data.

3. Final Answer:

   Airflow task execution metrics via StatsD exporter -> Option A

4. Quick Check:

   StatsD metrics = D [OK]

Hint: StatsD config means Prometheus scrapes metrics, not logs or DB [OK]

Common Mistakes:

• Thinking Prometheus scrapes DAG code
• Assuming Prometheus reads database directly
• Confusing logs with metrics

4. You enabled StatsD in Airflow but Prometheus shows no metrics. What is a likely cause?

medium

A. Prometheus scrape config missing or incorrect

B. Airflow DAGs are paused

C. Airflow scheduler is disabled

D. Airflow logs are too large

Solution

1. Step 1: Check Prometheus scraping setup

   If Prometheus shows no metrics, it may not be scraping the StatsD exporter endpoint correctly.

2. Step 2: Rule out unrelated causes

   Paused DAGs or scheduler off do not stop metrics exposure; large logs do not affect metrics scraping.

3. Final Answer:

   Prometheus scrape config missing or incorrect -> Option A

4. Quick Check:

   Scrape config issue = C [OK]

Hint: Check Prometheus scrape config if no metrics appear [OK]

Common Mistakes:

• Blaming paused DAGs for missing metrics
• Assuming scheduler off stops metrics
• Thinking logs size affects metrics

5. You want to create an alert in Prometheus for Airflow tasks that fail more than 5 times in 10 minutes. Which Prometheus query is correct?

hard

A. count(airflow_task_failures_total) == 5

B. sum(airflow_task_success_total) > 5

C. rate(airflow_task_failures_total[5m]) < 5

D. increase(airflow_task_failures_total[10m]) > 5

Solution

1. Step 1: Understand alert condition

   The alert triggers if failures increase by more than 5 in 10 minutes.

2. Step 2: Match Prometheus query

   increase(metric[10m]) > 5 checks if failures increased over 5 in last 10 minutes.

3. Step 3: Eliminate wrong options

   sum(airflow_task_success_total) > 5 checks successes, not failures; C uses wrong time and comparison; D counts current samples, not increase.

4. Final Answer:

   increase(airflow_task_failures_total[10m]) > 5 -> Option D

5. Quick Check:

   Use increase() over 10m > 5 for failure alert [OK]

Hint: Use increase() over time window to detect failure spikes [OK]

Common Mistakes:

• Using success metric instead of failure
• Using rate() incorrectly for count threshold
• Counting samples instead of increase