DAG performance tracking in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Identify DAG run duration from Airflow CLI output

You run the command airflow dags list-runs -d example_dag and get the following output snippet:

run_id          | state   | execution_date       | start_date          | end_date            
--------------- | ------- | --------------------| ------------------- | ------------------- 
manual__2024-06-01T00:00:00+00:00 | success | 2024-06-01 00:00:00 | 2024-06-01 00:01:00 | 2024-06-01 00:05:00

What is the duration of this DAG run?

A 4 minutes

B 5 minutes

C 1 minute

D 6 minutes

💡 Hint

Subtract the start_date from the end_date to find the duration.

The DAG started at 00:01 and ended at 00:05, so the duration is 4 minutes.

🧠 Conceptual

intermediate

Understanding Airflow task duration metrics

Which Airflow metric best helps you track how long individual tasks take to complete within a DAG run?

A Scheduler heartbeat

B Task duration

C DAG concurrency

D Pool slots

💡 Hint

Think about what measures time spent per task.

Task duration measures the time taken by each task to run, which helps track performance.

🔀 Workflow

advanced

Steps to enable DAG run duration logging in Airflow

You want to track DAG run durations by logging start and end times automatically. Which sequence of steps is correct?

A 3,1,2,4

B 1,3,2,4

C 1,2,3,4

D 2,1,3,4

💡 Hint

Start with enabling logging, then configure callbacks, then write functions.

First enable logging, then configure callbacks in the DAG, write the callback functions, and finally deploy and monitor.

❓ Troubleshoot

advanced

Diagnosing missing DAG run duration data in Airflow UI

You notice that DAG run durations are not showing in the Airflow UI. Which of the following is the most likely cause?

A The Airflow webserver is down

B The Airflow scheduler is running normally

C The DAG has completed successfully with recorded start and end times

D The DAG runs are stuck in queued state and never start

💡 Hint

If runs never start, duration cannot be calculated.

If DAG runs are stuck queued, they have no start or end time, so duration is missing.

✅ Best Practice

expert

Best practice for long-term DAG performance tracking in Airflow

What is the best practice to maintain historical DAG performance data over months without impacting Airflow database performance?

A Export DAG run and task instance metrics regularly to an external monitoring system

B Keep all DAG run data indefinitely in Airflow's metadata database

C Disable DAG run logging to reduce database size

D Manually delete DAG run records weekly

💡 Hint

Think about separating monitoring data from the main Airflow database.

Exporting metrics to external systems like Prometheus or ELK allows long-term tracking without slowing Airflow.