Why production Airflow needs careful setup - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why is a metadata database critical in production Airflow?

In a production Airflow setup, what is the main role of the metadata database?

A It runs the actual tasks defined in workflows.

B It stores the state and history of all workflows and tasks.

C It provides the user interface for monitoring workflows.

D It schedules the execution of tasks directly.

💡 Hint

Think about where Airflow keeps track of what tasks have run and their results.

The metadata database holds all information about DAG runs, task instances, and their states. This is essential for Airflow to manage and resume workflows correctly.

✅ Best Practice

intermediate

What is a recommended practice for Airflow scheduler in production?

Which practice helps ensure the Airflow scheduler runs reliably in production?

A Run multiple schedulers in parallel with a shared metadata database.

B Run a single scheduler on a local machine without backups.

C Use the scheduler only during business hours.

D Disable the scheduler and trigger tasks manually.

💡 Hint

Consider how to avoid a single point of failure in scheduling.

Running multiple schedulers with a shared metadata database provides high availability and prevents downtime if one scheduler fails.

❓ Troubleshoot

advanced

What error occurs if Airflow workers lose connection to the metadata database?

In production, if Airflow workers cannot connect to the metadata database, what is the most likely outcome?

A The webserver crashes immediately.

B Airflow automatically switches to a backup database without interruption.

C The scheduler stops scheduling new tasks but workers continue running.

D Tasks fail to update their status, causing stuck or orphaned tasks.

💡 Hint

Think about what happens when task state updates cannot be saved.

If workers cannot update the metadata database, task states remain unknown, leading to stuck or orphaned tasks that never complete properly.

🔀 Workflow

advanced

How to ensure Airflow DAGs are deployed safely in production?

Which approach best ensures safe deployment of DAGs in a production Airflow environment?

A Edit DAG files directly on the production server via SSH.

B Upload DAGs manually through the Airflow web UI.

C Use a version-controlled repository and CI/CD pipeline to deploy DAGs.

D Deploy DAGs by copying files randomly to the scheduler machine.

💡 Hint

Consider how to track changes and avoid errors during deployment.

Using version control and automated deployment pipelines ensures changes are tested, tracked, and deployed consistently without manual errors.

🧠 Conceptual

expert

Why is resource isolation important for Airflow workers in production?

In a production Airflow setup, why should workers be isolated in separate containers or virtual machines?

A To prevent one task's failure or resource use from affecting others.

B To allow all tasks to share the same memory and CPU freely.

C To reduce the number of machines needed by running all tasks together.

D To make the Airflow webserver run faster.

💡 Hint

Think about how one task might impact others if not isolated.

Isolating workers prevents resource conflicts and failures in one task from crashing or slowing down others, improving stability and security.