Managed Airflow (MWAA, Cloud Composer, Astronomer) - Step-by-Step Execution

Process Flow - Managed Airflow (MWAA, Cloud Composer, Astronomer)

Write DAG code

↓

Upload DAG to Managed Airflow service

↓

Service schedules DAG runs

↓

Workers execute tasks

↓

Monitor logs and status

↓

Adjust DAG or resources as needed

This flow shows how you write a workflow, upload it to a managed Airflow service, which then schedules and runs tasks automatically while you monitor progress.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('example', start_date=datetime(2024, 1, 1), schedule_interval='@daily')
t1 = BashOperator(task_id='print_date', bash_command='date', dag=dag)

Defines a simple DAG with one task that prints the date, ready to be uploaded to a managed Airflow service.

Process Table

Step	Action	Evaluation	Result
1	Write DAG code	Define DAG and tasks	DAG object created with task 'print_date'
2	Upload DAG	Place DAG file in service's DAG folder	Service detects new DAG
3	Schedule DAG	Service scheduler triggers DAG run at or after start_date	DAG run created
4	Execute task	Worker runs 'print_date' task	Date printed in logs
5	Monitor	Check logs and status in UI	Task success confirmed
6	Adjust	Modify DAG if needed	Changes applied on next upload

💡 DAG run completes after all tasks finish successfully

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
dag	None	DAG object created	Uploaded to service	Scheduled for run	Task executed	Run completed

Key Moments - 3 Insights

Why doesn't the DAG run immediately after uploading?

The execution_table row 3 shows the scheduler triggers the DAG run based on the start_date, so the DAG waits until that time to run.

How do I know if my task ran successfully?

Row 5 shows monitoring logs and status in the UI confirms task success, so checking logs is key.

What happens if I change the DAG code after upload?

Row 6 explains that changes apply on the next upload and next scheduled run, not immediately to running tasks.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the worker execute the task?

A Step 4

B Step 2

C Step 3

D Step 5

💡 Hint

Check the 'Action' column for 'Execute task' in the execution_table

According to variable_tracker, what is the state of 'dag' after Step 3?

A DAG object created

B Uploaded to service

C Scheduled for run

D Run completed

💡 Hint

Look at the 'After Step 3' column for 'dag' in variable_tracker

If the start_date is set to a future date, when will the DAG run start?

A Immediately after upload

B At the start_date time

C After the first task finishes

D Never

💡 Hint

Refer to execution_table row 3 about scheduling based on start_date

Concept Snapshot

Managed Airflow services let you upload DAG files that define workflows.
The service schedules and runs tasks automatically based on DAG settings.
You monitor runs via a web UI with logs and status.
Changes to DAGs apply on next upload and run.
Examples: AWS MWAA, Google Cloud Composer, Astronomer.

Full Transcript

Managed Airflow services simplify running workflows by handling scheduling and execution. You write DAG code defining tasks, upload it to the service, which then schedules runs based on your start_date. Workers execute tasks and you monitor progress through logs and UI. Changes to DAGs take effect on the next upload and scheduled run. This process helps automate complex workflows without managing infrastructure.

Practice

1. What is the main advantage of using a managed Airflow service like AWS MWAA or Google Cloud Composer?

easy

A. You don't need to manage the Airflow servers yourself.

B. You write DAGs in a new proprietary language.

C. You must manually install Airflow updates.

D. You cannot customize your workflows.

Solution

1. Step 1: Understand managed Airflow services

   Managed Airflow services handle the infrastructure and server management for you.

2. Step 2: Compare options

   The other options are incorrect because managed services still use Python DAGs, handle updates automatically, and allow workflow customization.

3. Final Answer:

   You don't need to manage the Airflow servers yourself. -> Option A

4. Quick Check:

   Managed Airflow = No server management [OK]

Hint: Managed means no server setup needed [OK]

Common Mistakes:

• Thinking you must manage Airflow servers manually
• Believing DAGs use a new language
• Assuming no workflow customization is possible

2. Which of the following is the correct way to upload your DAGs to AWS MWAA?

easy

A. Upload DAG files to an S3 bucket linked to MWAA.

B. SSH into the MWAA server and copy DAG files.

C. Use FTP to transfer DAG files to MWAA.

D. Email DAG files to AWS support.

Solution

1. Step 1: Identify MWAA DAG upload method

   AWS MWAA uses an S3 bucket to store DAG files that it reads automatically.

2. Step 2: Eliminate incorrect methods

   SSH and FTP are not supported because MWAA is fully managed and does not expose servers. Emailing files is not a valid method.

3. Final Answer:

   Upload DAG files to an S3 bucket linked to MWAA. -> Option A

4. Quick Check:

   MWAA DAGs stored in S3 bucket [OK]

Hint: MWAA DAGs live in S3 buckets [OK]

Common Mistakes:

• Trying to SSH into MWAA servers
• Using FTP which is unsupported
• Thinking AWS support manages DAG uploads

3. Given this snippet of a DAG uploaded to Google Cloud Composer, what will happen when the DAG runs?

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime
with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    task1 = BashOperator(task_id='print_date', bash_command='date')
    task2 = BashOperator(task_id='sleep', bash_command='sleep 5')
    task1 >> task2

medium

A. The DAG runs once and fails due to syntax error.

B. The DAG runs daily but skips the sleep task.

C. The DAG runs daily but tasks run in parallel.

D. The DAG runs daily, printing the date then sleeping for 5 seconds.

Solution

1. Step 1: Analyze DAG schedule and tasks

   The DAG is scheduled daily starting Jan 1, 2024. It has two tasks: print_date and sleep.

2. Step 2: Understand task dependencies

   task1 >> task2 means task1 runs before task2. So print_date runs first, then sleep runs for 5 seconds.

3. Final Answer:

   The DAG runs daily, printing the date then sleeping for 5 seconds. -> Option D

4. Quick Check:

   schedule=@daily; task1 >> task2 [OK]

Hint: >> means run first task before second [OK]

Common Mistakes:

• Assuming tasks run in parallel
• Thinking the DAG has syntax errors
• Believing the sleep task is skipped

4. You deployed a DAG to Astronomer but it does not appear in the Airflow UI. What is the most likely cause?

medium

A. Astronomer does not support Python DAGs.

B. The DAG file has a syntax error preventing it from loading.

C. You need to restart the Astronomer platform manually.

D. The DAG file is too large to upload.

Solution

1. Step 1: Check DAG loading requirements

   Astronomer runs standard Airflow and requires valid Python DAG files to load and show in UI.

2. Step 2: Identify common loading issues

   If the DAG has syntax errors, Airflow will skip loading it, so it won't appear in the UI.

3. Final Answer:

   The DAG file has a syntax error preventing it from loading. -> Option B

4. Quick Check:

   Syntax errors block DAG loading [OK]

Hint: Syntax errors hide DAGs in UI [OK]

Common Mistakes:

• Thinking Astronomer doesn't support Python DAGs
• Assuming manual restart is needed
• Believing file size blocks upload

5. You want to ensure your DAG in AWS MWAA uses a custom Python package not included by default. What is the best way to add this package?

hard

A. Include the package code inside your DAG Python file.

B. Manually install the package on the MWAA server via SSH.

C. Create a requirements.txt file with the package and upload it to the MWAA environment.

D. Request AWS support to install the package for you.

Solution

1. Step 1: Understand MWAA custom package installation

   MWAA allows adding Python packages by uploading a requirements.txt file listing them.

2. Step 2: Eliminate incorrect options

   SSH access is not allowed, embedding package code in DAG is bad practice, and AWS support does not install packages on demand.

3. Final Answer:

   Create a requirements.txt file with the package and upload it to the MWAA environment. -> Option C

4. Quick Check:

   requirements.txt is standard for MWAA packages [OK]

Hint: Use requirements.txt to add packages in MWAA [OK]

Common Mistakes:

• Trying to SSH into MWAA servers
• Embedding package code inside DAG files
• Expecting AWS support to install packages