Managed Airflow (MWAA, Cloud Composer, Astronomer) - Step-by-Step Execution

Process Flow - Managed Airflow (MWAA, Cloud Composer, Astronomer)

Write DAG code

↓

Upload DAG to Managed Airflow service

↓

Service schedules DAG runs

↓

Workers execute tasks

↓

Monitor logs and status

↓

Adjust DAG or resources as needed

This flow shows how you write a workflow, upload it to a managed Airflow service, which then schedules and runs tasks automatically while you monitor progress.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('example', start_date=datetime(2024, 1, 1), schedule_interval='@daily')
t1 = BashOperator(task_id='print_date', bash_command='date', dag=dag)

Defines a simple DAG with one task that prints the date, ready to be uploaded to a managed Airflow service.

Process Table

Step	Action	Evaluation	Result
1	Write DAG code	Define DAG and tasks	DAG object created with task 'print_date'
2	Upload DAG	Place DAG file in service's DAG folder	Service detects new DAG
3	Schedule DAG	Service scheduler triggers DAG run at or after start_date	DAG run created
4	Execute task	Worker runs 'print_date' task	Date printed in logs
5	Monitor	Check logs and status in UI	Task success confirmed
6	Adjust	Modify DAG if needed	Changes applied on next upload

💡 DAG run completes after all tasks finish successfully

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
dag	None	DAG object created	Uploaded to service	Scheduled for run	Task executed	Run completed

Key Moments - 3 Insights

Why doesn't the DAG run immediately after uploading?

The execution_table row 3 shows the scheduler triggers the DAG run based on the start_date, so the DAG waits until that time to run.

How do I know if my task ran successfully?

Row 5 shows monitoring logs and status in the UI confirms task success, so checking logs is key.

What happens if I change the DAG code after upload?

Row 6 explains that changes apply on the next upload and next scheduled run, not immediately to running tasks.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the worker execute the task?

A Step 4

B Step 2

C Step 3

D Step 5

💡 Hint

Check the 'Action' column for 'Execute task' in the execution_table

According to variable_tracker, what is the state of 'dag' after Step 3?

A DAG object created

B Uploaded to service

C Scheduled for run

D Run completed

💡 Hint

Look at the 'After Step 3' column for 'dag' in variable_tracker

If the start_date is set to a future date, when will the DAG run start?

A Immediately after upload

B At the start_date time

C After the first task finishes

D Never

💡 Hint

Refer to execution_table row 3 about scheduling based on start_date

Concept Snapshot

Managed Airflow services let you upload DAG files that define workflows.
The service schedules and runs tasks automatically based on DAG settings.
You monitor runs via a web UI with logs and status.
Changes to DAGs apply on next upload and run.
Examples: AWS MWAA, Google Cloud Composer, Astronomer.

Full Transcript

Managed Airflow services simplify running workflows by handling scheduling and execution. You write DAG code defining tasks, upload it to the service, which then schedules runs based on your start_date. Workers execute tasks and you monitor progress through logs and UI. Changes to DAGs take effect on the next upload and scheduled run. This process helps automate complex workflows without managing infrastructure.