Managed Airflow (MWAA, Cloud Composer, Astronomer) - Commands & Configuration

Introduction

Running workflows can be complex and managing the infrastructure for them takes time. Managed Airflow services like MWAA, Cloud Composer, and Astronomer handle the setup and maintenance for you, so you can focus on creating and running your workflows easily.

When you want to schedule and monitor data pipelines without managing servers.

When your team needs a reliable way to automate tasks in the cloud.

When you want automatic scaling and updates for your workflow system.

When you prefer a ready-to-use Airflow environment with built-in security.

When you want to integrate with cloud services easily without manual setup.

Config File - example_dag.py

example_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1),
}

dag = DAG(
    'example_managed_airflow_dag',
    default_args=default_args,
    schedule_interval='@daily',
    catchup=False
)

task1 = BashOperator(
    task_id='print_date',
    bash_command='date',
    dag=dag
)

task2 = BashOperator(
    task_id='sleep',
    bash_command='sleep 5',
    dag=dag
)

task1 >> task2

This is a simple Airflow DAG file that runs two tasks daily: printing the date and then sleeping for 5 seconds.

default_args sets the start date for the workflow.

DAG defines the workflow with a daily schedule.

BashOperator runs shell commands as tasks.

The last line sets the order: print_date runs before sleep.

Commands

This command creates a new MWAA environment named 'example-mwaa-env' with Airflow version 2.4.3. It specifies the S3 bucket where DAG files are stored, the execution role for permissions, and network settings for security.

Terminal

aws mwaa create-environment --name example-mwaa-env --airflow-version 2.4.3 --source-bucket-arn arn:aws:s3:::example-mwaa-bucket --dag-s3-path dags --execution-role-arn arn:aws:iam::123456789012:role/MWAA-Execution-Role --network-configuration SubnetIds=subnet-0123456789abcdef0,SecurityGroupIds=sg-0123456789abcdef0

Expected Output

{

    "Environment": {

        "Name": "example-mwaa-env",

        "Arn": "arn:aws:mwaa:us-east-1:123456789012:environment/example-mwaa-env",

        "AirflowVersion": "2.4.3",

        "Status": "CREATING"

    }

}

→

--name - Sets the environment name

→

--airflow-version - Specifies Airflow version

→

--source-bucket-arn - S3 bucket for DAGs

This command checks the status and details of the MWAA environment to know when it is ready to use.

Terminal

aws mwaa get-environment --name example-mwaa-env

Expected Output

{

    "Environment": {

        "Name": "example-mwaa-env",

        "Status": "AVAILABLE",

        "AirflowVersion": "2.4.3",

        "WebserverUrl": "https://example-mwaa-env.airflow.us-east-1.amazonaws.com"

    }

}

→

--name - Specifies which environment to check

Uploads the DAG file to the S3 bucket where MWAA reads DAGs from, so the workflow becomes available in the environment.

Terminal

aws s3 cp example_dag.py s3://example-mwaa-bucket/dags/example_dag.py

Expected Output

upload: ./example_dag.py to s3://example-mwaa-bucket/dags/example_dag.py

→

cp - Copy file to S3 bucket

Generates a login token URL to access the Airflow web interface for the managed environment.

Terminal

aws mwaa create-web-login-token --name example-mwaa-env

Expected Output

{

    "WebToken": "eyJraWQiOiJrMSIsImFsZyI6IlJTMjU2In0...",

    "WebServerHostname": "example-mwaa-env.airflow.us-east-1.amazonaws.com"

}

→

--name - Specifies the environment for login token

Key Concept

Managed Airflow services handle the infrastructure and let you focus on writing and running workflows easily in the cloud.

Common Mistakes

Uploading DAG files to the wrong S3 bucket or path.

The managed Airflow environment won't see the DAGs, so workflows won't run.

Always upload DAG files to the exact S3 bucket and folder configured for your environment.

Trying to access the Airflow UI before the environment status is AVAILABLE.

The environment is still setting up, so the UI is not ready and login will fail.

Check environment status with 'aws mwaa get-environment' and wait until it shows AVAILABLE.

Not assigning the correct IAM execution role with needed permissions.

The environment cannot access S3 or other AWS services, causing workflow failures.

Create and assign an IAM role with permissions for S3, CloudWatch, and other required services.

Summary

Create a managed Airflow environment with cloud CLI specifying version, S3 bucket, and network.

Upload your DAG files to the configured S3 bucket so the environment can run your workflows.

Check environment status before accessing the Airflow UI using a generated login token.