AWS operators (S3, Redshift, EMR) in Apache Airflow - Step-by-Step Execution

Process Flow - AWS operators (S3, Redshift, EMR)

Start Airflow DAG

↓

Trigger S3 Operator

↓

Upload/Download Data

↓

Trigger Redshift Operator

↓

Run SQL Queries

↓

Trigger EMR Operator

↓

Run Big Data Jobs

↓

Complete DAG Execution

The Airflow DAG triggers AWS operators in sequence: first S3 for data transfer, then Redshift for SQL tasks, and finally EMR for big data processing.

Execution Sample

Apache Airflow

from airflow.providers.amazon.aws.operators.s3 import S3CreateObjectOperator
from airflow.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperator
from airflow.providers.amazon.aws.operators.emr import EmrCreateJobFlowOperator

# Simplified DAG tasks

This code snippet shows how Airflow uses AWS operators to interact with S3, Redshift, and EMR services.

Process Table

Step	Operator Triggered	Action	AWS Service Response	Next Step
1	S3CreateObjectOperator	Upload file to S3 bucket	File uploaded successfully	Trigger RedshiftSQLOperator
2	RedshiftSQLOperator	Execute SQL query on Redshift cluster	Query executed successfully	Trigger EmrCreateJobFlowOperator
3	EmrCreateJobFlowOperator	Start EMR cluster and run job flow	EMR job started	DAG completes
4	DAG Completion	All AWS tasks done	Success	End

💡 All AWS operators executed successfully, DAG run completes.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
s3_file_status	Not uploaded	Uploaded	Uploaded	Uploaded	Uploaded
redshift_query_status	Not run	Not run	Executed	Executed	Executed
emr_job_status	Not started	Not started	Not started	Started	Completed

Key Moments - 2 Insights

Why does the Redshift operator run only after the S3 operator completes?

Because the Redshift operator depends on data uploaded to S3; as shown in execution_table step 1 and 2, S3 upload must finish before Redshift query runs.

What happens if the EMR job fails to start?

The DAG will stop or retry depending on configuration; the variable_tracker shows emr_job_status changes only after successful start (step 3).

Visual Quiz - 3 Questions

Look at the execution_table, what is the AWS service response after the S3 operator runs?

A File uploaded successfully

B Query executed successfully

C EMR job started

D DAG completed

💡 Hint

Check execution_table row 1, column 'AWS Service Response'

At which step does the Redshift query execute?

A Step 1

B Step 3

C Step 2

D Step 4

💡 Hint

Look at execution_table row 2, 'Operator Triggered' column

If the S3 upload fails, what happens to the Redshift operator?

A It runs anyway

B It waits until S3 upload succeeds

C It runs before S3 upload

D It skips and DAG completes

💡 Hint

Refer to key_moments about operator dependencies and execution_table step order

Concept Snapshot

Airflow AWS Operators:
- S3CreateObjectOperator: upload/download files
- RedshiftSQLOperator: run SQL queries
- EmrCreateJobFlowOperator: start EMR jobs
Operators run in sequence with dependencies
Success means each AWS service confirms action

Full Transcript

This visual execution shows how an Airflow DAG uses AWS operators to manage cloud services. First, the S3 operator uploads a file to a bucket. Once successful, the Redshift operator runs a SQL query on the Redshift cluster. After that, the EMR operator starts a cluster and runs a job flow. Variables track the status of each step, ensuring the next operator runs only after the previous completes. Key moments clarify why order matters and what happens on failure. The quiz tests understanding of the sequence and dependencies.