AWS operators (S3, Redshift, EMR) in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of the S3 operator in Airflow?

The S3 operator in Airflow helps automate tasks like uploading, downloading, or deleting files in Amazon S3 storage. It makes managing files in the cloud easy and automatic.

beginner

How does the Redshift operator in Airflow help with data workflows?

The Redshift operator runs SQL commands on Amazon Redshift clusters. It helps automate data loading, transformation, and querying tasks in a data warehouse.

intermediate

What is Amazon EMR and how does the EMR operator in Airflow interact with it?

Amazon EMR is a cloud service for big data processing using tools like Hadoop and Spark. The EMR operator in Airflow starts, monitors, and stops EMR clusters to run data jobs automatically.

intermediate

Why is it important to use AWS operators in Airflow instead of manual scripts?

AWS operators in Airflow provide reliable, repeatable, and easy-to-manage automation for cloud tasks. They handle errors and retries, making workflows more stable than manual scripts.

beginner

Name one best practice when using AWS operators in Airflow.

Use connection IDs and credentials stored securely in Airflow's connection manager instead of hardcoding keys. This keeps your cloud access safe and manageable.

Which Airflow operator would you use to upload a file to Amazon S3?

A EMRCreateJobFlowOperator

B RedshiftSQLOperator

C S3FileTransformOperator

D S3CreateBucketOperator

The S3FileTransformOperator is used to upload or transform files in S3. RedshiftSQLOperator runs SQL on Redshift, EMRCreateJobFlowOperator manages EMR clusters, and S3CreateBucketOperator creates buckets.

What does the Redshift operator in Airflow primarily execute?

A Run SQL queries on Redshift

B Start EMR clusters

C Upload files to S3

D Create S3 buckets

The Redshift operator runs SQL queries on Redshift clusters to manage data workflows.

Which AWS service does the EMR operator in Airflow control?

A Amazon S3

B Amazon Redshift

C Amazon RDS

D Amazon EMR

The EMR operator manages Amazon EMR clusters for big data processing.

Why should AWS credentials be stored in Airflow connections instead of hardcoded?

A To improve security and manageability

B To make code longer

C To slow down workflows

D To avoid using AWS

Storing credentials in Airflow connections keeps them secure and easy to update without changing code.

Which operator would you use to start an EMR cluster in Airflow?

A S3ListOperator

B EMRCreateJobFlowOperator

C RedshiftDataOperator

D S3DeleteObjectsOperator

EMRCreateJobFlowOperator starts EMR clusters. The others manage S3 or Redshift tasks.

Explain how Airflow AWS operators help automate cloud workflows for S3, Redshift, and EMR.

Think about what each AWS service does and how Airflow operators interact with them.

Describe best practices for securely using AWS operators in Airflow.

Focus on security and reliability in automation.