Connection management for cloud services in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a connection in Apache Airflow?

A connection in Airflow stores the information needed to connect to external systems like cloud services, databases, or APIs. It includes details like host, login, password, and extra parameters.

beginner

How do you add a new cloud service connection in Airflow UI?

Go to Admin > Connections, click the '+' button, fill in the connection details such as Conn Id, Conn Type (e.g., Google Cloud, AWS), host, login, password, and extra JSON parameters, then save.

intermediate

What is the purpose of the 'Extra' field in Airflow connections?

The 'Extra' field stores additional JSON-formatted parameters needed for the connection, like project IDs, scopes, or authentication keys for cloud services.

intermediate

How can you use a connection in an Airflow DAG?

You reference the connection by its Conn Id in operators or hooks. Airflow uses the stored credentials to authenticate and interact with the cloud service during task execution.

beginner

Why is managing connections centrally in Airflow beneficial?

It keeps sensitive credentials secure and separate from code, allows easy updates without changing DAGs, and promotes reuse of connection info across multiple tasks and DAGs.

Where do you manage cloud service connections in Airflow?

A DAG Editor

B Admin > Connections

C Task Logs

D Plugins

Connections are managed centrally under Admin > Connections in the Airflow UI.

What format is used in the 'Extra' field for connection parameters?

A JSON

B YAML

C Plain text

D XML

The 'Extra' field uses JSON format to store additional connection parameters.

Which Airflow component uses the connection information during task execution?

A Hook or Operator

B Executor

C Scheduler

D Webserver

Hooks and Operators use connection info to authenticate and interact with external services.

Why should credentials be stored in Airflow connections instead of hardcoded in DAGs?

A To speed up task execution

B To improve code readability

C To reduce DAG size

D To keep credentials secure and separate from code

Storing credentials in connections keeps them secure and separate from code, making management easier.

Which of these is NOT a typical connection type in Airflow for cloud services?

A Google Cloud

B AWS

C FTP Server

D Azure

FTP Server is not a cloud service connection type; it is a file transfer protocol connection.

Explain how Airflow manages connections to cloud services and why this is important.

Think about how Airflow separates connection info from code.

Describe the steps to add a new AWS connection in Airflow UI.

Focus on the UI path and required fields.