Kubernetes executor for dynamic scaling in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the Kubernetes Executor in Apache Airflow?

The Kubernetes Executor allows Airflow to run each task in its own Kubernetes pod, enabling dynamic scaling and isolation of tasks.

beginner

How does dynamic scaling work with the Kubernetes Executor?

Dynamic scaling means Airflow creates and deletes pods on demand based on the number of tasks to run, so resources are used efficiently.

beginner

Which Airflow configuration file is used to enable the Kubernetes Executor?

The airflow.cfg file is used to set the executor to KubernetesExecutor by changing the 'executor' parameter.

intermediate

What is the role of the Kubernetes pod template in the Kubernetes Executor?

The pod template defines how each task pod is created, including resources, environment variables, and volumes.

intermediate

Why is using the Kubernetes Executor beneficial for resource management?

It allows Airflow to use cluster resources only when needed, reducing idle resource waste and improving cost efficiency.

What does the Kubernetes Executor do in Airflow?

A Runs all tasks in a single pod

B Runs each task in a separate Kubernetes pod

C Runs tasks only on the Airflow scheduler node

D Runs tasks on virtual machines only

The Kubernetes Executor runs each task in its own pod, enabling dynamic scaling.

Which file do you edit to switch Airflow to use the Kubernetes Executor?

A airflow.cfg

B kubeconfig.yaml

C Dockerfile

D requirements.txt

The airflow.cfg file contains the executor setting to enable Kubernetes Executor.

What triggers Airflow to create new pods when using the Kubernetes Executor?

A Manual pod creation

B CPU usage on the scheduler

C Number of tasks to run

D Fixed number of pods at startup

Pods are created dynamically based on how many tasks need to run.

What is a key benefit of dynamic scaling with Kubernetes Executor?

A Efficient use of cluster resources

B Tasks run slower

C Requires manual pod management

D Uses fixed resources regardless of workload

Dynamic scaling helps use resources only when needed, saving costs.

What does the pod template specify in Kubernetes Executor?

A Scheduler settings

B Airflow DAG definitions

C Database connection info

D Pod configuration like resources and environment

The pod template defines how task pods are created and configured.

Explain how the Kubernetes Executor enables dynamic scaling in Airflow.

Think about how tasks and pods relate and how resources are managed.

Describe the role of the pod template in the Kubernetes Executor setup.

Consider what details are needed to create a pod for a task.