Kubernetes executor for dynamic scaling in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

How does the Kubernetes Executor scale Airflow tasks?

In Airflow's Kubernetes Executor, how does dynamic scaling of task pods happen?

A It creates a new pod for each task dynamically and deletes it after task completion.

B It pre-creates a fixed number of pods and reuses them for all tasks.

C It runs all tasks inside a single pod and scales the pod's CPU resources.

D It schedules tasks on existing Kubernetes nodes without creating pods.

💡 Hint

Think about how Kubernetes manages workloads with pods for isolation.

The Kubernetes Executor creates a separate pod for each task dynamically. This allows Airflow to scale tasks independently and clean up resources after task completion.

💻 Command Output

intermediate

Output of kubectl get pods during Airflow task execution

What is the expected output of kubectl get pods when Airflow is running tasks with the Kubernetes Executor?

Apache Airflow

kubectl get pods

A Only one pod named airflow-scheduler running all tasks

B Multiple pods named airflow-task-xxxx in Running or Completed state

C No pods related to Airflow tasks, only system pods

D Pods named airflow-webserver running tasks

💡 Hint

Remember each task runs in its own pod with Kubernetes Executor.

With Kubernetes Executor, each Airflow task runs in its own pod. So kubectl get pods shows multiple pods named like airflow-task-xxxx in Running or Completed states.

❓ Configuration

advanced

Configuring Airflow to use Kubernetes Executor with dynamic scaling

Which configuration snippet correctly enables the Kubernetes Executor with dynamic pod scaling in airflow.cfg?

A

[core]
executor = KubernetesExecutor

[kubernetes]
namespace = airflow
in_cluster = True
worker_container_repository = apache/airflow
worker_container_tag = 2.6.2

B

[core]
executor = LocalExecutor

[kubernetes]
namespace = airflow
in_cluster = False

C

[core]
executor = CeleryExecutor

[kubernetes]
namespace = default
in_cluster = True

D

[core]
executor = KubernetesExecutor

[kubernetes]
namespace = default
in_cluster = False
worker_container_repository = apache/airflow

💡 Hint

Check the executor type and Kubernetes section for correct settings.

To enable Kubernetes Executor with dynamic scaling, set executor = KubernetesExecutor under [core] and configure the Kubernetes section with namespace, in_cluster, and worker container details.

❓ Troubleshoot

advanced

Troubleshooting Airflow task pods stuck in Pending state

You notice Airflow task pods created by Kubernetes Executor remain in Pending state and never run. What is the most likely cause?

A The Airflow webserver is not reachable by the user.

B Airflow scheduler is down and not creating pods.

C Insufficient Kubernetes cluster resources or missing node selectors preventing pod scheduling.

D The DAG files are missing from the Airflow home directory.

💡 Hint

Think about what causes pods to stay Pending in Kubernetes.

Pods stay in Pending state if Kubernetes cannot find suitable nodes to schedule them. This often happens due to insufficient cluster resources or restrictive node selectors.

🔀 Workflow

expert

Order the steps for Airflow Kubernetes Executor dynamic scaling

Arrange the following steps in the correct order for how Airflow Kubernetes Executor dynamically scales task execution.

A 3,2,1,4

B 2,1,3,4

C 1,3,2,4

D 1,2,3,4

💡 Hint

Think about the natural flow from scheduler to pod creation to execution.

The scheduler first detects the task, then creates the pod manifest, Kubernetes schedules the pod, and finally the task runs inside the pod.