How to Use Kubernetes Executor in Airflow: Setup and Example

To use the KubernetesExecutor in Airflow, set executor = KubernetesExecutor in your airflow.cfg and configure the Kubernetes connection. This executor runs each task in its own Kubernetes pod, enabling scalable and isolated task execution.

Syntax

The main configuration to enable Kubernetes Executor in Airflow is done in the airflow.cfg file or environment variables. Key parts include:

• executor = KubernetesExecutor: sets the executor type.
• kubernetes_namespace: the Kubernetes namespace where pods run.
• kube_config: path to Kubernetes config file or use in-cluster config.
• worker_container_image: Docker image used for task pods.

These settings tell Airflow to launch each task as a separate pod in Kubernetes.

ini

[core]
executor = KubernetesExecutor

[kubernetes]
kubernetes_namespace = airflow
worker_container_image = apache/airflow:2.7.1-python3.10

# Optional: path to kubeconfig if running outside cluster
kube_config = /path/to/kubeconfig

Example

This example shows a simple Airflow DAG that runs a task using Kubernetes Executor. The task runs a bash command inside a Kubernetes pod.

python

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('k8s_executor_example', start_date=datetime(2024, 1, 1), schedule_interval='@once', catchup=False) as dag:
    task = BashOperator(
        task_id='print_date',
        bash_command='date'
    )

Output

Task 'print_date' runs inside a Kubernetes pod and prints the current date in the pod logs.

Common Pitfalls

• Not setting the executor: Forgetting to set executor = KubernetesExecutor in airflow.cfg means Airflow won’t use Kubernetes pods.
• Missing Kubernetes config: Airflow needs access to Kubernetes credentials either via kube_config or in-cluster config.
• Incorrect Docker image: The worker_container_image must have Airflow installed and compatible Python version.
• Namespace issues: Running pods in a namespace without proper permissions causes failures.

Always check Airflow scheduler logs for errors related to Kubernetes pod creation.

ini

## Wrong: executor not set
[core]
executor = LocalExecutor

## Right: executor set to KubernetesExecutor
[core]
executor = KubernetesExecutor

Quick Reference

Setting	Description	Example Value
executor	Type of executor Airflow uses	KubernetesExecutor
kubernetes_namespace	Kubernetes namespace for task pods	airflow
worker_container_image	Docker image for task pods	apache/airflow:2.7.1-python3.10
kube_config	Path to kubeconfig file (optional)	/path/to/kubeconfig
in_cluster	Use in-cluster config if running inside Kubernetes	True or False

Key Takeaways

Set 'executor = KubernetesExecutor' in airflow.cfg to enable Kubernetes Executor.

Configure Kubernetes namespace and worker container image correctly for task pods.

Ensure Airflow has access to Kubernetes credentials via kubeconfig or in-cluster config.

Each Airflow task runs in its own isolated Kubernetes pod for scalability.

Check scheduler logs for pod creation errors and permission issues.