How to Install Apache Airflow Using Docker Quickly

To install Apache Airflow using Docker, use the official Airflow Docker image and run it with docker-compose. This sets up Airflow and its dependencies in containers, making installation easy and isolated.

Syntax

Use docker-compose with the official Airflow Docker image to start Airflow services. The main parts are:

• image: specifies the Airflow Docker image version.
• environment: sets Airflow environment variables like user and password.
• volumes: mounts local folders for DAGs and logs.
• ports: exposes Airflow webserver port to your machine.
• command: runs Airflow components like scheduler or webserver.

yaml

version: '3'
services:
  airflow-webserver:
    image: apache/airflow:2.7.1
    restart: always
    environment:
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - AIRFLOW__CORE__FERNET_KEY=YOUR_FERNET_KEY
      - AIRFLOW__CORE__LOAD_EXAMPLES=False
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
    ports:
      - 8080:8080
    command: webserver
  airflow-scheduler:
    image: apache/airflow:2.7.1
    restart: always
    environment:
      - AIRFLOW__CORE__FERNET_KEY=YOUR_FERNET_KEY
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
    command: scheduler

Example

This example shows a minimal docker-compose.yml file to run Airflow webserver and scheduler. It mounts local folders for your DAGs and logs, and exposes port 8080 to access the Airflow UI in your browser.

Run docker-compose up -d in the folder with this file to start Airflow.

yaml

version: '3'
services:
  airflow-webserver:
    image: apache/airflow:2.7.1
    restart: always
    environment:
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - AIRFLOW__CORE__FERNET_KEY=YOUR_FERNET_KEY
      - AIRFLOW__CORE__LOAD_EXAMPLES=False
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
    ports:
      - 8080:8080
    command: webserver
  airflow-scheduler:
    image: apache/airflow:2.7.1
    restart: always
    environment:
      - AIRFLOW__CORE__FERNET_KEY=YOUR_FERNET_KEY
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
    command: scheduler

Output

Creating network "default" with the default driver
Creating airflow-webserver ... done
Creating airflow-scheduler ... done

Common Pitfalls

Common mistakes when installing Airflow with Docker include:

• Not setting a FERNET_KEY, which is required for encrypting connections.
• Forgetting to mount local folders for DAGs and logs, so your workflows won't be visible or logs won't persist.
• Not exposing port 8080, so you cannot access the Airflow web UI.
• Running only the webserver without the scheduler, so tasks never run.

Always check Docker logs with docker-compose logs if something does not start.

yaml

Wrong example (missing FERNET_KEY):

version: '3'
services:
  airflow-webserver:
    image: apache/airflow:2.7.1
    environment:
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
    ports:
      - 8080:8080
    command: webserver

Right example (with FERNET_KEY):

version: '3'
services:
  airflow-webserver:
    image: apache/airflow:2.7.1
    environment:
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - AIRFLOW__CORE__FERNET_KEY=YOUR_FERNET_KEY
    ports:
      - 8080:8080
    command: webserver

Quick Reference

Tips for installing Airflow with Docker:

• Use the official apache/airflow image with a specific version tag.
• Set FERNET_KEY environment variable for security.
• Mount local folders for DAGs and logs to keep your workflows and logs persistent.
• Run both webserver and scheduler services for full functionality.
• Expose port 8080 to access the Airflow UI in your browser at http://localhost:8080.

Key Takeaways

Use the official apache/airflow Docker image with docker-compose for easy setup.

Always set the AIRFLOW__CORE__FERNET_KEY environment variable for encryption.

Mount local folders for DAGs and logs to keep your workflows and logs persistent.

Run both the webserver and scheduler containers to have Airflow fully functional.

Expose port 8080 to access the Airflow web UI in your browser.