Secrets management in Apache Airflow - Commands & Configuration

Introduction

Secrets management helps keep sensitive information like passwords and API keys safe when running workflows. It prevents exposing secrets directly in code or configuration files.

When you need to store database passwords securely for your Airflow tasks.

When you want to use API keys in your workflows without hardcoding them.

When multiple team members run workflows but should not see the secrets directly.

When you want to rotate or update secrets without changing your DAG code.

When you want to comply with security policies that forbid storing secrets in plain text.

Config File - airflow.cfg

airflow.cfg

[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {"connections_path": "airflow/connections", "kv_engine_version": 2, "url": "http://localhost:8200"}

This configuration tells Airflow to use HashiCorp Vault as the secrets backend.

backend: The Python class that manages secrets retrieval.

backend_kwargs: Parameters for connecting to Vault, including the path where secrets are stored and Vault URL.

Commands

Stores the database password securely in Vault under the path airflow/connections/my_db.

Terminal

vault kv put airflow/connections/my_db password='mysecretpassword'

Expected Output

Success! Data written to: airflow/connections/my_db

Retrieves the database connection details from Airflow, which fetches the password from Vault using the configured secrets backend.

Terminal

airflow connections get my_db

Expected Output

Conn Id: my_db

Conn Type: 

Host: 

Login: 

Password: mysecretpassword

Schema: 

Port: 

Extra: 

Lists all DAGs to confirm Airflow is running and can access secrets without errors.

Terminal

airflow dags list

Expected Output

dag_id

example_bash_operator

example_python_operator

Key Concept

If you remember nothing else from this pattern, remember: never store secrets directly in your DAG code or config files; always use a secrets backend.

Common Mistakes

Hardcoding passwords directly in DAG Python files.

This exposes sensitive data to anyone who can read the code and makes rotating secrets difficult.

Use Airflow's secrets backend to fetch passwords at runtime securely.

Not configuring the secrets backend in airflow.cfg.

Airflow will not know where to get secrets from and may fail or expose secrets in logs.

Set the backend and backend_kwargs properly in airflow.cfg before running workflows.

Storing secrets in plain text environment variables without encryption.

Environment variables can be exposed in process lists or logs.

Use a dedicated secrets manager like Vault integrated with Airflow.

Summary

Configure airflow.cfg to use a secrets backend like HashiCorp Vault.

Store secrets securely in Vault using the vault CLI.

Airflow fetches secrets at runtime, keeping them out of code and config files.