Audit logging in Apache Airflow - Commands & Configuration

Introduction

Audit logging helps you keep track of who did what and when in your Airflow environment. It records actions like task runs and configuration changes to improve security and troubleshooting.

When you want to see who triggered a specific workflow or task in Airflow

When you need to track changes made to DAGs or Airflow configurations

When you want to monitor user activity for compliance or security audits

When troubleshooting failures and you want to check recent user actions

When you want to keep a history of Airflow events for reporting

Config File - airflow.cfg

airflow.cfg

[logging]
audit_log = True
audit_log_filename = /opt/airflow/logs/audit.log
audit_log_level = INFO

This configuration enables audit logging in Airflow.

audit_log: Turns audit logging on.

audit_log_filename: Path where audit logs are saved.

audit_log_level: Sets the detail level of audit logs.

Commands

Initializes the Airflow database to prepare for audit logging and other features.

Terminal

airflow db init

Expected Output

DB: Initialized the metadata database.

Starts the Airflow scheduler which will now log audit events as configured.

Terminal

airflow scheduler

Expected Output

[2024-06-01 12:00:00,000] {scheduler_job.py:123} INFO - Starting scheduler

Shows the last 10 lines of the audit log to verify audit logging is working.

Terminal

tail -n 10 /opt/airflow/logs/audit.log

Expected Output

2024-06-01 12:05:00 INFO User 'admin' triggered DAG 'example_dag'

2024-06-01 12:06:00 INFO Task 'task1' in DAG 'example_dag' succeeded

→

-n 10 - Shows only the last 10 lines of the log file

Key Concept

If you remember nothing else from audit logging, remember: it records user actions and system events to help track changes and troubleshoot issues.

Common Mistakes

Not enabling audit_log in airflow.cfg

Without enabling audit_log, no audit events will be recorded, so you won't have any audit trail.

Set audit_log = True in the [logging] section of airflow.cfg and restart Airflow.

Checking the wrong log file path

Audit logs won't appear if you look at the default logs instead of the audit_log_filename path.

Verify the audit_log_filename path in airflow.cfg and check that file for audit entries.

Summary

Enable audit logging by setting audit_log = True in airflow.cfg.

Start Airflow scheduler to begin recording audit events.

Check the audit log file to see recorded user actions and system events.