Audit logging in Apache Airflow - Step-by-Step Execution

Process Flow - Audit logging

User triggers action

↓

Airflow intercepts event

↓

Create audit log entry

↓

Store log in audit database/file

↓

Admin reviews logs for security/compliance

Audit logging in Airflow captures user actions and system events, storing them for review and compliance.

Execution Sample

Apache Airflow

from airflow.utils.log.logging_mixin import LoggingMixin
logger = LoggingMixin().log
logger.info('User X triggered DAG run')

This code logs an audit message when a user triggers a DAG run in Airflow.

Process Table

Step	Action	Event Captured	Log Entry Created	Storage Location
1	User triggers DAG run	DAG run request by user X	Log entry with timestamp and user info	Audit log file or DB
2	Airflow processes DAG run	DAG execution started	Log entry with DAG id and start time	Audit log file or DB
3	Task completes	Task success/failure status	Log entry with task id and status	Audit log file or DB
4	Admin queries logs	Audit log retrieval	No new log created	Logs displayed to admin

💡 All relevant user and system events are logged for auditing purposes.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
audit_log	empty	entry for DAG run request	entry for DAG start	entry for task status	all entries stored

Key Moments - 2 Insights

Why do we log both user actions and system events?

Logging both user actions and system events ensures a complete audit trail, as shown in execution_table rows 1 and 2, capturing who did what and what happened in the system.

Where are audit logs stored in Airflow?

Audit logs are stored in a dedicated file or database, as indicated in the 'Storage Location' column of the execution_table, ensuring logs are safe and accessible.

Visual Quiz - 3 Questions

Look at the execution table, what event is captured at Step 3?

A User triggers DAG run

B Task success/failure status

C DAG execution started

D Audit log retrieval

💡 Hint

Check the 'Event Captured' column for Step 3 in the execution_table.

At which step is no new log entry created?

A Step 4

B Step 2

C Step 1

D Step 3

💡 Hint

Look at the 'Log Entry Created' column in the execution_table for Step 4.

If the user action logging is removed, which step's log entry would be missing?

A Step 2

B Step 3

C Step 1

D Step 4

💡 Hint

Refer to the 'Action' and 'Log Entry Created' columns for Step 1 in the execution_table.

Concept Snapshot

Audit logging in Airflow:
- Captures user actions and system events
- Logs include timestamps, user info, event details
- Stored in files or databases for security
- Enables review and compliance checks
- Essential for tracking changes and troubleshooting

Full Transcript

Audit logging in Airflow works by capturing every important action and event. When a user triggers a DAG run, Airflow creates a log entry with the user and timestamp. As the DAG runs, system events like task starts and completions are also logged. These logs are saved in a secure location such as a file or database. Admins can later review these logs to check for security or compliance. This process ensures a full record of what happened and who did it, helping with troubleshooting and audits.

Practice

1. What is the main purpose of audit logging in Apache Airflow?

easy

A. To speed up task execution

B. To manage Airflow plugins

C. To store DAG definitions

D. To record user actions for security and troubleshooting

Solution

1. Step 1: Understand audit logging concept

   Audit logging is designed to keep track of user actions and system events.

2. Step 2: Identify its purpose in Airflow

   This helps with security monitoring and troubleshooting issues by reviewing what happened.

3. Final Answer:

   To record user actions for security and troubleshooting -> Option D

4. Quick Check:

   Audit logging = record user actions [OK]

Hint: Audit logs track user actions for security [OK]

Common Mistakes:

• Confusing audit logs with performance logs
• Thinking audit logs store DAG code
• Assuming audit logs speed up tasks

2. Which section in airflow.cfg should you modify to enable audit logging?

easy

A. [core]

B. [logging]

C. [scheduler]

D. [webserver]

Solution

1. Step 1: Locate airflow.cfg configuration file

   This file contains sections for different Airflow components.

2. Step 2: Identify the section for logging settings

   The [logging] section controls log-related configurations including audit logging.

3. Final Answer:

   [logging] -> Option B

4. Quick Check:

   Audit logging config is under [logging] [OK]

Hint: Audit logging is set in [logging] section [OK]

Common Mistakes:

• Choosing [core] which is for general settings
• Selecting [scheduler] which controls scheduling
• Picking [webserver] which manages UI settings

3. Given this snippet from airflow.cfg:

[logging]
audit_log_enabled = True
audit_log_location = /var/log/airflow/audit.log

What will happen when a user triggers a DAG run?

medium

A. User action will be recorded in /var/log/airflow/audit.log

B. DAG will fail to start due to missing audit log

C. No audit log will be created because audit_log_enabled is False

D. Audit logs will be stored in the default Airflow logs folder

Solution

1. Step 1: Check audit_log_enabled value

   It is set to True, so audit logging is active.

2. Step 2: Check audit_log_location setting

   Logs will be saved to /var/log/airflow/audit.log as specified.

3. Final Answer:

   User action will be recorded in /var/log/airflow/audit.log -> Option A

4. Quick Check:

   Enabled True + location set = logs saved there [OK]

Hint: Enabled True means logs go to specified location [OK]

Common Mistakes:

• Assuming logs go to default folder ignoring config
• Thinking audit_log_enabled is False
• Believing DAG fails without audit log

4. You enabled audit logging but no logs appear in the specified file. Which is the most likely cause?

medium

A. Airflow scheduler is not running

B. audit_log_enabled is set to False

C. The audit_log_location path is incorrect or lacks write permission

D. DAGs are not defined properly

Solution

1. Step 1: Verify audit_log_enabled setting

   If it was False, no logs would be created, but question states it is enabled.

2. Step 2: Check file path and permissions

   If the path is wrong or Airflow lacks permission to write, logs won't appear.

3. Final Answer:

   The audit_log_location path is incorrect or lacks write permission -> Option C

4. Quick Check:

   Wrong path or no write permission blocks logs [OK]

Hint: Check file path and permissions first [OK]

Common Mistakes:

• Assuming scheduler not running causes no audit logs
• Blaming DAG definitions for audit logging issues
• Ignoring file system permissions

5. You want to ensure audit logs do not fill up disk space. Which approach is best to manage audit logging in Airflow?

hard

A. Configure log rotation on the audit_log_location file

B. Disable audit logging completely

C. Store audit logs in the Airflow database

D. Manually delete audit logs daily

Solution

1. Step 1: Understand audit log growth risk

   Audit logs can grow large and fill disk space if unmanaged.

2. Step 2: Choose best management method

   Configuring log rotation automatically archives and deletes old logs safely.

3. Step 3: Evaluate other options

   Disabling logs loses audit trail; storing in DB is not default; manual deletion is error-prone.

4. Final Answer:

   Configure log rotation on the audit_log_location file -> Option A

5. Quick Check:

   Log rotation prevents disk full from audit logs [OK]

Hint: Use log rotation to manage audit log size [OK]

Common Mistakes:

• Disabling audit logs instead of managing size
• Relying on manual deletion which is unreliable
• Assuming database storage is default or better