Audit logging in Apache Airflow - Step-by-Step Execution

Process Flow - Audit logging

User triggers action

↓

Airflow intercepts event

↓

Create audit log entry

↓

Store log in audit database/file

↓

Admin reviews logs for security/compliance

Audit logging in Airflow captures user actions and system events, storing them for review and compliance.

Execution Sample

Apache Airflow

from airflow.utils.log.logging_mixin import LoggingMixin
logger = LoggingMixin().log
logger.info('User X triggered DAG run')

This code logs an audit message when a user triggers a DAG run in Airflow.

Process Table

Step	Action	Event Captured	Log Entry Created	Storage Location
1	User triggers DAG run	DAG run request by user X	Log entry with timestamp and user info	Audit log file or DB
2	Airflow processes DAG run	DAG execution started	Log entry with DAG id and start time	Audit log file or DB
3	Task completes	Task success/failure status	Log entry with task id and status	Audit log file or DB
4	Admin queries logs	Audit log retrieval	No new log created	Logs displayed to admin

💡 All relevant user and system events are logged for auditing purposes.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
audit_log	empty	entry for DAG run request	entry for DAG start	entry for task status	all entries stored

Key Moments - 2 Insights

Why do we log both user actions and system events?

Logging both user actions and system events ensures a complete audit trail, as shown in execution_table rows 1 and 2, capturing who did what and what happened in the system.

Where are audit logs stored in Airflow?

Audit logs are stored in a dedicated file or database, as indicated in the 'Storage Location' column of the execution_table, ensuring logs are safe and accessible.

Visual Quiz - 3 Questions

Look at the execution table, what event is captured at Step 3?

A User triggers DAG run

B Task success/failure status

C DAG execution started

D Audit log retrieval

💡 Hint

Check the 'Event Captured' column for Step 3 in the execution_table.

At which step is no new log entry created?

A Step 4

B Step 2

C Step 1

D Step 3

💡 Hint

Look at the 'Log Entry Created' column in the execution_table for Step 4.

If the user action logging is removed, which step's log entry would be missing?

A Step 2

B Step 3

C Step 1

D Step 4

💡 Hint

Refer to the 'Action' and 'Log Entry Created' columns for Step 1 in the execution_table.

Concept Snapshot

Audit logging in Airflow:
- Captures user actions and system events
- Logs include timestamps, user info, event details
- Stored in files or databases for security
- Enables review and compliance checks
- Essential for tracking changes and troubleshooting

Full Transcript

Audit logging in Airflow works by capturing every important action and event. When a user triggers a DAG run, Airflow creates a log entry with the user and timestamp. As the DAG runs, system events like task starts and completions are also logged. These logs are saved in a secure location such as a file or database. Admins can later review these logs to check for security or compliance. This process ensures a full record of what happened and who did it, helping with troubleshooting and audits.