Audit logging in Apache Airflow - Mini Project: Build & Apply

Audit Logging Setup in Apache Airflow

📖 Scenario: You are working as a DevOps engineer managing Apache Airflow workflows. Your team wants to keep track of who triggered which DAGs and when, to improve security and troubleshooting. You will set up a simple audit logging mechanism in Airflow to record DAG run events.

🎯 Goal: Build a basic audit logging setup in Apache Airflow that logs DAG run events with user and timestamp information to a file.

📋 What You'll Learn

Create a Python dictionary to simulate DAG run event data

Add a configuration variable for the audit log file path

Write a function to log the event data to the audit log file

Print the content of the audit log file to verify the logging

💡 Why This Matters

🌍 Real World

Audit logging in Airflow helps track who triggered workflows and when, which is important for security and troubleshooting in production data pipelines.

💼 Career

DevOps engineers and data engineers often implement audit logging to meet compliance and operational monitoring requirements.

1

Create DAG run event data

Create a Python dictionary called dag_run_event with these exact keys and values: 'dag_id': 'example_dag', 'run_id': 'manual__2024-06-01T12:00:00', 'user': 'admin', 'execution_date': '2024-06-01T12:00:00'.

Apache Airflow

# Create the dag_run_event dictionary with specified keys and values
# Your code here

Hint

Use curly braces to create a dictionary with the exact keys and values.

2

Add audit log file path configuration

Create a variable called audit_log_file and set it to the string 'audit_log.txt' to specify the audit log file path.

Apache Airflow

dag_run_event = {
    'dag_id': 'example_dag',
    'run_id': 'manual__2024-06-01T12:00:00',
    'user': 'admin',
    'execution_date': '2024-06-01T12:00:00'
}
# Create audit_log_file variable with value 'audit_log.txt'
# Your code here

Hint

Assign the string 'audit_log.txt' to the variable audit_log_file.

3

Write audit logging function

Define a function called log_audit_event that takes event and log_file as parameters. Inside the function, open log_file in append mode and write a line with the format: DAG {dag_id} run {run_id} triggered by {user} at {execution_date} using the values from event.

Apache Airflow

dag_run_event = {
    'dag_id': 'example_dag',
    'run_id': 'manual__2024-06-01T12:00:00',
    'user': 'admin',
    'execution_date': '2024-06-01T12:00:00'
}
audit_log_file = 'audit_log.txt'
# Define log_audit_event(event, log_file) function to append formatted event info to log_file
# Your code here

Hint

Use a function with parameters and open the file in append mode to write the formatted string.

4

Log event and display audit log content

Call log_audit_event with dag_run_event and audit_log_file. Then open audit_log_file in read mode and print its entire content.

Apache Airflow

dag_run_event = {
    'dag_id': 'example_dag',
    'run_id': 'manual__2024-06-01T12:00:00',
    'user': 'admin',
    'execution_date': '2024-06-01T12:00:00'
}
audit_log_file = 'audit_log.txt'

def log_audit_event(event, log_file):
    with open(log_file, 'a') as f:
        f.write(f"DAG {event['dag_id']} run {event['run_id']} triggered by {event['user']} at {event['execution_date']}\n")

# Call log_audit_event with dag_run_event and audit_log_file
# Then open audit_log_file in read mode and print its content
# Your code here

Hint

Call the logging function then read and print the file content exactly.

Practice

1. What is the main purpose of audit logging in Apache Airflow?

easy

A. To speed up task execution

B. To manage Airflow plugins

C. To store DAG definitions

D. To record user actions for security and troubleshooting

Solution

1. Step 1: Understand audit logging concept

   Audit logging is designed to keep track of user actions and system events.

2. Step 2: Identify its purpose in Airflow

   This helps with security monitoring and troubleshooting issues by reviewing what happened.

3. Final Answer:

   To record user actions for security and troubleshooting -> Option D

4. Quick Check:

   Audit logging = record user actions [OK]

Hint: Audit logs track user actions for security [OK]

Common Mistakes:

• Confusing audit logs with performance logs
• Thinking audit logs store DAG code
• Assuming audit logs speed up tasks

2. Which section in airflow.cfg should you modify to enable audit logging?

easy

A. [core]

B. [logging]

C. [scheduler]

D. [webserver]

Solution

1. Step 1: Locate airflow.cfg configuration file

   This file contains sections for different Airflow components.

2. Step 2: Identify the section for logging settings

   The [logging] section controls log-related configurations including audit logging.

3. Final Answer:

   [logging] -> Option B

4. Quick Check:

   Audit logging config is under [logging] [OK]

Hint: Audit logging is set in [logging] section [OK]

Common Mistakes:

• Choosing [core] which is for general settings
• Selecting [scheduler] which controls scheduling
• Picking [webserver] which manages UI settings

3. Given this snippet from airflow.cfg:

[logging]
audit_log_enabled = True
audit_log_location = /var/log/airflow/audit.log

What will happen when a user triggers a DAG run?

medium

A. User action will be recorded in /var/log/airflow/audit.log

B. DAG will fail to start due to missing audit log

C. No audit log will be created because audit_log_enabled is False

D. Audit logs will be stored in the default Airflow logs folder

Solution

1. Step 1: Check audit_log_enabled value

   It is set to True, so audit logging is active.

2. Step 2: Check audit_log_location setting

   Logs will be saved to /var/log/airflow/audit.log as specified.

3. Final Answer:

   User action will be recorded in /var/log/airflow/audit.log -> Option A

4. Quick Check:

   Enabled True + location set = logs saved there [OK]

Hint: Enabled True means logs go to specified location [OK]

Common Mistakes:

• Assuming logs go to default folder ignoring config
• Thinking audit_log_enabled is False
• Believing DAG fails without audit log

4. You enabled audit logging but no logs appear in the specified file. Which is the most likely cause?

medium

A. Airflow scheduler is not running

B. audit_log_enabled is set to False

C. The audit_log_location path is incorrect or lacks write permission

D. DAGs are not defined properly

Solution

1. Step 1: Verify audit_log_enabled setting

   If it was False, no logs would be created, but question states it is enabled.

2. Step 2: Check file path and permissions

   If the path is wrong or Airflow lacks permission to write, logs won't appear.

3. Final Answer:

   The audit_log_location path is incorrect or lacks write permission -> Option C

4. Quick Check:

   Wrong path or no write permission blocks logs [OK]

Hint: Check file path and permissions first [OK]

Common Mistakes:

• Assuming scheduler not running causes no audit logs
• Blaming DAG definitions for audit logging issues
• Ignoring file system permissions

5. You want to ensure audit logs do not fill up disk space. Which approach is best to manage audit logging in Airflow?

hard

A. Configure log rotation on the audit_log_location file

B. Disable audit logging completely

C. Store audit logs in the Airflow database

D. Manually delete audit logs daily

Solution

1. Step 1: Understand audit log growth risk

   Audit logs can grow large and fill disk space if unmanaged.

2. Step 2: Choose best management method

   Configuring log rotation automatically archives and deletes old logs safely.

3. Step 3: Evaluate other options

   Disabling logs loses audit trail; storing in DB is not default; manual deletion is error-prone.

4. Final Answer:

   Configure log rotation on the audit_log_location file -> Option A

5. Quick Check:

   Log rotation prevents disk full from audit logs [OK]

Hint: Use log rotation to manage audit log size [OK]

Common Mistakes:

• Disabling audit logs instead of managing size
• Relying on manual deletion which is unreliable
• Assuming database storage is default or better