Audit logging in Apache Airflow - Mini Project: Build & Apply

Audit Logging Setup in Apache Airflow

📖 Scenario: You are working as a DevOps engineer managing Apache Airflow workflows. Your team wants to keep track of who triggered which DAGs and when, to improve security and troubleshooting. You will set up a simple audit logging mechanism in Airflow to record DAG run events.

🎯 Goal: Build a basic audit logging setup in Apache Airflow that logs DAG run events with user and timestamp information to a file.

📋 What You'll Learn

Create a Python dictionary to simulate DAG run event data

Add a configuration variable for the audit log file path

Write a function to log the event data to the audit log file

Print the content of the audit log file to verify the logging

💡 Why This Matters

🌍 Real World

Audit logging in Airflow helps track who triggered workflows and when, which is important for security and troubleshooting in production data pipelines.

💼 Career

DevOps engineers and data engineers often implement audit logging to meet compliance and operational monitoring requirements.

1

Create DAG run event data

Create a Python dictionary called dag_run_event with these exact keys and values: 'dag_id': 'example_dag', 'run_id': 'manual__2024-06-01T12:00:00', 'user': 'admin', 'execution_date': '2024-06-01T12:00:00'.

Apache Airflow

# Create the dag_run_event dictionary with specified keys and values
# Your code here

Hint

Use curly braces to create a dictionary with the exact keys and values.

2

Add audit log file path configuration

Create a variable called audit_log_file and set it to the string 'audit_log.txt' to specify the audit log file path.

Apache Airflow

dag_run_event = {
    'dag_id': 'example_dag',
    'run_id': 'manual__2024-06-01T12:00:00',
    'user': 'admin',
    'execution_date': '2024-06-01T12:00:00'
}
# Create audit_log_file variable with value 'audit_log.txt'
# Your code here

Hint

Assign the string 'audit_log.txt' to the variable audit_log_file.

3

Write audit logging function

Define a function called log_audit_event that takes event and log_file as parameters. Inside the function, open log_file in append mode and write a line with the format: DAG {dag_id} run {run_id} triggered by {user} at {execution_date} using the values from event.

Apache Airflow

dag_run_event = {
    'dag_id': 'example_dag',
    'run_id': 'manual__2024-06-01T12:00:00',
    'user': 'admin',
    'execution_date': '2024-06-01T12:00:00'
}
audit_log_file = 'audit_log.txt'
# Define log_audit_event(event, log_file) function to append formatted event info to log_file
# Your code here

Hint

Use a function with parameters and open the file in append mode to write the formatted string.

4

Log event and display audit log content

Call log_audit_event with dag_run_event and audit_log_file. Then open audit_log_file in read mode and print its entire content.

Apache Airflow

dag_run_event = {
    'dag_id': 'example_dag',
    'run_id': 'manual__2024-06-01T12:00:00',
    'user': 'admin',
    'execution_date': '2024-06-01T12:00:00'
}
audit_log_file = 'audit_log.txt'

def log_audit_event(event, log_file):
    with open(log_file, 'a') as f:
        f.write(f"DAG {event['dag_id']} run {event['run_id']} triggered by {event['user']} at {event['execution_date']}\n")

# Call log_audit_event with dag_run_event and audit_log_file
# Then open audit_log_file in read mode and print its content
# Your code here

Hint

Call the logging function then read and print the file content exactly.