Audit logging in Hadoop - Mini Project: Build & Apply

Audit Logging Setup in Hadoop

📖 Scenario: You are a Hadoop administrator responsible for ensuring that all user activities on the Hadoop cluster are recorded for security and compliance. Audit logging helps track who accessed what data and when.

🎯 Goal: Set up a basic audit logging configuration in Hadoop by creating the necessary configuration entries and verifying the audit log output.

📋 What You'll Learn

Create a Hadoop configuration dictionary with core properties

Add audit logging specific properties to the configuration

Implement the logic to enable audit logging in the configuration

Print the final configuration to verify audit logging is enabled

💡 Why This Matters

🌍 Real World

Audit logging in Hadoop is essential for security audits, compliance, and troubleshooting by recording user actions on the cluster.

💼 Career

Hadoop administrators and DevOps engineers use audit logging configurations to maintain secure and compliant big data environments.

1

Create initial Hadoop configuration dictionary

Create a dictionary called hadoop_config with these exact entries: 'fs.defaultFS': 'hdfs://localhost:9000', 'dfs.replication': '1'

Hadoop

# Create a dictionary called hadoop_config with the specified entries
# Your code here

Need a hint?

Use curly braces to create a dictionary and include the exact keys and values.

2

Add audit logging properties to configuration

Add these two entries to the hadoop_config dictionary: 'dfs.namenode.audit.log.enabled': 'true' and 'dfs.namenode.audit.log.async': 'false'

Hadoop

hadoop_config = {
    'fs.defaultFS': 'hdfs://localhost:9000',
    'dfs.replication': '1'
}
# Add audit logging properties to hadoop_config below
# Your code here

Need a hint?

Add the new key-value pairs inside the existing dictionary with commas separating entries.

3

Enable audit logging in the configuration

Write a function called enable_audit_logging that takes a dictionary config and sets config['dfs.namenode.audit.log.enabled'] to 'true'. Then call this function with hadoop_config.

Hadoop

hadoop_config = {
    'fs.defaultFS': 'hdfs://localhost:9000',
    'dfs.replication': '1',
    'dfs.namenode.audit.log.enabled': 'true',
    'dfs.namenode.audit.log.async': 'false'
}
# Define enable_audit_logging function and call it with hadoop_config
# Your code here

Need a hint?

Define a function that updates the dictionary key and call it with the existing configuration.

4

Print the final Hadoop configuration

Write a print statement to display the hadoop_config dictionary.

Hadoop

hadoop_config = {
    'fs.defaultFS': 'hdfs://localhost:9000',
    'dfs.replication': '1',
    'dfs.namenode.audit.log.enabled': 'true',
    'dfs.namenode.audit.log.async': 'false'
}

def enable_audit_logging(config):
    config['dfs.namenode.audit.log.enabled'] = 'true'

enable_audit_logging(hadoop_config)
# Print the hadoop_config dictionary below
# Your code here

Need a hint?

Use print(hadoop_config) to show the dictionary content.