Audit logging in Hadoop - Step-by-Step Execution

Concept Flow - Audit logging

User Action Initiated

↓

Hadoop Service Receives Request

↓

Audit Logger Captures Event Details

↓

Write Event to Audit Log File

↓

Log Entry Stored with Timestamp, User, Action

↓

Audit Log Available for Review or Analysis

This flow shows how a user action in Hadoop triggers audit logging, capturing details and storing them for later review.

Execution Sample

Hadoop

hadoop.security.authorization.enabled=true
hadoop.security.audit.logger=org.apache.hadoop.security.authorize.AuditLogger
# User 'alice' reads file /data/file1

This config enables audit logging in Hadoop and logs a user reading a file.

Execution Table

Step	Action	Event Captured	Log Entry Content	Result
1	User 'alice' requests to read /data/file1	User, Action, Resource	alice READ /data/file1	Audit logger triggered
2	Audit logger formats log entry	Timestamp, User, Action, Resource	2024-06-01T10:00:00Z alice READ /data/file1	Log entry ready
3	Write log entry to audit log file	Log entry	2024-06-01T10:00:00Z alice READ /data/file1	Entry stored in audit log
4	Audit log file updated	File size increased	Audit log file contains new entry	Audit log ready for review
5	No further actions	N/A	N/A	End of audit logging for this event

💡 Audit log entry created and stored successfully for the user action

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
User Action	None	alice reads /data/file1	alice reads /data/file1	alice reads /data/file1	alice reads /data/file1
Log Entry	Empty	Empty	2024-06-01T10:00:00Z alice READ /data/file1	2024-06-01T10:00:00Z alice READ /data/file1	Stored in audit log file

Key Moments - 2 Insights

Why does the audit logger capture the event before writing to the log file?

The audit logger first formats and captures all event details (Step 2) to ensure the log entry is complete and accurate before writing it to the file (Step 3).

What information is always included in an audit log entry?

Each audit log entry includes a timestamp, the user who performed the action, the action type, and the resource involved, as shown in Step 2 and Step 3.

Visual Quiz - 3 Questions

Look at the execution table, what is the log entry content at Step 2?

A User 'alice' requests to read /data/file1

B alice READ /data/file1

C 2024-06-01T10:00:00Z alice READ /data/file1

D Audit log file contains new entry

💡 Hint

Check the 'Log Entry Content' column at Step 2 in the execution table.

At which step is the audit log file actually updated with the new entry?

A Step 1

B Step 4

C Step 2

D Step 3

💡 Hint

Look for the step where the 'Audit log file updated' action occurs in the execution table.

If the user action was 'write' instead of 'read', how would the log entry change at Step 2?

A Timestamp alice WRITE /data/file1

B Timestamp alice READ /data/file1

C alice WRITE /data/file1

D User 'alice' writes to /data/file1

💡 Hint

Refer to the pattern of log entries in Step 2 and replace the action accordingly.

Concept Snapshot

Audit logging in Hadoop captures user actions like reads or writes.
It records details: timestamp, user, action, and resource.
Logs are stored in files for security and review.
Enable audit logging via configuration settings.
Each user action triggers a log entry creation and storage.

Full Transcript

Audit logging in Hadoop works by capturing user actions such as reading or writing files. When a user performs an action, the Hadoop service receives the request and the audit logger captures the event details including the user name, action type, and resource accessed. This information is formatted into a log entry with a timestamp. The log entry is then written to an audit log file. This file grows as more actions are logged and can be reviewed later for security or troubleshooting. The process ensures every important user action is recorded reliably.

Practice

1. What is the main purpose of audit logging in Hadoop?

easy

A. To backup data automatically

B. To improve data processing speed

C. To compress files for storage

D. To track user actions for security and troubleshooting

Solution

1. Step 1: Understand audit logging function

   Audit logging records user actions like access and changes in Hadoop.

2. Step 2: Identify main goal

   This helps with security checks and troubleshooting issues.

3. Final Answer:

   To track user actions for security and troubleshooting -> Option D

4. Quick Check:

   Audit logging = Track user actions [OK]

Hint: Audit logs = user actions tracking for security [OK]

Common Mistakes:

• Confusing audit logging with data backup
• Thinking audit logs speed up processing
• Mixing audit logging with file compression

2. Which configuration property must be set to true to enable audit logging in Hadoop?

easy

A. hadoop.logging.audit.enabled

B. hadoop.security.audit.enable

C. hadoop.audit.logging.active

D. hadoop.security.authorization

Solution

1. Step 1: Identify correct property name

   The official property to enable audit logging is hadoop.security.audit.enable.

2. Step 2: Confirm value to enable

   Setting it to true activates audit logging features.

3. Final Answer:

   hadoop.security.audit.enable -> Option B

4. Quick Check:

   Enable audit logging = hadoop.security.audit.enable true [OK]

Hint: Enable audit with hadoop.security.audit.enable true [OK]

Common Mistakes:

• Using incorrect property names
• Setting property to false
• Confusing audit logging with general logging

3. Given the following audit log entry:

2024-06-01 10:15:30 INFO User: alice accessed /data/input/file1.txt

What does this log entry indicate?

medium

A. User alice modified the file /data/input/file1.txt

B. User alice deleted the file /data/input/file1.txt

C. User alice accessed the file /data/input/file1.txt

D. User alice created the file /data/input/file1.txt

Solution

1. Step 1: Read the log message carefully

   The log states "User: alice accessed /data/input/file1.txt" which means alice opened or read the file.

2. Step 2: Understand the action word

   "accessed" means viewing or reading, not modifying or deleting.

3. Final Answer:

   User alice accessed the file /data/input/file1.txt -> Option C

4. Quick Check:

   Access means read/view, not modify [OK]

Hint: Look for action verb in log entry to know user activity [OK]

Common Mistakes:

• Assuming access means modification
• Confusing access with deletion
• Ignoring the exact log message wording

4. You enabled audit logging by setting hadoop.security.authorization to true, but no audit logs are generated. What is the most likely cause?

medium

A. Audit logger classes are not configured properly

B. Hadoop service is not running

C. User permissions are too high

D. Audit logging is disabled by default

Solution

1. Step 1: Check audit logging prerequisites

   Besides enabling authorization, audit logger classes must be configured to capture logs.

2. Step 2: Identify missing configuration

   If audit logger classes are missing or misconfigured, no logs will appear despite enabling authorization.

3. Final Answer:

   Audit logger classes are not configured properly -> Option A

4. Quick Check:

   Enable + configure logger classes = audit logs [OK]

Hint: Enable plus configure logger classes to get audit logs [OK]

Common Mistakes:

• Assuming service not running causes no logs
• Thinking user permissions block audit logs
• Ignoring need to configure logger classes

5. You want to audit all user actions on Hadoop files and see who changed data and when. Which combination of steps is correct?

hard

A. Set hadoop.security.authorization=true and configure audit logger classes

B. Set hadoop.security.authorization=false and enable debug logging

C. Only configure audit logger classes without enabling authorization

D. Enable file system replication and audit logging

Solution

1. Step 1: Enable security authorization

   Setting hadoop.security.authorization=true activates audit logging capability.

2. Step 2: Configure audit logger classes

   These classes capture and record user actions in logs.

3. Final Answer:

   Set hadoop.security.authorization=true and configure audit logger classes -> Option A

4. Quick Check:

   Enable authorization + logger classes = full audit [OK]

Hint: Enable authorization and logger classes for full audit [OK]

Common Mistakes:

• Disabling authorization but expecting audit logs
• Relying on debug logging for audit
• Confusing replication with audit logging