Log management and troubleshooting in Hadoop - Mini Project: Build & Apply

Log Management and Troubleshooting with Hadoop

📖 Scenario: You are a data engineer working with Hadoop. You receive a log file from a Hadoop cluster. The log contains entries with timestamps, log levels (INFO, WARN, ERROR), and messages. Your task is to analyze the log data to find how many ERROR messages occurred each hour.

🎯 Goal: Build a simple Hadoop MapReduce job to count the number of ERROR log entries per hour from the log data.

📋 What You'll Learn

Create a sample log dataset with timestamps and log levels

Set a filter to select only ERROR log entries

Write the MapReduce logic to count ERROR entries per hour

Output the count of ERROR messages grouped by hour

💡 Why This Matters

🌍 Real World

Analyzing logs helps find problems in Hadoop clusters quickly by showing when errors happen most often.

💼 Career

Data engineers and system administrators use log analysis to monitor system health and troubleshoot issues.

1

Create the sample log data

Create a list called logs with these exact entries as strings: "2024-06-01 10:15:23 INFO Job started", "2024-06-01 10:45:00 ERROR Disk failure", "2024-06-01 11:05:12 WARN Memory usage high", "2024-06-01 11:20:45 ERROR Network timeout", "2024-06-01 12:00:00 INFO Job finished"

Hadoop

# Create the list called logs with the exact entries
# Your code here

Hint

Use a Python list with the exact log strings inside double quotes.

2

Set the filter for ERROR logs

Create a variable called error_logs that contains only the entries from logs which include the exact substring "ERROR".

Hadoop

logs = [
    "2024-06-01 10:15:23 INFO Job started",
    "2024-06-01 10:45:00 ERROR Disk failure",
    "2024-06-01 11:05:12 WARN Memory usage high",
    "2024-06-01 11:20:45 ERROR Network timeout",
    "2024-06-01 12:00:00 INFO Job finished"
]
# Create error_logs list with only entries containing "ERROR"
# Your code here

Hint

Use a list comprehension to filter logs containing the substring "ERROR".

3

Count ERROR logs per hour

Create a dictionary called error_counts. Use a for loop with variable log to iterate over error_logs. Extract the hour from the timestamp (the substring from index 11 to 13) and count how many ERROR logs occur in each hour.

Hadoop

logs = [
    "2024-06-01 10:15:23 INFO Job started",
    "2024-06-01 10:45:00 ERROR Disk failure",
    "2024-06-01 11:05:12 WARN Memory usage high",
    "2024-06-01 11:20:45 ERROR Network timeout",
    "2024-06-01 12:00:00 INFO Job finished"
]

error_logs = [log for log in logs if "ERROR" in log]

# Create error_counts dictionary and count ERROR logs per hour
# Your code here

Hint

Use string slicing to get the hour from the timestamp and a dictionary to count occurrences.

4

Print the ERROR counts per hour

Write a print statement to display the error_counts dictionary.

Hadoop

logs = [
    "2024-06-01 10:15:23 INFO Job started",
    "2024-06-01 10:45:00 ERROR Disk failure",
    "2024-06-01 11:05:12 WARN Memory usage high",
    "2024-06-01 11:20:45 ERROR Network timeout",
    "2024-06-01 12:00:00 INFO Job finished"
]

error_logs = [log for log in logs if "ERROR" in log]

error_counts = {}
for log in error_logs:
    hour = log[11:13]
    if hour in error_counts:
        error_counts[hour] += 1
    else:
        error_counts[hour] = 1

# Print the error_counts dictionary
# Your code here

Hint

Use print(error_counts) to show the dictionary.

Practice

1. What is the main purpose of Hadoop logs in managing a cluster?

easy

A. To speed up data processing

B. To show what happens inside Hadoop and help find problems

C. To store user data permanently

D. To backup Hadoop configuration files

Solution

1. Step 1: Understand the role of logs

   Logs record events and actions inside Hadoop components.

2. Step 2: Identify the purpose of logs in troubleshooting

   Logs help find and fix problems by showing errors and system behavior.

3. Final Answer:

   To show what happens inside Hadoop and help find problems -> Option B

4. Quick Check:

   Logs = Problem detection [OK]

Hint: Logs help spot problems by showing system events [OK]

Common Mistakes:

• Confusing logs with data storage
• Thinking logs speed up processing
• Assuming logs backup configs

2. Which command is used to view logs of a YARN application in Hadoop?

easy

A. yarn logs -applicationId <app_id>

B. hdfs dfs -ls

C. hadoop fs -put

D. yarn application -status

Solution

1. Step 1: Identify commands related to logs

   Only yarn logs fetches logs for YARN apps.

2. Step 2: Match command with correct syntax

   yarn logs -applicationId <app_id> shows logs for the given app.

3. Final Answer:

   yarn logs -applicationId <app_id> -> Option A

4. Quick Check:

   YARN logs command = yarn logs -applicationId [OK]

Hint: Use 'yarn logs -applicationId' to get app logs fast [OK]

Common Mistakes:

• Using 'hdfs dfs -ls' which lists files, not logs
• Confusing 'hadoop fs -put' with log commands
• Using 'yarn application -status' which shows status, not logs

3. Given the command hdfs dfs -cat /logs/app1.log | grep ERROR, what is the expected output?

medium

A. A list of files in /logs directory

B. All lines from app1.log

C. Only lines without the word 'ERROR'

D. Only lines containing the word 'ERROR' from app1.log

Solution

1. Step 1: Understand the command parts

   hdfs dfs -cat /logs/app1.log outputs the file content; grep ERROR filters lines with 'ERROR'.

2. Step 2: Combine command effects

   The pipe sends file content to grep, which shows only lines containing 'ERROR'.

3. Final Answer:

   Only lines containing the word 'ERROR' from app1.log -> Option D

4. Quick Check:

   grep ERROR filters lines with ERROR [OK]

Hint: Pipe file content to grep ERROR to find error lines [OK]

Common Mistakes:

• Thinking grep removes ERROR lines
• Expecting full file output
• Confusing grep with listing files

4. You run yarn logs -applicationId app_12345 but get an error saying 'Application not found'. What is the likely cause?

medium

A. The logs directory is empty

B. The command syntax is wrong

C. The application ID is incorrect or does not exist

D. YARN is not installed on the cluster

Solution

1. Step 1: Analyze the error message

   'Application not found' means the given app ID is not recognized by YARN.

2. Step 2: Check command correctness

   The command syntax is correct; error is about app ID validity.

3. Final Answer:

   The application ID is incorrect or does not exist -> Option C

4. Quick Check:

   Invalid app ID = Application not found error [OK]

Hint: Check app ID spelling if 'Application not found' error appears [OK]

Common Mistakes:

• Assuming command syntax is wrong
• Thinking YARN is missing
• Believing logs directory emptiness causes this error

5. You want to quickly find all error messages in Hadoop logs stored in HDFS at /var/log/hadoop. Which command sequence will help you achieve this efficiently?

hard

A. hdfs dfs -cat /var/log/hadoop/* | grep ERROR

B. yarn logs -applicationId * | grep ERROR

C. hdfs dfs -ls /var/log/hadoop | grep ERROR

D. hadoop fs -copyToLocal /var/log/hadoop /tmp && grep ERROR /tmp/*

Solution

1. Step 1: Identify command to read all logs in HDFS directory

   hdfs dfs -cat /var/log/hadoop/* outputs all files in that directory.

2. Step 2: Filter output for error lines

   Piping to grep ERROR shows only lines with 'ERROR'.

3. Final Answer:

   hdfs dfs -cat /var/log/hadoop/* | grep ERROR -> Option A

4. Quick Check:

   Cat all logs + grep ERROR = quick error search [OK]

Hint: Cat all logs then grep ERROR to find errors fast [OK]

Common Mistakes:

• Using yarn logs without specifying app ID
• Using ls which lists files, not content
• Copying logs locally is slower and unnecessary