Log management and troubleshooting in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Understanding Hadoop Log Levels

What will be the output of the following Hadoop log configuration snippet when the log level is set to WARN?

log4j.logger.org.apache.hadoop=INFO, console
log4j.logger.org.apache.hadoop.hdfs=WARN, console
log4j.logger.org.apache.hadoop.mapreduce=ERROR, console

Assuming a log event of level INFO from org.apache.hadoop.hdfs and a log event of level WARN from org.apache.hadoop.mapreduce, which will be printed to the console?

Hadoop

log4j.logger.org.apache.hadoop=INFO, console
log4j.logger.org.apache.hadoop.hdfs=WARN, console
log4j.logger.org.apache.hadoop.mapreduce=ERROR, console

A Only the INFO log from org.apache.hadoop.hdfs is printed

B Only the WARN log from org.apache.hadoop.hdfs is printed

C Only the WARN log from org.apache.hadoop.mapreduce is printed

D Both INFO from org.apache.hadoop.hdfs and WARN from org.apache.hadoop.mapreduce are printed

💡 Hint

Remember that logs are printed only if their level is equal or higher than the configured level.

The org.apache.hadoop.hdfs logger is set to WARN, so INFO logs are ignored. The WARN log from org.apache.hadoop.mapreduce is below ERROR level, so it is ignored. Only WARN logs from org.apache.hadoop.hdfs or higher are printed.

❓ data_output

intermediate

Parsing Hadoop Logs for Error Counts

Given a Hadoop log file with lines like:

2024-06-01 12:00:01,234 INFO Client: Connection established
2024-06-01 12:00:02,345 ERROR DataNode: Disk failure detected
2024-06-01 12:00:03,456 WARN NameNode: High memory usage

Which Python code snippet correctly counts the number of ERROR log entries?

Hadoop

log_lines = [
    '2024-06-01 12:00:01,234 INFO Client: Connection established',
    '2024-06-01 12:00:02,345 ERROR DataNode: Disk failure detected',
    '2024-06-01 12:00:03,456 WARN NameNode: High memory usage'
]

# Count ERROR logs

A error_count = len([line for line in log_lines if 'error' in line.lower()])

B error_count = len([line for line in log_lines if line.startswith('ERROR')])

C error_count = sum(1 for line in log_lines if line.split()[2] == 'ERROR')

D error_count = sum(1 for line in log_lines if 'ERROR' in line)

💡 Hint

Check how the log level appears in each line and how to detect it.

The log level appears after the timestamp and comma, so checking if 'ERROR' is in the line works. Option D fails because lines do not start with 'ERROR'. Option D fails because the third word is the time with comma, not the log level. Option D works but is case insensitive and less direct.

🔧 Debug

advanced

Troubleshooting Missing Hadoop Logs

A Hadoop cluster admin notices that logs from org.apache.hadoop.mapreduce are missing in the log files, even though tasks are running. The log4j.properties file contains:

log4j.logger.org.apache.hadoop=INFO, console
log4j.logger.org.apache.hadoop.mapreduce=OFF, console

What is the cause of missing logs for MapReduce?

A The console appender is not defined, so logs are not saved

B The log4j.properties file is missing the root logger configuration

C The log level OFF disables all logging for org.apache.hadoop.mapreduce

D The INFO level is too low to capture MapReduce logs

💡 Hint

Check what the OFF log level means in log4j.

Setting log level to OFF disables all logging for that logger. Hence, no MapReduce logs appear. The console appender is defined. INFO level is higher than OFF, so it does not affect MapReduce logger. Root logger missing would affect all logs.

🚀 Application

advanced

Visualizing Hadoop Log Error Trends

You have extracted daily counts of ERROR logs from Hadoop over 7 days:

errors = [5, 7, 3, 8, 6, 10, 4]

Which Python code using matplotlib will correctly plot these error counts as a line chart with days on the x-axis labeled from 1 to 7?

Hadoop

import matplotlib.pyplot as plt
errors = [5, 7, 3, 8, 6, 10, 4]

A

plt.plot(range(1, 8), errors)
plt.xlabel('Day')
plt.ylabel('Error Count')
plt.title('Hadoop Error Trends')
plt.show()

B

plt.plot(errors)
plt.xlabel('Error Count')
plt.ylabel('Day')
plt.title('Hadoop Error Trends')
plt.show()

C

plt.bar(range(7), errors)
plt.xlabel('Day')
plt.ylabel('Error Count')
plt.title('Hadoop Error Trends')
plt.show()

D

plt.scatter(range(1, 8), errors)
plt.xlabel('Error Count')
plt.ylabel('Day')
plt.title('Hadoop Error Trends')
plt.show()

💡 Hint

Check axis labels and plot type for a line chart with days on x-axis.

Option A correctly plots a line chart with days 1 to 7 on x-axis and error counts on y-axis. Option A plots a bar chart, not line. Option A swaps axes labels incorrectly. Option A uses scatter plot with swapped labels.

🧠 Conceptual

expert

Root Cause Analysis Using Hadoop Logs

During a Hadoop job failure, the logs show repeated java.net.ConnectException: Connection refused errors from DataNode to NameNode. Which is the most likely root cause?

A NameNode service is down or unreachable from DataNode

B DataNode has disk failure causing connection errors

C Network bandwidth is saturated causing slow connections

D MapReduce job configuration has incorrect memory settings

💡 Hint

Think about what 'Connection refused' means in network communication.

'Connection refused' means the target service is not accepting connections, likely because NameNode is down or unreachable. Disk failure or bandwidth issues cause different errors. Memory settings affect job execution but not connection refusals.

Practice

1. What is the main purpose of Hadoop logs in managing a cluster?

easy

A. To speed up data processing

B. To show what happens inside Hadoop and help find problems

C. To store user data permanently

D. To backup Hadoop configuration files

Solution

1. Step 1: Understand the role of logs

   Logs record events and actions inside Hadoop components.

2. Step 2: Identify the purpose of logs in troubleshooting

   Logs help find and fix problems by showing errors and system behavior.

3. Final Answer:

   To show what happens inside Hadoop and help find problems -> Option B

4. Quick Check:

   Logs = Problem detection [OK]

Hint: Logs help spot problems by showing system events [OK]

Common Mistakes:

• Confusing logs with data storage
• Thinking logs speed up processing
• Assuming logs backup configs

2. Which command is used to view logs of a YARN application in Hadoop?

easy

A. yarn logs -applicationId <app_id>

B. hdfs dfs -ls

C. hadoop fs -put

D. yarn application -status

Solution

1. Step 1: Identify commands related to logs

   Only yarn logs fetches logs for YARN apps.

2. Step 2: Match command with correct syntax

   yarn logs -applicationId <app_id> shows logs for the given app.

3. Final Answer:

   yarn logs -applicationId <app_id> -> Option A

4. Quick Check:

   YARN logs command = yarn logs -applicationId [OK]

Hint: Use 'yarn logs -applicationId' to get app logs fast [OK]

Common Mistakes:

• Using 'hdfs dfs -ls' which lists files, not logs
• Confusing 'hadoop fs -put' with log commands
• Using 'yarn application -status' which shows status, not logs

3. Given the command hdfs dfs -cat /logs/app1.log | grep ERROR, what is the expected output?

medium

A. A list of files in /logs directory

B. All lines from app1.log

C. Only lines without the word 'ERROR'

D. Only lines containing the word 'ERROR' from app1.log

Solution

1. Step 1: Understand the command parts

   hdfs dfs -cat /logs/app1.log outputs the file content; grep ERROR filters lines with 'ERROR'.

2. Step 2: Combine command effects

   The pipe sends file content to grep, which shows only lines containing 'ERROR'.

3. Final Answer:

   Only lines containing the word 'ERROR' from app1.log -> Option D

4. Quick Check:

   grep ERROR filters lines with ERROR [OK]

Hint: Pipe file content to grep ERROR to find error lines [OK]

Common Mistakes:

• Thinking grep removes ERROR lines
• Expecting full file output
• Confusing grep with listing files

4. You run yarn logs -applicationId app_12345 but get an error saying 'Application not found'. What is the likely cause?

medium

A. The logs directory is empty

B. The command syntax is wrong

C. The application ID is incorrect or does not exist

D. YARN is not installed on the cluster

Solution

1. Step 1: Analyze the error message

   'Application not found' means the given app ID is not recognized by YARN.

2. Step 2: Check command correctness

   The command syntax is correct; error is about app ID validity.

3. Final Answer:

   The application ID is incorrect or does not exist -> Option C

4. Quick Check:

   Invalid app ID = Application not found error [OK]

Hint: Check app ID spelling if 'Application not found' error appears [OK]

Common Mistakes:

• Assuming command syntax is wrong
• Thinking YARN is missing
• Believing logs directory emptiness causes this error

5. You want to quickly find all error messages in Hadoop logs stored in HDFS at /var/log/hadoop. Which command sequence will help you achieve this efficiently?

hard

A. hdfs dfs -cat /var/log/hadoop/* | grep ERROR

B. yarn logs -applicationId * | grep ERROR

C. hdfs dfs -ls /var/log/hadoop | grep ERROR

D. hadoop fs -copyToLocal /var/log/hadoop /tmp && grep ERROR /tmp/*

Solution

1. Step 1: Identify command to read all logs in HDFS directory

   hdfs dfs -cat /var/log/hadoop/* outputs all files in that directory.

2. Step 2: Filter output for error lines

   Piping to grep ERROR shows only lines with 'ERROR'.

3. Final Answer:

   hdfs dfs -cat /var/log/hadoop/* | grep ERROR -> Option A

4. Quick Check:

   Cat all logs + grep ERROR = quick error search [OK]

Hint: Cat all logs then grep ERROR to find errors fast [OK]

Common Mistakes:

• Using yarn logs without specifying app ID
• Using ls which lists files, not content
• Copying logs locally is slower and unnecessary