Log management and troubleshooting in Hadoop

Introduction

Logs help us see what is happening inside Hadoop. They show errors and important events. This helps fix problems fast.

When a Hadoop job fails and you want to find out why.

When the system is slow and you want to check what caused it.

When you want to monitor Hadoop cluster health over time.

When you need to audit who accessed data and when.

When debugging configuration or permission issues.

Syntax

Hadoop

yarn logs -applicationId <app_id>
# or
hdfs dfs -cat /path/to/logfile
# or
yarn logs -applicationId <app_id>

Use yarn logs to get logs for a specific application in YARN.

Log files are usually stored in HDFS or local node directories.

Examples

Fetch logs for a specific YARN application by its ID.

Hadoop

yarn logs -applicationId application_1680000000000_0001

View a Hadoop job log stored in HDFS.

Hadoop

hdfs dfs -cat /user/hadoop/logs/job_1680000000000_0001.log

Watch the NameNode log file live on a Hadoop node.

Hadoop

tail -f /var/log/hadoop/hadoop-hdfs-namenode.log

Sample Program

This code fetches logs for a Hadoop YARN application using its ID. It then counts how many lines contain the word 'ERROR' to help find problems quickly.

Hadoop

# This example shows how to fetch and analyze logs for a failed Hadoop job
import subprocess

# Replace with your actual application ID
app_id = 'application_1680000000000_0001'

# Run yarn logs command to get logs
result = subprocess.run(['yarn', 'logs', '-applicationId', app_id], capture_output=True, text=True)

logs = result.stdout

# Simple analysis: count error lines
error_lines = [line for line in logs.splitlines() if 'ERROR' in line]

print(f'Total log lines: {len(logs.splitlines())}')
print(f'Error lines found: {len(error_lines)}')

# Show first 5 error lines
print('\nFirst 5 error lines:')
for line in error_lines[:5]:
    print(line)

Important Notes

Logs can be very large; use filters or search tools to find relevant parts.

Check both stdout and stderr logs for full error details.

Keep Hadoop log levels configurable to control verbosity.

Summary

Logs show what happens inside Hadoop and help find problems.

Use yarn logs or hdfs dfs -cat to access logs.

Look for 'ERROR' lines to quickly spot issues.