Small files problem and solutions in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why does the small files problem occur in Hadoop?

In Hadoop, why is having many small files a problem?

A Because each file requires metadata storage in the NameNode, causing memory overload.

B Because small files are automatically merged by Hadoop, causing data loss.

C Because small files increase the block size, reducing storage efficiency.

D Because small files are compressed by default, slowing down processing.

💡 Hint

Think about how Hadoop manages file information in its master node.

Hadoop's NameNode stores metadata for each file. Many small files mean many metadata entries, which can overload the NameNode's memory.

❓ Predict Output

intermediate

Output of merging small files using Hadoop Archive (HAR)

What is the output of the following Hadoop command?

hadoop archive -archiveName smallfiles.har -p /user/hadoop/smallfiles /user/hadoop/harfiles

Assuming /user/hadoop/smallfiles contains 1000 small files.

A The small files remain unchanged; no archive is created.

B All small files are deleted and replaced by a single large file in /user/hadoop/harfiles.

C A HAR file named smallfiles.har is created in /user/hadoop/harfiles containing all small files.

D The command fails because HAR does not support more than 500 files.

💡 Hint

HAR archives combine files logically without deleting originals.

The Hadoop Archive (HAR) command creates a single archive file that logically groups many small files, improving NameNode efficiency without deleting originals.

❓ data_output

advanced

Result of using CombineFileInputFormat on small files

Given a Hadoop job configured with CombineFileInputFormat to process 1000 small files of 1MB each, what is the expected number of input splits?

A No input splits are created; job fails to start.

B Exactly 1000 input splits, one per file.

C One input split containing all 1000 files combined.

D Approximately 10 input splits, each combining about 100MB of data.

💡 Hint

CombineFileInputFormat groups small files into splits based on size limits.

CombineFileInputFormat merges small files into fewer splits to reduce overhead. If the split size is about 128MB, 1000 files of 1MB each will form about 10 splits.

🔧 Debug

advanced

Identify the cause of slow MapReduce job with many small files

A MapReduce job processing many small files runs very slowly. Which of the following is the most likely cause?

A The small files are compressed, causing decompression delays.

B Too many map tasks are created, causing overhead in task scheduling and startup.

C The job uses CombineFileInputFormat, which increases the number of splits.

D The NameNode is running out of disk space due to large files.

💡 Hint

Think about how many tasks are created for many small files.

Many small files cause many map tasks, increasing scheduling and startup overhead, which slows down the job.

🚀 Application

expert

Best approach to handle millions of small log files in Hadoop

You have millions of small log files generated daily. You want to optimize Hadoop processing and reduce NameNode memory usage. Which approach is best?

A Use a daily batch job to merge small files into larger sequence files before processing.

B Increase the NameNode heap size to handle more metadata entries.

C Process files as-is using default TextInputFormat to avoid data loss.

D Delete small files older than one day to reduce file count.

💡 Hint

Think about combining files logically before processing.

Merging small files into larger sequence files reduces the number of files and metadata entries, improving NameNode efficiency and processing speed.