Input splits and data locality in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Input Splits in Hadoop

In Hadoop MapReduce, what is the primary purpose of input splits?

A To store the output data after processing

B To compress data before sending it to reducers

C To divide the input data into manageable chunks for parallel processing

D To replicate data across nodes for fault tolerance

💡 Hint

Think about how Hadoop processes large datasets efficiently.

Input splits break the input data into smaller pieces so that each mapper can process a chunk independently, enabling parallelism.

🧠 Conceptual

intermediate

Data Locality Importance

Why is data locality important in Hadoop's processing model?

A It ensures data is encrypted during processing

B It reduces network traffic by processing data on the node where it is stored

C It balances the load evenly across all nodes regardless of data location

D It compresses data to save storage space

💡 Hint

Consider how moving computation close to data affects performance.

Data locality reduces the need to transfer large data over the network, improving speed and efficiency.

❓ data_output

advanced

Input Split Sizes Calculation

Given a 640 MB file stored in HDFS with a block size of 128 MB, how many input splits will Hadoop create by default?

A 5

B 4

C 6

D 8

💡 Hint

Divide the total file size by the block size.

640 MB / 128 MB = 5 splits, each corresponding to one block.

❓ Predict Output

advanced

Effect of Custom InputFormat on Splits

Consider this Hadoop MapReduce job snippet using a custom InputFormat that combines small files into larger splits:

job.setInputFormatClass(CombineFileInputFormat.class);

If the input directory contains 10 files of 10 MB each and the block size is 128 MB, how many input splits will be created?

A 5

B 10

C 0

D 1

💡 Hint

CombineFileInputFormat merges small files into splits up to block size.

Since total size is 100 MB, less than 128 MB block size, CombineFileInputFormat creates one split.

🚀 Application

expert

Optimizing Data Locality in a Hadoop Cluster

You have a Hadoop cluster with 5 nodes. A large dataset is stored unevenly: 80% on node 1, and the rest spread across nodes 2-5. You want to maximize data locality for a MapReduce job. Which strategy will best improve data locality?

A Distribute the dataset evenly across all nodes before running the job

B Run all mappers only on node 1 to process the majority of data

C Increase the number of reducers to balance the load

D Compress the dataset to reduce its size

💡 Hint

Think about how data distribution affects task assignment.

Evenly distributing data allows mappers to run on multiple nodes close to their data, improving locality and parallelism.