User-defined functions (UDFs) in Hadoop - Mini Project: Build & Apply

User-defined functions (UDFs) in Hadoop MapReduce

📖 Scenario: You work with a large text dataset stored in Hadoop. You want to count how many times each word appears. To do this, you will write a simple User-defined Function (UDF) in Hadoop MapReduce that processes the text data.

🎯 Goal: Build a Hadoop MapReduce program with a user-defined mapper function that splits lines into words and counts each word's occurrences.

📋 What You'll Learn

Create a mapper function that splits input lines into words

Create a reducer function that sums counts for each word

Use the Hadoop MapReduce framework to run the job

Print the final word counts

💡 Why This Matters

🌍 Real World

Counting word frequencies is a common task in analyzing large text data like logs, documents, or social media posts using Hadoop.

💼 Career

Understanding how to write user-defined functions in Hadoop MapReduce is essential for data engineers and data scientists working with big data.

1

Create the input data list

Create a list called input_data with these exact three strings: 'hello world', 'hello hadoop', and 'hello mapreduce'.

Hadoop

# Create the input_data list with the three exact strings
# Your code here

Hint

Use square brackets to create a list and put each string inside quotes separated by commas.

2

Define the mapper function

Define a function called mapper that takes a single argument line. Inside, split line into words using line.split() and return the list of words.

Hadoop

input_data = ['hello world', 'hello hadoop', 'hello mapreduce']

# Define the mapper function below
# Your code here

Hint

Use the split() method on the input string to get words.

3

Apply the mapper and count words

Create an empty dictionary called word_counts. Use a for loop with variable line to iterate over input_data. Inside the loop, call mapper(line) to get words. Then use another for loop with variable word to iterate over the words. For each word, add 1 to its count in word_counts (initialize to 0 if not present).

Hadoop

input_data = ['hello world', 'hello hadoop', 'hello mapreduce']

def mapper(line):
    words = line.split()
    return words

# Create word_counts dictionary and count words using mapper
# Your code here

Hint

Use dict.get(key, 0) to handle missing keys when counting.

4

Print the final word counts

Write a print statement to display the word_counts dictionary.

Hadoop

input_data = ['hello world', 'hello hadoop', 'hello mapreduce']

def mapper(line):
    words = line.split()
    return words

word_counts = {}
for line in input_data:
    words = mapper(line)
    for word in words:
        word_counts[word] = word_counts.get(word, 0) + 1

# Print the word_counts dictionary
# Your code here

Hint

The printed dictionary should show the count of each word.

Practice

1. What is the main purpose of a User-defined Function (UDF) in Hadoop?

easy

A. To add custom data processing logic not available in built-in functions

B. To store data in Hadoop Distributed File System (HDFS)

C. To manage cluster resources automatically

D. To monitor network traffic between nodes

Solution

1. Step 1: Understand what UDFs do in Hadoop

   UDFs allow users to write their own functions to process data beyond what built-in functions offer.

2. Step 2: Compare options with UDF purpose

   Only To add custom data processing logic not available in built-in functions describes adding custom data processing logic, which matches UDF use.

3. Final Answer:

   To add custom data processing logic not available in built-in functions -> Option A

4. Quick Check:

   UDF purpose = custom processing [OK]

Hint: UDFs add your own data steps in Hadoop [OK]

Common Mistakes:

• Confusing UDFs with storage or resource management
• Thinking UDFs handle system monitoring
• Assuming UDFs replace built-in functions entirely

2. Which method must you implement when creating a Hadoop UDF class?

easy

A. initialize()

B. execute()

C. evaluate()

D. process()

Solution

1. Step 1: Recall UDF class requirements

   Hadoop UDFs must extend the UDF class and implement the evaluate method to process input.

2. Step 2: Match method names

   Only evaluate() is the required method for UDF logic; others are incorrect or do not exist.

3. Final Answer:

   evaluate() -> Option C

4. Quick Check:

   UDF method = evaluate() [OK]

Hint: UDFs always implement evaluate() method [OK]

Common Mistakes:

• Using initialize() or execute() instead of evaluate()
• Forgetting to override evaluate()
• Confusing with other Hadoop APIs

3. Given this UDF code snippet, what will be the output when input is null?

public class MyUDF extends UDF {
  public String evaluate(String input) {
    if (input == null) return "empty";
    return input.toUpperCase();
  }
}

medium

A. null

B. "empty"

C. "EMPTY"

D. Runtime error

Solution

1. Step 1: Analyze input null check in evaluate()

   The code checks if input is null and returns the string "empty" exactly as is.

2. Step 2: Determine output for null input

   Since input is null, the method returns "empty" (lowercase), not uppercase or error.

3. Final Answer:

   "empty" -> Option B

4. Quick Check:

   null input returns "empty" [OK]

Hint: Check null input handling in evaluate() [OK]

Common Mistakes:

• Assuming output is uppercase "EMPTY"
• Expecting null to be returned
• Thinking code throws an error on null

4. Identify the error in this UDF code:

public class SampleUDF extends UDF {
  public Integer evaluate(Integer num) {
    return num + 10;
  }
}

medium

A. Method name should be process, not evaluate

B. evaluate method must return String, not Integer

C. Class must implement interface, not extend UDF

D. Missing null check for input parameter

Solution

1. Step 1: Review evaluate method logic

   The method adds 10 to input num but does not check if num is null, which can cause NullPointerException.

2. Step 2: Validate other options

   Returning Integer is allowed; extending UDF is correct; method name evaluate is required.

3. Final Answer:

   Missing null check for input parameter -> Option D

4. Quick Check:

   Always check for null inputs in UDFs [OK]

Hint: Always check for null inputs in evaluate() [OK]

Common Mistakes:

• Ignoring null input checks
• Confusing return types for evaluate()
• Misunderstanding class inheritance rules

5. You want to create a UDF that returns the length of a string, but returns 0 if input is null or empty. Which evaluate method implementation is correct?

public Integer evaluate(String input) {
  // Choose the correct code
}

hard

A. if (input == null || input.isEmpty()) return 0; else return input.length();

B. return input.length();

C. if (input == null) return null; else return input.length();

D. return input == null ? 0 : input.length() + 1;

Solution

1. Step 1: Understand requirements for null or empty input

   The method must return 0 if input is null or empty string, otherwise return string length.

2. Step 2: Evaluate each option

   if (input == null || input.isEmpty()) return 0; else return input.length(); correctly checks null or empty and returns 0; others either miss empty check or return wrong values.

3. Final Answer:

   if (input == null || input.isEmpty()) return 0; else return input.length(); -> Option A

4. Quick Check:

   Null or empty returns 0, else length [OK]

Hint: Check both null and empty before length [OK]

Common Mistakes:

• Not checking for empty string
• Returning null instead of 0
• Adding extra 1 to length incorrectly