User-defined functions (UDFs) in Hadoop - Step-by-Step Execution

Concept Flow - User-defined functions (UDFs)

Define UDF class

↓

Implement evaluate() method

↓

Compile and add UDF to Hadoop

↓

Use UDF in Hive query

↓

Hive calls evaluate() for each row

↓

Return processed result

↓

Query outputs transformed data

This flow shows how you create a UDF by defining a class with an evaluate method, compile it, use it in Hive, and get transformed results.

Execution Sample

Hadoop

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

public class MyUpper extends UDF {
  public Text evaluate(Text input) {
    if (input == null) return null;
    return new Text(input.toString().toUpperCase());
  }
}

This UDF converts input text to uppercase when called from Hive.

Execution Table

Step	Action	Input	evaluate() Output	Hive Query Output
1	Call evaluate() with 'hello'	'hello'	'HELLO'	'HELLO'
2	Call evaluate() with 'world'	'world'	'WORLD'	'WORLD'
3	Call evaluate() with null	null	null	null
4	No more rows	-	-	-

💡 All input rows processed, UDF calls complete.

Variable Tracker

Variable	Start	After 1	After 2	After 3	Final
input	null	'hello'	'world'	null	null
output	null	'HELLO'	'WORLD'	null	null

Key Moments - 3 Insights

Why does evaluate() return null when input is null?

Because the code checks if input is null and returns null immediately (see execution_table step 3), avoiding errors.

How does Hive use the UDF for each row?

Hive calls the evaluate() method once per row with that row's input (shown in execution_table steps 1-3).

What happens if you forget to compile and add the UDF to Hive?

Hive will not recognize the UDF and will give an error when you try to use it in queries.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output of evaluate() when input is 'world'?

A 'world'

B 'WORLD'

C null

D 'World'

💡 Hint

Check execution_table row 2 under 'evaluate() Output'

At which step does evaluate() receive a null input?

A Step 3

B Step 2

C Step 1

D Step 4

💡 Hint

Look at execution_table row 3 under 'Input'

If you remove the null check in evaluate(), what will happen when input is null?

A It will return null safely

B It will return empty string

C It will throw an error

D It will convert null to 'NULL'

💡 Hint

Refer to key_moments about null input handling and execution_table step 3

Concept Snapshot

User-defined functions (UDFs) in Hadoop allow custom processing in Hive queries.
Define a Java class extending UDF with an evaluate() method.
evaluate() processes one input row and returns a result.
Compile and add the UDF jar to Hive before use.
Hive calls evaluate() for each row during query execution.
Always handle null inputs to avoid errors.

Full Transcript

User-defined functions (UDFs) let you add your own code to process data in Hive queries. You write a Java class that extends UDF and implement an evaluate() method. This method takes input from each row and returns a result. After compiling your UDF into a jar file and adding it to Hive, you can call it in your queries. Hive runs evaluate() on every row, transforming data as you want. It's important to check for null inputs in evaluate() to prevent errors. The execution table shows how evaluate() is called with different inputs and returns uppercase strings or null safely.

Practice

1. What is the main purpose of a User-defined Function (UDF) in Hadoop?

easy

A. To add custom data processing logic not available in built-in functions

B. To store data in Hadoop Distributed File System (HDFS)

C. To manage cluster resources automatically

D. To monitor network traffic between nodes

Solution

1. Step 1: Understand what UDFs do in Hadoop

   UDFs allow users to write their own functions to process data beyond what built-in functions offer.

2. Step 2: Compare options with UDF purpose

   Only To add custom data processing logic not available in built-in functions describes adding custom data processing logic, which matches UDF use.

3. Final Answer:

   To add custom data processing logic not available in built-in functions -> Option A

4. Quick Check:

   UDF purpose = custom processing [OK]

Hint: UDFs add your own data steps in Hadoop [OK]

Common Mistakes:

• Confusing UDFs with storage or resource management
• Thinking UDFs handle system monitoring
• Assuming UDFs replace built-in functions entirely

2. Which method must you implement when creating a Hadoop UDF class?

easy

A. initialize()

B. execute()

C. evaluate()

D. process()

Solution

1. Step 1: Recall UDF class requirements

   Hadoop UDFs must extend the UDF class and implement the evaluate method to process input.

2. Step 2: Match method names

   Only evaluate() is the required method for UDF logic; others are incorrect or do not exist.

3. Final Answer:

   evaluate() -> Option C

4. Quick Check:

   UDF method = evaluate() [OK]

Hint: UDFs always implement evaluate() method [OK]

Common Mistakes:

• Using initialize() or execute() instead of evaluate()
• Forgetting to override evaluate()
• Confusing with other Hadoop APIs

3. Given this UDF code snippet, what will be the output when input is null?

public class MyUDF extends UDF {
  public String evaluate(String input) {
    if (input == null) return "empty";
    return input.toUpperCase();
  }
}

medium

A. null

B. "empty"

C. "EMPTY"

D. Runtime error

Solution

1. Step 1: Analyze input null check in evaluate()

   The code checks if input is null and returns the string "empty" exactly as is.

2. Step 2: Determine output for null input

   Since input is null, the method returns "empty" (lowercase), not uppercase or error.

3. Final Answer:

   "empty" -> Option B

4. Quick Check:

   null input returns "empty" [OK]

Hint: Check null input handling in evaluate() [OK]

Common Mistakes:

• Assuming output is uppercase "EMPTY"
• Expecting null to be returned
• Thinking code throws an error on null

4. Identify the error in this UDF code:

public class SampleUDF extends UDF {
  public Integer evaluate(Integer num) {
    return num + 10;
  }
}

medium

A. Method name should be process, not evaluate

B. evaluate method must return String, not Integer

C. Class must implement interface, not extend UDF

D. Missing null check for input parameter

Solution

1. Step 1: Review evaluate method logic

   The method adds 10 to input num but does not check if num is null, which can cause NullPointerException.

2. Step 2: Validate other options

   Returning Integer is allowed; extending UDF is correct; method name evaluate is required.

3. Final Answer:

   Missing null check for input parameter -> Option D

4. Quick Check:

   Always check for null inputs in UDFs [OK]

Hint: Always check for null inputs in evaluate() [OK]

Common Mistakes:

• Ignoring null input checks
• Confusing return types for evaluate()
• Misunderstanding class inheritance rules

5. You want to create a UDF that returns the length of a string, but returns 0 if input is null or empty. Which evaluate method implementation is correct?

public Integer evaluate(String input) {
  // Choose the correct code
}

hard

A. if (input == null || input.isEmpty()) return 0; else return input.length();

B. return input.length();

C. if (input == null) return null; else return input.length();

D. return input == null ? 0 : input.length() + 1;

Solution

1. Step 1: Understand requirements for null or empty input

   The method must return 0 if input is null or empty string, otherwise return string length.

2. Step 2: Evaluate each option

   if (input == null || input.isEmpty()) return 0; else return input.length(); correctly checks null or empty and returns 0; others either miss empty check or return wrong values.

3. Final Answer:

   if (input == null || input.isEmpty()) return 0; else return input.length(); -> Option A

4. Quick Check:

   Null or empty returns 0, else length [OK]

Hint: Check both null and empty before length [OK]

Common Mistakes:

• Not checking for empty string
• Returning null instead of 0
• Adding extra 1 to length incorrectly