User-defined functions (UDFs) in Hadoop

Introduction

User-defined functions (UDFs) let you add your own special commands to process data in Hadoop. They help you do tasks that built-in commands cannot do easily.

When you want to clean or change data in a way Hadoop does not support by default.

When you need to calculate new values from existing data columns.

When you want to filter data using your own rules.

When you want to reuse a custom operation many times in your data processing.

When built-in functions are too slow or not flexible enough for your task.

Syntax

Hadoop

import org.apache.hadoop.hive.ql.exec.UDF;

public class MyCustomUDF extends UDF {
    public String evaluate(String input) {
        if (input == null) return null;
        // Your custom logic here
        return input.toUpperCase();
    }
}

The class must extend UDF from Hadoop Hive.

The method evaluate is where you write your code. It can have different input types and return types.

Examples

This UDF converts a string to uppercase. If the input is null, it returns null.

Hadoop

import org.apache.hadoop.hive.ql.exec.UDF;

public class ToUpperCaseUDF extends UDF {
    public String evaluate(String input) {
        if (input == null) return null;
        return input.toUpperCase();
    }
}

This UDF adds a prefix to the input string. It handles null inputs safely.

Hadoop

import org.apache.hadoop.hive.ql.exec.UDF;

public class AddPrefixUDF extends UDF {
    public String evaluate(String input) {
        if (input == null) return null;
        return "prefix_" + input;
    }
}

This UDF divides two numbers but returns null if the denominator is zero or null to avoid errors.

Hadoop

import org.apache.hadoop.hive.ql.exec.UDF;

public class SafeDivideUDF extends UDF {
    public Double evaluate(Double numerator, Double denominator) {
        if (denominator == null || denominator == 0) return null;
        return numerator / denominator;
    }
}

This UDF shows how to handle null input by returning a default string.

Hadoop

import org.apache.hadoop.hive.ql.exec.UDF;

public class NullInputUDF extends UDF {
    public String evaluate(String input) {
        if (input == null) return "empty";
        return input;
    }
}

Sample Program

This program defines a UDF that reverses a string. It then tests the UDF with a normal string and a null input, printing the results.

Hadoop

import org.apache.hadoop.hive.ql.exec.UDF;

public class ReverseStringUDF extends UDF {
    public String evaluate(String input) {
        if (input == null) return null;
        return new StringBuilder(input).reverse().toString();
    }
}

// Sample usage in Hive:
// CREATE TEMPORARY FUNCTION reverse_string AS 'ReverseStringUDF';
// SELECT reverse_string(name) FROM users;

class TestReverseStringUDF {
    public static void main(String[] args) {
        ReverseStringUDF reverseStringUDF = new ReverseStringUDF();

        String original = "hadoop";
        System.out.println("Original: " + original);

        String reversed = reverseStringUDF.evaluate(original);
        System.out.println("Reversed: " + reversed);

        String nullInput = null;
        System.out.println("Null input reversed: " + reverseStringUDF.evaluate(nullInput));
    }
}

Important Notes

Time complexity is usually O(n) where n is the input size, depending on your logic.

Space complexity depends on what you store; simple UDFs use little extra space.

Common mistake: forgetting to handle null inputs, which can cause errors.

Use UDFs when built-in functions do not meet your needs or for reusable custom logic.

Summary

User-defined functions let you add your own data processing steps in Hadoop.

They must extend the UDF class and implement an evaluate method.

Always handle null inputs and test your UDF with different cases.

Practice

1. What is the main purpose of a User-defined Function (UDF) in Hadoop?

easy

A. To add custom data processing logic not available in built-in functions

B. To store data in Hadoop Distributed File System (HDFS)

C. To manage cluster resources automatically

D. To monitor network traffic between nodes

Solution

1. Step 1: Understand what UDFs do in Hadoop

   UDFs allow users to write their own functions to process data beyond what built-in functions offer.

2. Step 2: Compare options with UDF purpose

   Only To add custom data processing logic not available in built-in functions describes adding custom data processing logic, which matches UDF use.

3. Final Answer:

   To add custom data processing logic not available in built-in functions -> Option A

4. Quick Check:

   UDF purpose = custom processing [OK]

Hint: UDFs add your own data steps in Hadoop [OK]

Common Mistakes:

• Confusing UDFs with storage or resource management
• Thinking UDFs handle system monitoring
• Assuming UDFs replace built-in functions entirely

2. Which method must you implement when creating a Hadoop UDF class?

easy

A. initialize()

B. execute()

C. evaluate()

D. process()

Solution

1. Step 1: Recall UDF class requirements

   Hadoop UDFs must extend the UDF class and implement the evaluate method to process input.

2. Step 2: Match method names

   Only evaluate() is the required method for UDF logic; others are incorrect or do not exist.

3. Final Answer:

   evaluate() -> Option C

4. Quick Check:

   UDF method = evaluate() [OK]

Hint: UDFs always implement evaluate() method [OK]

Common Mistakes:

• Using initialize() or execute() instead of evaluate()
• Forgetting to override evaluate()
• Confusing with other Hadoop APIs

3. Given this UDF code snippet, what will be the output when input is null?

public class MyUDF extends UDF {
  public String evaluate(String input) {
    if (input == null) return "empty";
    return input.toUpperCase();
  }
}

medium

A. null

B. "empty"

C. "EMPTY"

D. Runtime error

Solution

1. Step 1: Analyze input null check in evaluate()

   The code checks if input is null and returns the string "empty" exactly as is.

2. Step 2: Determine output for null input

   Since input is null, the method returns "empty" (lowercase), not uppercase or error.

3. Final Answer:

   "empty" -> Option B

4. Quick Check:

   null input returns "empty" [OK]

Hint: Check null input handling in evaluate() [OK]

Common Mistakes:

• Assuming output is uppercase "EMPTY"
• Expecting null to be returned
• Thinking code throws an error on null

4. Identify the error in this UDF code:

public class SampleUDF extends UDF {
  public Integer evaluate(Integer num) {
    return num + 10;
  }
}

medium

A. Method name should be process, not evaluate

B. evaluate method must return String, not Integer

C. Class must implement interface, not extend UDF

D. Missing null check for input parameter

Solution

1. Step 1: Review evaluate method logic

   The method adds 10 to input num but does not check if num is null, which can cause NullPointerException.

2. Step 2: Validate other options

   Returning Integer is allowed; extending UDF is correct; method name evaluate is required.

3. Final Answer:

   Missing null check for input parameter -> Option D

4. Quick Check:

   Always check for null inputs in UDFs [OK]

Hint: Always check for null inputs in evaluate() [OK]

Common Mistakes:

• Ignoring null input checks
• Confusing return types for evaluate()
• Misunderstanding class inheritance rules

5. You want to create a UDF that returns the length of a string, but returns 0 if input is null or empty. Which evaluate method implementation is correct?

public Integer evaluate(String input) {
  // Choose the correct code
}

hard

A. if (input == null || input.isEmpty()) return 0; else return input.length();

B. return input.length();

C. if (input == null) return null; else return input.length();

D. return input == null ? 0 : input.length() + 1;

Solution

1. Step 1: Understand requirements for null or empty input

   The method must return 0 if input is null or empty string, otherwise return string length.

2. Step 2: Evaluate each option

   if (input == null || input.isEmpty()) return 0; else return input.length(); correctly checks null or empty and returns 0; others either miss empty check or return wrong values.

3. Final Answer:

   if (input == null || input.isEmpty()) return 0; else return input.length(); -> Option A

4. Quick Check:

   Null or empty returns 0, else length [OK]

Hint: Check both null and empty before length [OK]

Common Mistakes:

• Not checking for empty string
• Returning null instead of 0
• Adding extra 1 to length incorrectly