UDFs (User Defined Functions) in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a UDF in Apache Spark?

A UDF (User Defined Function) is a custom function that you write to perform operations on data in Spark DataFrames when built-in functions are not enough.

beginner

How do you register a Python function as a UDF in Spark?

You use the function spark.udf.register or pyspark.sql.functions.udf to convert a Python function into a Spark UDF that can be used in DataFrame operations.

intermediate

Why should you avoid using UDFs when possible in Spark?

UDFs can be slower because they break Spark's optimization and run code outside of Spark's engine. Using built-in Spark functions is faster and better optimized.

intermediate

What data types must you specify when creating a UDF in Spark?

You must specify the return data type of the UDF, like StringType, IntegerType, etc., so Spark knows how to handle the output.

beginner

Example: How to create a UDF that adds 10 to a number in PySpark?

Define a Python function: def add_ten(x): return x + 10. Then create UDF: from pyspark.sql.functions import udf; from pyspark.sql.types import IntegerType; add_ten_udf = udf(add_ten, IntegerType()). Use it in DataFrame with df.withColumn('new_col', add_ten_udf(df['col'])).

What does UDF stand for in Apache Spark?

A Unified Data Frame

B User Defined Function

C Universal Data Format

D User Data File

UDF means User Defined Function, which is a custom function created by the user.

Which Spark module is used to create UDFs in PySpark?

A pyspark.sql.functions

B pyspark.ml

C pyspark.streaming

D pyspark.sql.types

The pyspark.sql.functions module contains the udf function to create UDFs.

Why might using a UDF slow down your Spark job?

A Because UDFs run outside Spark's optimized engine

B Because UDFs use too much memory

C Because UDFs require internet connection

D Because UDFs only work on small datasets

UDFs run outside Spark's optimized engine, which can reduce performance.

What must you specify when defining a UDF in Spark?

A The file path

B The input data type

C The return data type

D The Spark version

You must specify the return data type so Spark knows how to handle the UDF output.

Which of these is a correct way to apply a UDF to a DataFrame column?

A df.groupBy(my_udf)

B df.select(my_udf)

C df.filter(my_udf)

D df.withColumn('new_col', my_udf(df['col']))

You apply a UDF to a column using withColumn and passing the UDF applied to the column.

Explain what a UDF is in Apache Spark and when you might need to use one.

Think about extending Spark's capabilities with your own code.

Describe the steps to create and use a UDF in PySpark with a simple example.

Remember the example of adding 10 to a number.

Practice

1. What is the main purpose of a User Defined Function (UDF) in Apache Spark?

easy

A. To create new Spark sessions

B. To manage cluster resources

C. To optimize Spark SQL queries automatically

D. To apply custom logic to each row of a DataFrame

Solution

1. Step 1: Understand UDF role in Spark

   A UDF lets you add your own logic to process each row in a DataFrame, beyond built-in functions.

2. Step 2: Compare options

   The other options relate to Spark setup or optimization, not custom row logic.

3. Final Answer:

   To apply custom logic to each row of a DataFrame -> Option D

4. Quick Check:

   UDF = custom row logic [OK]

Hint: UDFs add your own row-wise logic in Spark [OK]

Common Mistakes:

• Confusing UDFs with Spark session management
• Thinking UDFs optimize queries automatically
• Assuming UDFs manage cluster resources

2. Which of the following is the correct way to register a UDF in PySpark with a return type of IntegerType?

easy

A. spark.udf.register('my_udf', func, 'IntegerType')

B. spark.udf.register('my_udf', func)

C. spark.udf.register('my_udf', func, IntegerType())

D. spark.udf.create('my_udf', func, IntegerType())

Solution

1. Step 1: Recall UDF registration syntax

   The correct method is spark.udf.register with function name, function, and return type as a DataType object.

2. Step 2: Evaluate options

   spark.udf.register('my_udf', func, IntegerType()) uses IntegerType() correctly. spark.udf.register('my_udf', func) misses return type. spark.udf.create('my_udf', func, IntegerType()) uses a non-existent method. spark.udf.register('my_udf', func, 'int') uses a string instead of IntegerType().

3. Final Answer:

   spark.udf.register('my_udf', func, IntegerType()) -> Option C

4. Quick Check:

   Register UDF with return type object [OK]

Hint: Always specify return type as DataType object when registering UDF [OK]

Common Mistakes:

• Omitting the return type parameter
• Using incorrect method names like create
• Passing return type as string instead of DataType

3. Given the following PySpark code, what will be the output of df.select('name', 'age_plus_one').show()?

from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

def add_one(age):
    return age + 1

add_one_udf = udf(add_one, IntegerType())
df = spark.createDataFrame([('Alice', 30), ('Bob', 25)], ['name', 'age'])
df = df.withColumn('age_plus_one', add_one_udf('age'))

medium

A. [[Alice, 31], [Bob, 26]]

B. [[Alice, 30], [Bob, 25]]

C. Error: UDF missing return type

D. [[Alice, 1], [Bob, 1]]

Solution

1. Step 1: Understand UDF logic

   The UDF adds 1 to the 'age' column values: 30 + 1 = 31, 25 + 1 = 26.

2. Step 2: Check DataFrame transformation

   The new column 'age_plus_one' will have values 31 and 26 for Alice and Bob respectively.

3. Final Answer:

   [[Alice, 31], [Bob, 26]] -> Option A

4. Quick Check:

   age + 1 = 31, 26 [OK]

Hint: UDF applies function row-wise, transforming each value [OK]

Common Mistakes:

• Forgetting to specify return type causing errors
• Assuming original age remains unchanged
• Confusing UDF output with input data

4. Identify the error in the following PySpark UDF code snippet:

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

def greet(name):
    return 'Hello ' + name

greet_udf = udf(greet)
df = spark.createDataFrame([('John',), ('Jane',)], ['name'])
df = df.withColumn('greeting', greet_udf('name'))
df.show()

medium

A. UDF missing return type specification

B. Function greet has wrong parameter type

C. DataFrame creation syntax is incorrect

D. withColumn method is used incorrectly

Solution

1. Step 1: Check UDF registration

   The UDF is created without specifying the return type, which is required.

2. Step 2: Verify other code parts

   Function parameter and DataFrame creation are correct. withColumn usage is proper.

3. Final Answer:

   UDF missing return type specification -> Option A

4. Quick Check:

   Always specify return type when creating UDF [OK]

Hint: Always specify return type when defining UDF [OK]

Common Mistakes:

• Omitting return type causes runtime errors
• Assuming return type is optional
• Misunderstanding function parameter types

5. You want to create a UDF that returns the length of a string column but safely handles null values by returning 0 instead of failing. Which of the following UDF definitions correctly implements this behavior?

hard

A. def safe_length(s):
return len(s)
safe_length_udf = udf(safe_length, IntegerType())

B. def safe_length(s):
if s is None:
return 0
else:
return len(s)
safe_length_udf = udf(safe_length, IntegerType())

C. def safe_length(s):
if s == '':
return 0
return len(s)
safe_length_udf = udf(safe_length, StringType())

D. def safe_length(s):
return s.length()
safe_length_udf = udf(safe_length, IntegerType())

Solution

1. Step 1: Handle null values safely

   def safe_length(s): if s is None: return 0 else: return len(s) safe_length_udf = udf(safe_length, IntegerType()) checks if input is None and returns 0, preventing errors.

2. Step 2: Check return type and logic

   def safe_length(s): if s is None: return 0 else: return len(s) safe_length_udf = udf(safe_length, IntegerType()) returns IntegerType and uses len(s) correctly. Others either miss null check or use wrong return type or method.

3. Final Answer:

   def safe_length(s): if s is None: return 0 else: return len(s) safe_length_udf = udf(safe_length, IntegerType()) -> Option B

4. Quick Check:

   Check nulls first, then return length [OK]

Hint: Check for None before calling len() in UDF [OK]

Common Mistakes:

• Not handling None causing runtime errors
• Returning wrong data type from UDF
• Using string methods incorrectly in Python