UDFs (User Defined Functions) in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a simple UDF application

What is the output of this Spark code that uses a UDF to square numbers in a DataFrame?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

spark = SparkSession.builder.master('local').appName('UDF Test').getOrCreate()
data = [(1,), (2,), (3,)]
df = spark.createDataFrame(data, ['num'])

square_udf = udf(lambda x: x * x, IntegerType())
df2 = df.withColumn('square', square_udf(df['num']))
df2.show()

A

+---+------+
|num|square|
+---+------+
|  1|     1|
|  2|     4|
|  3|     9|
+---+------+

B TypeError: 'int' object is not callable

C SyntaxError: invalid syntax

D

+---+------+
|num|square|
+---+------+
|  1|     2|
|  2|     4|
|  3|     6|
+---+------+

💡 Hint

Think about what the lambda function does to each number.

The UDF squares each number in the 'num' column, so the 'square' column contains the squares: 1, 4, and 9.

❓ data_output

intermediate

Result of UDF with string manipulation

Given this Spark code with a UDF that reverses strings, what is the resulting DataFrame output?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

spark = SparkSession.builder.master('local').appName('UDF String').getOrCreate()
data = [('apple',), ('banana',), ('cherry',)]
df = spark.createDataFrame(data, ['fruit'])

reverse_udf = udf(lambda s: s[::-1], StringType())
df2 = df.withColumn('reversed', reverse_udf(df['fruit']))
df2.show()

A

+------+--------+
| fruit|reversed|
+------+--------+
| apple|  elppa|
|banana| ananab|
|cherry| yrrehc|
+------+--------+

B

+------+--------+
| fruit|reversed|
+------+--------+
| apple|  apple|
|banana| banana|
|cherry| cherry|
+------+--------+

C TypeError: 'NoneType' object is not subscriptable

D SyntaxError: invalid syntax

💡 Hint

The lambda reverses the string using slicing.

The UDF reverses each string in the 'fruit' column, so the 'reversed' column shows the reversed fruit names.

🔧 Debug

advanced

Identify the error in this UDF code

What error will this Spark code raise when trying to apply the UDF?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

spark = SparkSession.builder.master('local').appName('UDF Error').getOrCreate()
data = [(1,), (2,), (3,)]
df = spark.createDataFrame(data, ['num'])

def add_five(x):
    return x + 5

add_five_udf = udf(add_five)
df2 = df.withColumn('plus_five', add_five_udf(df['num']))
df2.show()

A SyntaxError: invalid syntax

B TypeError: add_five() missing 1 required positional argument: 'x'

C

+---+---------+
|num|plus_five|
+---+---------+
|  1|        6|
|  2|        7|
|  3|        8|
+---+---------+

D AnalysisException: UDF return type must be specified

💡 Hint

Check how the UDF is defined and registered.

When creating a UDF in Spark, you must specify the return type. Omitting it causes an AnalysisException.

🚀 Application

advanced

Using UDF to categorize ages

You want to add a new column 'age_group' to a Spark DataFrame with ages, categorizing ages as 'child' (<18), 'adult' (18-64), or 'senior' (65+). Which UDF definition and usage will produce the correct output?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

spark = SparkSession.builder.master('local').appName('Age Group').getOrCreate()
data = [(5,), (30,), (70,)]
df = spark.createDataFrame(data, ['age'])

A

age_group_udf = udf(lambda age: 'child' if age < 18 else 'adult' if age < 65 else 'senior', StringType())
df2 = df.withColumn('age_group', age_group_udf(df['age']))
df2.show()

B

age_group_udf = udf(lambda age: 'child' if age < 18 else 'adult' if age <= 64 else 'senior', StringType())
df2 = df.withColumn('age_group', age_group_udf(df['age']))
df2.show()

C

age_group_udf = udf(lambda age: 'child' if age <= 18 else 'adult' if age <= 65 else 'senior', StringType())
df2 = df.withColumn('age_group', age_group_udf(df['age']))
df2.show()

D

age_group_udf = udf(lambda age: 'child' if age <= 18 else 'adult' if age < 65 else 'senior', StringType())
df2 = df.withColumn('age_group', age_group_udf(df['age']))
df2.show()

💡 Hint

Check the age boundaries carefully for each group.

The correct categorization is: child if age < 18, adult if 18 <= age <= 64, senior if age >= 65. Option B matches this logic.

🧠 Conceptual

expert

Why avoid UDFs when possible in Spark?

Which is the main reason to avoid using UDFs in Spark when native functions exist?

A UDFs cannot handle null values, causing runtime exceptions.

B UDFs always cause syntax errors in Spark SQL queries.

C UDFs are slower because they break Spark's optimization and serialization, causing performance loss.

D UDFs automatically convert data to Pandas DataFrames, which is inefficient.

💡 Hint

Think about how Spark optimizes queries and executes code.

UDFs run outside Spark's optimized engine, preventing query optimization and causing slower execution.