String functions in Spark in Apache Spark - Step-by-Step Execution

Concept Flow - String functions in Spark

Start with DataFrame

↓

Select string column

↓

Apply string function

↓

New column with result

↓

Show or use result

The flow shows how Spark applies string functions to a DataFrame column, creating a new column with the transformed string.

Execution Sample

Apache Spark

from pyspark.sql.functions import upper

# Convert 'name' column to uppercase
result_df = df.select(upper(df['name']).alias('name_upper'))
result_df.show()

This code converts the 'name' column in a Spark DataFrame to uppercase and shows the result.

Execution Table

Step	Input 'name' value	Function Applied	Output 'name_upper' value	Action
1	alice	upper	ALICE	Convert 'alice' to uppercase
2	bob	upper	BOB	Convert 'bob' to uppercase
3	charlie	upper	CHARLIE	Convert 'charlie' to uppercase
4	diana	upper	DIANA	Convert 'diana' to uppercase
5	edward	upper	EDWARD	Convert 'edward' to uppercase
6	-	-	-	All rows processed, show result

💡 All rows processed, no more data to transform

Variable Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	Final
name	['alice','bob','charlie','diana','edward']	alice	bob	charlie	diana	edward	-
name_upper	-	ALICE	BOB	CHARLIE	DIANA	EDWARD	['ALICE','BOB','CHARLIE','DIANA','EDWARD']

Key Moments - 2 Insights

Why does the original 'name' column stay unchanged after applying upper()?

Because Spark string functions create a new column with the transformed data, they do not modify the original column in place. See execution_table rows 1-5 where 'name' stays the same but 'name_upper' changes.

What happens if the input string is null or empty?

Spark string functions like upper() return null if the input is null, and return empty string if input is empty. This is consistent with Spark's handling of nulls in execution_table row logic.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output 'name_upper' value at Step 3?

A charlie

B CHARLIE

C Charlie

D charLie

💡 Hint

Check the 'Output name_upper value' column at Step 3 in execution_table

At which step does the function finish processing all rows?

A Step 6

B Step 5

C Step 4

D Step 3

💡 Hint

Look at the 'Action' column in execution_table for the step indicating all rows processed

If we used lower() instead of upper(), what would be the output for 'name_upper' at Step 2?

A Bob

B BOB

C bob

D bOb

💡 Hint

Consider how lower() changes the input string compared to upper(), check variable_tracker for 'name' at After 2

Concept Snapshot

Spark string functions transform DataFrame string columns.
Use functions like upper(), lower(), trim() on columns.
They create new columns; original data stays unchanged.
Null inputs return null; empty strings stay empty.
Apply with select() or withColumn() for new columns.

Full Transcript

This visual execution shows how Spark string functions work step-by-step. Starting with a DataFrame containing a 'name' column, we apply the upper() function to convert each name to uppercase. Each step processes one row, transforming the string and storing it in a new column 'name_upper'. The original 'name' column remains unchanged throughout. After all rows are processed, the result is displayed. Key points include that Spark functions do not modify original columns but create new ones, and that null or empty strings are handled gracefully. The execution table and variable tracker clearly show the input and output values at each step, helping beginners understand the flow and state changes. The quiz questions reinforce understanding by asking about specific steps and expected outputs. The snapshot summarizes the main ideas for quick reference.

Practice

1. Which Spark string function converts all characters in a column to lowercase?

easy

A. substring()

B. upper()

C. trim()

D. lower()

Solution

1. Step 1: Understand the function purpose

   The lower() function changes all letters in a string to lowercase.

2. Step 2: Compare with other functions

   upper() makes letters uppercase, trim() removes spaces, and substring() extracts part of the string.

3. Final Answer:

   lower() -> Option D

4. Quick Check:

   Lowercase conversion = lower() [OK]

Hint: Lowercase text? Use lower() function [OK]

Common Mistakes:

• Confusing lower() with upper()
• Using trim() to change case
• Thinking substring() changes case

2. Which of the following is the correct syntax to remove leading and trailing spaces from a Spark DataFrame column named name?

easy

A. df.select(trim(col('name')))

B. df.select(trim(name))

C. df.select(trim())

D. df.select(trim('df.name'))

Solution

1. Step 1: Understand Spark column referencing

   In Spark, to refer to a column by name in functions, use col('columnName').

2. Step 2: Apply trim function correctly

   The trim() function expects a Column object, so trim(col('name')) is correct syntax.

3. Final Answer:

   df.select(trim(col('name'))) -> Option A

4. Quick Check:

   Use col() inside trim() for column reference [OK]

Hint: Use col('colName') inside string functions [OK]

Common Mistakes:

• Passing string directly without col()
• Using variable name without quotes or col()
• Using trim() without any argument

3. What will be the output of this Spark code snippet?

from pyspark.sql.functions import substring
df = spark.createDataFrame([('SparkData',)], ['word'])
df.select(substring('word', 1, 5).alias('sub')).collect()

medium

A. [Row(sub='parkD')]

B. [Row(sub='SparkD')]

C. [Row(sub='Spark')]

D. [Row(sub='SparkData')]

Solution

1. Step 1: Understand substring parameters

   substring('word', 1, 5) extracts 5 characters starting at position 1 (1-based index).

2. Step 2: Extract substring from 'SparkData'

   Characters 1 to 5 are 'S', 'p', 'a', 'r', 'k' forming 'Spark'.

3. Final Answer:

   [Row(sub='Spark')] -> Option C

4. Quick Check:

   substring start=1, length=5 extracts 'Spark' [OK]

Hint: substring uses 1-based start index [OK]

Common Mistakes:

• Assuming 0-based index for substring
• Extracting wrong number of characters
• Confusing substring with slice

4. Identify the error in this Spark code that tries to replace all spaces with underscores in column text:

from pyspark.sql.functions import regexp_replace
df.select(regexp_replace(text, ' ', '_')).show()

medium

A. Incorrect number of arguments to regexp_replace

B. Missing column reference using col()

C. Wrong function name, should be replace()

D. Using show() instead of collect()

Solution

1. Step 1: Check argument types for regexp_replace

   The first argument must be a Column object, not a bare identifier.

2. Step 2: Correct usage with col()

   Use regexp_replace(col('text'), ' ', '_') to refer to the column properly.

3. Final Answer:

   Missing column reference using col() -> Option B

4. Quick Check:

   Use col() to reference columns in functions [OK]

Hint: Always wrap column names with col() in functions [OK]

Common Mistakes:

• Passing column name as string without col()
• Confusing regexp_replace with replace
• Ignoring function argument requirements

5. You want to create a new column clean_name in a Spark DataFrame by trimming spaces, converting to lowercase, and replacing all digits with empty strings from column name. Which code snippet correctly does this?

hard

A. df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', ''))

B. df.withColumn('clean_name', lower(trim(regexp_replace(col(name), '\\d', ''))))

C. df.withColumn('clean_name', trim(lower(regexp_replace(col(name), '\\d', ''))))

D. df.withColumn('clean_name', trim(regexp_replace(lower(col(name)), '\\d', '')))

Solution

1. Step 1: Order of operations matters

   First trim spaces, then convert to lowercase, then remove digits.

2. Step 2: Check each option's function nesting

   df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) trims first, then lowers, then replaces digits last, matching the required order.

3. Final Answer:

   df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) -> Option A

4. Quick Check:

   Trim -> Lower -> Replace digits = df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) [OK]

Hint: Chain trim(), lower(), then regexp_replace() in that order [OK]

Common Mistakes:

• Wrong order of function calls
• Missing col() around column name
• Incorrect regex pattern for digits