String functions in Spark in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of substring extraction in Spark

What is the output of this Spark code snippet that extracts a substring from a column?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import substring

spark = SparkSession.builder.getOrCreate()
data = [(1, "DataScience"), (2, "SparkFun"), (3, "Python")]
df = spark.createDataFrame(data, ["id", "word"])
df.select("id", substring("word", 2, 4).alias("sub_word")).orderBy("id").show()

A

+---+--------+
|id |sub_word|
+---+--------+
|1  |ataS    |
|2  |park    |
|3  |yth     |
+---+--------+

B

+---+--------+
|id |sub_word|
+---+--------+
|1  |ataS    |
|2  |par     |
|3  |ytho    |
+---+--------+

C

+---+--------+
|id |sub_word|
+---+--------+
|1  |ataS    |
|2  |parkF   |
|3  |ytho    |
+---+--------+

D

+---+--------+
|id |sub_word|
+---+--------+
|1  |ataS    |
|2  |park    |
|3  |ytho    |
+---+--------+

💡 Hint

Remember that substring in Spark starts at position 1 and length is the number of characters to extract.

The substring function extracts 4 characters starting from position 2 (1-based index). For 'DataScience', characters 2 to 5 are 'ataS'. For 'SparkFun', characters 2 to 5 are 'park'. For 'Python', characters 2 to 5 are 'ytho'.

❓ data_output

intermediate

Result of trimming spaces in Spark DataFrame

Given this Spark DataFrame, what is the output after applying the trim function to the 'text' column?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import trim

spark = SparkSession.builder.getOrCreate()
data = [(1, "  hello  "), (2, " world"), (3, "spark  ")]
df = spark.createDataFrame(data, ["id", "text"])
df.select("id", trim("text").alias("trimmed_text")).orderBy("id").show()

A

+---+------------+
|id |trimmed_text|
+---+------------+
|1  |hello       |
|2  |world       |
|3  |spark       |
+---+------------+

B

+---+------------+
|id |trimmed_text|
+---+------------+
|1  |  hello     |
|2  | world      |
|3  |spark       |
+---+------------+

C

+---+------------+
|id |trimmed_text|
+---+------------+
|1  |hello       |
|2  |world       |
|3  |spark  	   |
+---+------------+

D

+---+------------+
|id |trimmed_text|
+---+------------+
|1  |hello       |
|2  |world       |
|3  | spark      |
+---+------------+

💡 Hint

The trim function removes spaces from both ends of the string.

The trim function removes leading and trailing spaces. So ' hello ' becomes 'hello', ' world' becomes 'world', and 'spark ' becomes 'spark'.

🔧 Debug

advanced

Identify the error in Spark string concatenation

What error will this Spark code raise when trying to concatenate two string columns?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import concat

spark = SparkSession.builder.getOrCreate()
data = [(1, "Hello", "World"), (2, "Spark", "Fun")]
df = spark.createDataFrame(data, ["id", "col1", "col2"])
df.select("id", concat("col1", "col2").alias("greeting")).show()

A No error, outputs concatenated strings without space

B AnalysisException: cannot resolve 'concat(col1, col2)' due to data type mismatch

C TypeError: concat() argument must be a Column or a list of Columns

D SyntaxError: invalid syntax in concat function call

💡 Hint

Check the argument types passed to concat function in Spark.

The concat function expects Column objects, not string column names. Passing strings causes a TypeError.

🚀 Application

advanced

Using regexp_replace to clean data

You want to remove all digits from the 'info' column in a Spark DataFrame. Which code snippet correctly does this?

A df.withColumn('clean_info', regexp_replace('info', '\\d', '')).show()

B df.withColumn('clean_info', regexp_replace('info', '[0-9]+', '')).show()

C df.withColumn('clean_info', regexp_replace('info', '[a-z]', '')).show()

D df.withColumn('clean_info', regexp_replace('info', '\\D', '')).show()

💡 Hint

Digits are represented by \d or [0-9] in regex, but consider the difference between \d and \D.

Option B uses '[0-9]+' which matches one or more digits and replaces them with empty string, effectively removing digits. Option B uses '\d' but the backslash needs to be escaped properly in Python strings. Option B removes letters, not digits. Option B removes non-digits, which is the opposite of what is wanted.

🧠 Conceptual

expert

Understanding Spark string function behavior with nulls

What is the result of applying the Spark function upper() to a column containing null values?

A The upper() function returns null for null input values.

B The upper() function raises a NullPointerException when encountering null values.

C The upper() function converts null values to empty strings before converting to uppercase.

D The upper() function ignores null values and leaves them unchanged in the output.

💡 Hint

Think about how Spark functions handle null inputs generally.

Spark string functions like upper() return null if the input is null. They do not raise errors or convert nulls to empty strings.

Practice

1. Which Spark string function converts all characters in a column to lowercase?

easy

A. substring()

B. upper()

C. trim()

D. lower()

Solution

1. Step 1: Understand the function purpose

   The lower() function changes all letters in a string to lowercase.

2. Step 2: Compare with other functions

   upper() makes letters uppercase, trim() removes spaces, and substring() extracts part of the string.

3. Final Answer:

   lower() -> Option D

4. Quick Check:

   Lowercase conversion = lower() [OK]

Hint: Lowercase text? Use lower() function [OK]

Common Mistakes:

• Confusing lower() with upper()
• Using trim() to change case
• Thinking substring() changes case

2. Which of the following is the correct syntax to remove leading and trailing spaces from a Spark DataFrame column named name?

easy

A. df.select(trim(col('name')))

B. df.select(trim(name))

C. df.select(trim())

D. df.select(trim('df.name'))

Solution

1. Step 1: Understand Spark column referencing

   In Spark, to refer to a column by name in functions, use col('columnName').

2. Step 2: Apply trim function correctly

   The trim() function expects a Column object, so trim(col('name')) is correct syntax.

3. Final Answer:

   df.select(trim(col('name'))) -> Option A

4. Quick Check:

   Use col() inside trim() for column reference [OK]

Hint: Use col('colName') inside string functions [OK]

Common Mistakes:

• Passing string directly without col()
• Using variable name without quotes or col()
• Using trim() without any argument

3. What will be the output of this Spark code snippet?

from pyspark.sql.functions import substring
df = spark.createDataFrame([('SparkData',)], ['word'])
df.select(substring('word', 1, 5).alias('sub')).collect()

medium

A. [Row(sub='parkD')]

B. [Row(sub='SparkD')]

C. [Row(sub='Spark')]

D. [Row(sub='SparkData')]

Solution

1. Step 1: Understand substring parameters

   substring('word', 1, 5) extracts 5 characters starting at position 1 (1-based index).

2. Step 2: Extract substring from 'SparkData'

   Characters 1 to 5 are 'S', 'p', 'a', 'r', 'k' forming 'Spark'.

3. Final Answer:

   [Row(sub='Spark')] -> Option C

4. Quick Check:

   substring start=1, length=5 extracts 'Spark' [OK]

Hint: substring uses 1-based start index [OK]

Common Mistakes:

• Assuming 0-based index for substring
• Extracting wrong number of characters
• Confusing substring with slice

4. Identify the error in this Spark code that tries to replace all spaces with underscores in column text:

from pyspark.sql.functions import regexp_replace
df.select(regexp_replace(text, ' ', '_')).show()

medium

A. Incorrect number of arguments to regexp_replace

B. Missing column reference using col()

C. Wrong function name, should be replace()

D. Using show() instead of collect()

Solution

1. Step 1: Check argument types for regexp_replace

   The first argument must be a Column object, not a bare identifier.

2. Step 2: Correct usage with col()

   Use regexp_replace(col('text'), ' ', '_') to refer to the column properly.

3. Final Answer:

   Missing column reference using col() -> Option B

4. Quick Check:

   Use col() to reference columns in functions [OK]

Hint: Always wrap column names with col() in functions [OK]

Common Mistakes:

• Passing column name as string without col()
• Confusing regexp_replace with replace
• Ignoring function argument requirements

5. You want to create a new column clean_name in a Spark DataFrame by trimming spaces, converting to lowercase, and replacing all digits with empty strings from column name. Which code snippet correctly does this?

hard

A. df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', ''))

B. df.withColumn('clean_name', lower(trim(regexp_replace(col(name), '\\d', ''))))

C. df.withColumn('clean_name', trim(lower(regexp_replace(col(name), '\\d', ''))))

D. df.withColumn('clean_name', trim(regexp_replace(lower(col(name)), '\\d', '')))

Solution

1. Step 1: Order of operations matters

   First trim spaces, then convert to lowercase, then remove digits.

2. Step 2: Check each option's function nesting

   df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) trims first, then lowers, then replaces digits last, matching the required order.

3. Final Answer:

   df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) -> Option A

4. Quick Check:

   Trim -> Lower -> Replace digits = df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) [OK]

Hint: Chain trim(), lower(), then regexp_replace() in that order [OK]

Common Mistakes:

• Wrong order of function calls
• Missing col() around column name
• Incorrect regex pattern for digits