String functions in Spark in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to convert the 'name' column to lowercase using Spark string functions.

Apache Spark

from pyspark.sql.functions import [1]
df_lower = df.select([1]("name").alias("name_lower"))

Drag options to blanks, or click blank then click option'

Alower

Bupper

Ctrim

Dlength

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'upper' instead of 'lower' changes text to uppercase.

Using 'trim' removes spaces but does not change case.

The lower function converts all characters in a string column to lowercase.

2fill in blank

medium

Complete the code to remove spaces from the start and end of the 'city' column.

Apache Spark

from pyspark.sql.functions import [1]
df_trimmed = df.select([1]("city").alias("city_trimmed"))

Drag options to blanks, or click blank then click option'

Alower

Bconcat

Csubstring

Dtrim

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'lower' changes case but does not remove spaces.

Using 'concat' joins strings but does not trim spaces.

The trim function removes leading and trailing spaces from string columns.

3fill in blank

hard

Fix the error in the code to extract the first 3 characters from the 'code' column.

Apache Spark

from pyspark.sql.functions import substring
df_sub = df.select(substring("code", [1], 3).alias("code_sub"))

Drag options to blanks, or click blank then click option'

A3

B4

C1

D0

Attempts:

3 left

💡 Hint

Common Mistakes

Starting substring at 0 causes an error or empty result.

Starting at 3 or 4 skips the first characters.

The substring function in Spark starts counting positions at 1, so to get the first 3 characters, start at position 1.

4fill in blank

hard

Fill both blanks to create a dictionary with words as keys and their lengths as values, only for words longer than 4 characters.

Apache Spark

words = ["apple", "bat", "carrot", "dog"]
lengths = { [1] : len([1]) for [1] in words if len([1]) [2] 4 }

Drag options to blanks, or click blank then click option'

Aword

B>

C<

Dlen

Attempts:

3 left

💡 Hint

Common Mistakes

Using '<' filters words shorter than 4.

Using 'len' as a variable name causes confusion.

We use word as the variable name and filter words with length greater than 4 using >.

5fill in blank

hard

Fill all three blanks to create a dictionary with uppercase words as keys and their lengths as values, only for words containing the letter 'a'.

Apache Spark

words = ["apple", "bat", "carrot", "dog"]
result = { [1] : [2] for [3] in words if 'a' in [3] }

Drag options to blanks, or click blank then click option'

Aword.upper()

Blen(word)

Cword

Dword.lower()

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'word.lower()' instead of 'word.upper()' changes case incorrectly.

Using 'word' as key does not convert to uppercase.

We use word as the loop variable, convert it to uppercase for keys with word.upper(), and get lengths with len(word).

Practice

1. Which Spark string function converts all characters in a column to lowercase?

easy

A. substring()

B. upper()

C. trim()

D. lower()

Solution

1. Step 1: Understand the function purpose

   The lower() function changes all letters in a string to lowercase.

2. Step 2: Compare with other functions

   upper() makes letters uppercase, trim() removes spaces, and substring() extracts part of the string.

3. Final Answer:

   lower() -> Option D

4. Quick Check:

   Lowercase conversion = lower() [OK]

Hint: Lowercase text? Use lower() function [OK]

Common Mistakes:

• Confusing lower() with upper()
• Using trim() to change case
• Thinking substring() changes case

2. Which of the following is the correct syntax to remove leading and trailing spaces from a Spark DataFrame column named name?

easy

A. df.select(trim(col('name')))

B. df.select(trim(name))

C. df.select(trim())

D. df.select(trim('df.name'))

Solution

1. Step 1: Understand Spark column referencing

   In Spark, to refer to a column by name in functions, use col('columnName').

2. Step 2: Apply trim function correctly

   The trim() function expects a Column object, so trim(col('name')) is correct syntax.

3. Final Answer:

   df.select(trim(col('name'))) -> Option A

4. Quick Check:

   Use col() inside trim() for column reference [OK]

Hint: Use col('colName') inside string functions [OK]

Common Mistakes:

• Passing string directly without col()
• Using variable name without quotes or col()
• Using trim() without any argument

3. What will be the output of this Spark code snippet?

from pyspark.sql.functions import substring
df = spark.createDataFrame([('SparkData',)], ['word'])
df.select(substring('word', 1, 5).alias('sub')).collect()

medium

A. [Row(sub='parkD')]

B. [Row(sub='SparkD')]

C. [Row(sub='Spark')]

D. [Row(sub='SparkData')]

Solution

1. Step 1: Understand substring parameters

   substring('word', 1, 5) extracts 5 characters starting at position 1 (1-based index).

2. Step 2: Extract substring from 'SparkData'

   Characters 1 to 5 are 'S', 'p', 'a', 'r', 'k' forming 'Spark'.

3. Final Answer:

   [Row(sub='Spark')] -> Option C

4. Quick Check:

   substring start=1, length=5 extracts 'Spark' [OK]

Hint: substring uses 1-based start index [OK]

Common Mistakes:

• Assuming 0-based index for substring
• Extracting wrong number of characters
• Confusing substring with slice

4. Identify the error in this Spark code that tries to replace all spaces with underscores in column text:

from pyspark.sql.functions import regexp_replace
df.select(regexp_replace(text, ' ', '_')).show()

medium

A. Incorrect number of arguments to regexp_replace

B. Missing column reference using col()

C. Wrong function name, should be replace()

D. Using show() instead of collect()

Solution

1. Step 1: Check argument types for regexp_replace

   The first argument must be a Column object, not a bare identifier.

2. Step 2: Correct usage with col()

   Use regexp_replace(col('text'), ' ', '_') to refer to the column properly.

3. Final Answer:

   Missing column reference using col() -> Option B

4. Quick Check:

   Use col() to reference columns in functions [OK]

Hint: Always wrap column names with col() in functions [OK]

Common Mistakes:

• Passing column name as string without col()
• Confusing regexp_replace with replace
• Ignoring function argument requirements

5. You want to create a new column clean_name in a Spark DataFrame by trimming spaces, converting to lowercase, and replacing all digits with empty strings from column name. Which code snippet correctly does this?

hard

A. df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', ''))

B. df.withColumn('clean_name', lower(trim(regexp_replace(col(name), '\\d', ''))))

C. df.withColumn('clean_name', trim(lower(regexp_replace(col(name), '\\d', ''))))

D. df.withColumn('clean_name', trim(regexp_replace(lower(col(name)), '\\d', '')))

Solution

1. Step 1: Order of operations matters

   First trim spaces, then convert to lowercase, then remove digits.

2. Step 2: Check each option's function nesting

   df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) trims first, then lowers, then replaces digits last, matching the required order.

3. Final Answer:

   df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) -> Option A

4. Quick Check:

   Trim -> Lower -> Replace digits = df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) [OK]

Hint: Chain trim(), lower(), then regexp_replace() in that order [OK]

Common Mistakes:

• Wrong order of function calls
• Missing col() around column name
• Incorrect regex pattern for digits