String functions in Spark in Apache Spark - Mini Project: Build & Apply

String functions in Spark

📖 Scenario: You work in a company that collects customer feedback. The feedback is stored as text strings in a Spark DataFrame. You want to clean and analyze this text data using Spark's string functions.

🎯 Goal: Learn how to use basic string functions in Spark to manipulate and analyze text data in a DataFrame.

📋 What You'll Learn

Create a Spark DataFrame with customer feedback strings

Add a configuration variable for a keyword to search

Use Spark string functions to find feedback containing the keyword and convert text to uppercase

Display the filtered and transformed feedback

💡 Why This Matters

🌍 Real World

Cleaning and analyzing customer feedback text is common in many businesses to improve products and services.

💼 Career

Data scientists and analysts often use Spark string functions to preprocess and analyze large text datasets efficiently.

1

Create a Spark DataFrame with customer feedback

Create a Spark DataFrame called feedback_df with one column named feedback containing these exact strings: 'Great service', 'Poor quality', 'Excellent support', 'Average experience', 'Poor response time'.

Apache Spark

# Create a Spark DataFrame called feedback_df with the specified feedback strings
# Your code here

Hint

Use spark.createDataFrame with a list of tuples and specify the column name as ['feedback'].

2

Add a keyword variable to search in feedback

Create a variable called keyword and set it to the string 'Poor'.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('StringFunctions').getOrCreate()

feedback_list = [
    ('Great service',),
    ('Poor quality',),
    ('Excellent support',),
    ('Average experience',),
    ('Poor response time',)
]

feedback_df = spark.createDataFrame(feedback_list, ['feedback'])

# Create a variable called keyword and set it to 'Poor'
# Your code here

Hint

Just assign the string 'Poor' to the variable keyword.

3

Filter feedback containing the keyword and convert to uppercase

Use Spark string functions to create a new DataFrame called filtered_df that contains only rows where the feedback column contains the keyword. Also add a new column called feedback_upper that has the feedback text converted to uppercase. Use filter, contains, and upper functions from pyspark.sql.functions.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('StringFunctions').getOrCreate()

feedback_list = [
    ('Great service',),
    ('Poor quality',),
    ('Excellent support',),
    ('Average experience',),
    ('Poor response time',)
]

feedback_df = spark.createDataFrame(feedback_list, ['feedback'])

keyword = 'Poor'

# Create filtered_df with rows where feedback contains keyword
# Add a new column feedback_upper with uppercase feedback text
# Your code here

Hint

Use filter(col('feedback').contains(keyword)) to select rows, then withColumn('feedback_upper', upper(col('feedback'))) to add uppercase column.

4

Display the filtered and transformed feedback

Use show() on filtered_df to display the rows with the original and uppercase feedback.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, upper

spark = SparkSession.builder.appName('StringFunctions').getOrCreate()

feedback_list = [
    ('Great service',),
    ('Poor quality',),
    ('Excellent support',),
    ('Average experience',),
    ('Poor response time',)
]

feedback_df = spark.createDataFrame(feedback_list, ['feedback'])

keyword = 'Poor'

filtered_df = feedback_df.filter(col('feedback').contains(keyword))
filtered_df = filtered_df.withColumn('feedback_upper', upper(col('feedback')))

# Display the filtered_df
# Your code here

Hint

Use filtered_df.show() to print the DataFrame with the new column.

Practice

1. Which Spark string function converts all characters in a column to lowercase?

easy

A. substring()

B. upper()

C. trim()

D. lower()

Solution

1. Step 1: Understand the function purpose

   The lower() function changes all letters in a string to lowercase.

2. Step 2: Compare with other functions

   upper() makes letters uppercase, trim() removes spaces, and substring() extracts part of the string.

3. Final Answer:

   lower() -> Option D

4. Quick Check:

   Lowercase conversion = lower() [OK]

Hint: Lowercase text? Use lower() function [OK]

Common Mistakes:

• Confusing lower() with upper()
• Using trim() to change case
• Thinking substring() changes case

2. Which of the following is the correct syntax to remove leading and trailing spaces from a Spark DataFrame column named name?

easy

A. df.select(trim(col('name')))

B. df.select(trim(name))

C. df.select(trim())

D. df.select(trim('df.name'))

Solution

1. Step 1: Understand Spark column referencing

   In Spark, to refer to a column by name in functions, use col('columnName').

2. Step 2: Apply trim function correctly

   The trim() function expects a Column object, so trim(col('name')) is correct syntax.

3. Final Answer:

   df.select(trim(col('name'))) -> Option A

4. Quick Check:

   Use col() inside trim() for column reference [OK]

Hint: Use col('colName') inside string functions [OK]

Common Mistakes:

• Passing string directly without col()
• Using variable name without quotes or col()
• Using trim() without any argument

3. What will be the output of this Spark code snippet?

from pyspark.sql.functions import substring
df = spark.createDataFrame([('SparkData',)], ['word'])
df.select(substring('word', 1, 5).alias('sub')).collect()

medium

A. [Row(sub='parkD')]

B. [Row(sub='SparkD')]

C. [Row(sub='Spark')]

D. [Row(sub='SparkData')]

Solution

1. Step 1: Understand substring parameters

   substring('word', 1, 5) extracts 5 characters starting at position 1 (1-based index).

2. Step 2: Extract substring from 'SparkData'

   Characters 1 to 5 are 'S', 'p', 'a', 'r', 'k' forming 'Spark'.

3. Final Answer:

   [Row(sub='Spark')] -> Option C

4. Quick Check:

   substring start=1, length=5 extracts 'Spark' [OK]

Hint: substring uses 1-based start index [OK]

Common Mistakes:

• Assuming 0-based index for substring
• Extracting wrong number of characters
• Confusing substring with slice

4. Identify the error in this Spark code that tries to replace all spaces with underscores in column text:

from pyspark.sql.functions import regexp_replace
df.select(regexp_replace(text, ' ', '_')).show()

medium

A. Incorrect number of arguments to regexp_replace

B. Missing column reference using col()

C. Wrong function name, should be replace()

D. Using show() instead of collect()

Solution

1. Step 1: Check argument types for regexp_replace

   The first argument must be a Column object, not a bare identifier.

2. Step 2: Correct usage with col()

   Use regexp_replace(col('text'), ' ', '_') to refer to the column properly.

3. Final Answer:

   Missing column reference using col() -> Option B

4. Quick Check:

   Use col() to reference columns in functions [OK]

Hint: Always wrap column names with col() in functions [OK]

Common Mistakes:

• Passing column name as string without col()
• Confusing regexp_replace with replace
• Ignoring function argument requirements

5. You want to create a new column clean_name in a Spark DataFrame by trimming spaces, converting to lowercase, and replacing all digits with empty strings from column name. Which code snippet correctly does this?

hard

A. df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', ''))

B. df.withColumn('clean_name', lower(trim(regexp_replace(col(name), '\\d', ''))))

C. df.withColumn('clean_name', trim(lower(regexp_replace(col(name), '\\d', ''))))

D. df.withColumn('clean_name', trim(regexp_replace(lower(col(name)), '\\d', '')))

Solution

1. Step 1: Order of operations matters

   First trim spaces, then convert to lowercase, then remove digits.

2. Step 2: Check each option's function nesting

   df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) trims first, then lowers, then replaces digits last, matching the required order.

3. Final Answer:

   df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) -> Option A

4. Quick Check:

   Trim -> Lower -> Replace digits = df.withColumn('clean_name', regexp_replace(lower(trim(col('name'))), '\\d', '')) [OK]

Hint: Chain trim(), lower(), then regexp_replace() in that order [OK]

Common Mistakes:

• Wrong order of function calls
• Missing col() around column name
• Incorrect regex pattern for digits