Column expressions and functions in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of chained column expressions

What is the output of the following Apache Spark code snippet?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
data = [(1, 10), (2, 20), (3, 30)]
df = spark.createDataFrame(data, ["id", "value"])

result = df.select((col("value") + 5).alias("value_plus_5"))
result.show()

A

+------------+
|value_plus_5|
+------------+
|          15|
|          25|
|          35|
+------------+

B

+-----+-----+
|   id|value|
+-----+-----+
|    1|   10|
|    2|   20|
|    3|   30|
+-----+-----+

C

+------------+
|value_plus_5|
+------------+
|           5|
|          15|
|          25|
+------------+

D SyntaxError: invalid syntax

💡 Hint

Look at how the column 'value' is transformed by adding 5 and aliased.

The code adds 5 to each value in the 'value' column and selects only this new column with alias 'value_plus_5'. The output shows the incremented values.

❓ data_output

intermediate

Result of filtering with column functions

Given the DataFrame below, what is the output after applying the filter?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
data = [("Alice", 25), ("Bob", 30), ("Charlie", 35)]
df = spark.createDataFrame(data, ["name", "age"])

filtered_df = df.filter(col("age") > 28)
filtered_df.show()

A

+-------+---+
|   name|age|
+-------+---+
|    Bob| 30|
|Charlie| 35|
+-------+---+

B

+-------+---+
|   name|age|
+-------+---+
|  Alice| 25|
|    Bob| 30|
+-------+---+

C

+-------+---+
|   name|age|
+-------+---+
|  Alice| 25|
+-------+---+

D TypeError: '>' not supported between instances of 'Column' and 'int'

💡 Hint

Filter keeps rows where age is greater than 28.

The filter condition selects rows with age > 28, so Bob (30) and Charlie (35) remain.

❓ visualization

advanced

Visualizing aggregated data with groupBy and functions

You have a DataFrame with sales data. Which option correctly shows the output of this aggregation?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import sum

spark = SparkSession.builder.getOrCreate()
data = [("A", 100), ("B", 200), ("A", 300), ("B", 400)]
df = spark.createDataFrame(data, ["category", "sales"])

agg_df = df.groupBy("category").agg(sum("sales").alias("total_sales"))
agg_df.orderBy("category").show()

A

+--------+-----------+
|category|total_sales|
+--------+-----------+
|       A|        300|
|       B|        400|
+--------+-----------+

B

+--------+-----------+
|category|total_sales|
+--------+-----------+
|       A|        400|
|       B|        600|
+--------+-----------+

C

+--------+-----------+
|category|total_sales|
+--------+-----------+
|       A|        100|
|       B|        200|
+--------+-----------+

D AttributeError: 'DataFrame' object has no attribute 'groupby'

💡 Hint

Sum sales grouped by category and order by category.

The sum of sales for category A is 100 + 300 = 400, for B is 200 + 400 = 600.

🔧 Debug

advanced

Identify the error in column expression

What error does the following code produce?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
data = [(1, 2), (3, 4)]
df = spark.createDataFrame(data, ["a", "b"])

result = df.select(col("a") + col("c"))
result.show()

A TypeError: unsupported operand type(s) for +: 'Column' and 'Column'

B NameError: name 'col' is not defined

C AnalysisException: cannot resolve '`c`' given input columns: [a, b]

D No error, outputs sum of columns 'a' and 'c'

💡 Hint

Check if column 'c' exists in the DataFrame.

Column 'c' does not exist in the DataFrame, so Spark raises an AnalysisException.

🚀 Application

expert

Calculate new column with conditional logic

Which option correctly creates a new column 'status' with value 'adult' if age >= 18, else 'minor'?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import when, col

spark = SparkSession.builder.getOrCreate()
data = [("John", 17), ("Jane", 20)]
df = spark.createDataFrame(data, ["name", "age"])

result = df.withColumn("status", ???)
result.select("name", "age", "status").show()

A case when col("age") >= 18 then "adult" else "minor" end

B col("age") >= 18 ? "adult" : "minor"

C if(col("age") >= 18, "adult", "minor")

D when(col("age") >= 18, "adult").otherwise("minor")

💡 Hint

Use Spark SQL functions for conditional column creation.

Option D uses the correct Spark function 'when' with 'otherwise' to create the conditional column.

Practice

1. What does the withColumn function do in Apache Spark DataFrame?

easy

A. It adds a new column or replaces an existing column with a new value.

B. It deletes a column from the DataFrame.

C. It filters rows based on a condition.

D. It groups data by a column.

Solution

1. Step 1: Understand the purpose of withColumn

   The withColumn function is used to add a new column or replace an existing one in a DataFrame.

2. Step 2: Compare with other functions

   Deleting columns is done by drop, filtering by filter, and grouping by groupBy. So these are not correct.

3. Final Answer:

   It adds a new column or replaces an existing column with a new value. -> Option A

4. Quick Check:

   withColumn = add/replace column [OK]

Hint: Remember: withColumn adds or changes columns fast [OK]

Common Mistakes:

• Confusing withColumn with drop or filter
• Thinking it deletes columns
• Mixing it with groupBy

2. Which of the following is the correct syntax to select a column named age from a DataFrame df using Spark's column expression?

easy

A. df.select("age")

B. df.select(age)

C. df.select(df.age())

D. df.select(`age`)

Solution

1. Step 1: Understand select syntax

   The select method expects column names as strings or Column objects. Using double quotes around column names is the standard correct syntax.

2. Step 2: Check each option

   df.select("age") (A) uses a valid string literal.
   df.select(age) (B) references undefined variable age - error.
   df.select(df.age()) (C) tries to call df.age as a function - Column not callable.
   df.select(`age`) (D) uses invalid backtick syntax.

3. Step 3: Identify the correct syntax

   Only df.select("age") executes without error and selects the column correctly.

4. Final Answer:

   df.select("age") -> Option A

5. Quick Check:

   Column names as strings in select = correct [OK]

Hint: Use quotes around column names in select [OK]

Common Mistakes:

• Using variable names without quotes
• Confusing Column objects with strings
• Missing quotes causing errors

3. Given the following Spark code, what will be the output of df.show()?

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
data = [(1, 10), (2, 20), (3, 30)]
df = spark.createDataFrame(data, ['id', 'value'])
df = df.withColumn('double_value', col('value') * 2)
df.show()

medium

A. +---+-----+------------+
|id |value|double_value|
+---+-----+------------+
|1 |10 |10 |
|2 |20 |20 |
|3 |30 |30 |
+---+-----+------------+

B. Error: col function not found

C. +---+-----+------------+
|id |value|double_value|
+---+-----+------------+
|1 |10 |20 |
|2 |20 |40 |
|3 |30 |60 |
+---+-----+------------+

D. +---+-----+
|id |value|
+---+-----+
|1 |10 |
|2 |20 |
|3 |30 |
+---+-----+

Solution

1. Step 1: Understand the data and transformation

   The DataFrame starts with columns 'id' and 'value' with values (1,10), (2,20), (3,30). The withColumn adds 'double_value' by multiplying 'value' by 2.

2. Step 2: Calculate the new column values

   For each row, double_value = value * 2, so 10*2=20, 20*2=40, 30*2=60.

3. Final Answer:

   The DataFrame will show three columns with double_value as twice value: -> Option C

4. Quick Check:

   double_value = value * 2 [OK]

Hint: Multiply column with col('name') * number [OK]

Common Mistakes:

• Forgetting to import col function
• Using wrong column name
• Expecting original DataFrame without new column

4. Identify the error in this Spark code snippet:

from pyspark.sql.functions import col

df = spark.createDataFrame([(1, 'Alice'), (2, 'Bob')], ['id', 'name'])
df = df.withColumn('name_length', length(col('name')))
df.show()

medium

A. Syntax error in withColumn usage

B. Incorrect column name 'name'

C. DataFrame creation syntax is wrong

D. Missing import for length function

Solution

1. Step 1: Check imports for used functions

   The code uses length function but only imports col. The length function must be imported from pyspark.sql.functions.

2. Step 2: Verify other parts

   Column name 'name' exists, syntax for withColumn is correct, and DataFrame creation is valid.

3. Final Answer:

   Missing import for length function -> Option D

4. Quick Check:

   Import all functions you use [OK]

Hint: Always import functions like length explicitly [OK]

Common Mistakes:

• Forgetting to import needed functions
• Assuming all functions are auto-imported
• Typos in column names

5. You have a DataFrame df with columns id, score1, and score2. You want to create a new column max_score that contains the maximum value between score1 and score2 for each row. Which code snippet correctly achieves this?

hard

A. df = df.withColumn('max_score', max(col('score1'), col('score2')))

B. from pyspark.sql.functions import greatest

df = df.withColumn('max_score', greatest(col('score1'), col('score2')))

C. df = df.withColumn('max_score', col('score1') if col('score1') > col('score2') else col('score2'))

D. df = df.withColumn('max_score', col('score1').max(col('score2')))

Solution

1. Step 1: Identify the correct function for max between columns

   Spark provides the greatest function to get the maximum value among columns for each row.

2. Step 2: Check each option

   from pyspark.sql.functions import greatest df = df.withColumn('max_score', greatest(col('score1'), col('score2'))) uses greatest correctly with imported function. df = df.withColumn('max_score', max(col('score1'), col('score2'))) uses max which is an aggregate function, not for row-wise max. df = df.withColumn('max_score', col('score1') if col('score1') > col('score2') else col('score2')) tries Python if-else which does not work on columns. df = df.withColumn('max_score', col('score1').max(col('score2'))) uses invalid syntax.

3. Final Answer:

   Use greatest function with columns -> Option B

4. Quick Check:

   greatest(col1, col2) = max per row [OK]

Hint: Use greatest() for max of columns per row [OK]

Common Mistakes:

• Using max() instead of greatest() for columns
• Trying Python if-else on columns
• Wrong method chaining syntax