Column expressions and functions in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of chained column expressions

What is the output of the following Apache Spark code snippet?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
data = [(1, 10), (2, 20), (3, 30)]
df = spark.createDataFrame(data, ["id", "value"])

result = df.select((col("value") + 5).alias("value_plus_5"))
result.show()

A

+------------+
|value_plus_5|
+------------+
|          15|
|          25|
|          35|
+------------+

B

+-----+-----+
|   id|value|
+-----+-----+
|    1|   10|
|    2|   20|
|    3|   30|
+-----+-----+

C

+------------+
|value_plus_5|
+------------+
|           5|
|          15|
|          25|
+------------+

D SyntaxError: invalid syntax

💡 Hint

Look at how the column 'value' is transformed by adding 5 and aliased.

The code adds 5 to each value in the 'value' column and selects only this new column with alias 'value_plus_5'. The output shows the incremented values.

❓ data_output

intermediate

Result of filtering with column functions

Given the DataFrame below, what is the output after applying the filter?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
data = [("Alice", 25), ("Bob", 30), ("Charlie", 35)]
df = spark.createDataFrame(data, ["name", "age"])

filtered_df = df.filter(col("age") > 28)
filtered_df.show()

A

+-------+---+
|   name|age|
+-------+---+
|    Bob| 30|
|Charlie| 35|
+-------+---+

B

+-------+---+
|   name|age|
+-------+---+
|  Alice| 25|
|    Bob| 30|
+-------+---+

C

+-------+---+
|   name|age|
+-------+---+
|  Alice| 25|
+-------+---+

D TypeError: '>' not supported between instances of 'Column' and 'int'

💡 Hint

Filter keeps rows where age is greater than 28.

The filter condition selects rows with age > 28, so Bob (30) and Charlie (35) remain.

❓ visualization

advanced

Visualizing aggregated data with groupBy and functions

You have a DataFrame with sales data. Which option correctly shows the output of this aggregation?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import sum

spark = SparkSession.builder.getOrCreate()
data = [("A", 100), ("B", 200), ("A", 300), ("B", 400)]
df = spark.createDataFrame(data, ["category", "sales"])

agg_df = df.groupBy("category").agg(sum("sales").alias("total_sales"))
agg_df.orderBy("category").show()

A

+--------+-----------+
|category|total_sales|
+--------+-----------+
|       A|        300|
|       B|        400|
+--------+-----------+

B

+--------+-----------+
|category|total_sales|
+--------+-----------+
|       A|        400|
|       B|        600|
+--------+-----------+

C

+--------+-----------+
|category|total_sales|
+--------+-----------+
|       A|        100|
|       B|        200|
+--------+-----------+

D AttributeError: 'DataFrame' object has no attribute 'groupby'

💡 Hint

Sum sales grouped by category and order by category.

The sum of sales for category A is 100 + 300 = 400, for B is 200 + 400 = 600.

🔧 Debug

advanced

Identify the error in column expression

What error does the following code produce?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
data = [(1, 2), (3, 4)]
df = spark.createDataFrame(data, ["a", "b"])

result = df.select(col("a") + col("c"))
result.show()

A TypeError: unsupported operand type(s) for +: 'Column' and 'Column'

B NameError: name 'col' is not defined

C AnalysisException: cannot resolve '`c`' given input columns: [a, b]

D No error, outputs sum of columns 'a' and 'c'

💡 Hint

Check if column 'c' exists in the DataFrame.

Column 'c' does not exist in the DataFrame, so Spark raises an AnalysisException.

🚀 Application

expert

Calculate new column with conditional logic

Which option correctly creates a new column 'status' with value 'adult' if age >= 18, else 'minor'?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import when, col

spark = SparkSession.builder.getOrCreate()
data = [("John", 17), ("Jane", 20)]
df = spark.createDataFrame(data, ["name", "age"])

result = df.withColumn("status", ???)
result.select("name", "age", "status").show()

A case when col("age") >= 18 then "adult" else "minor" end

B col("age") >= 18 ? "adult" : "minor"

C if(col("age") >= 18, "adult", "minor")

D when(col("age") >= 18, "adult").otherwise("minor")

💡 Hint

Use Spark SQL functions for conditional column creation.

Option D uses the correct Spark function 'when' with 'otherwise' to create the conditional column.