What is Apache Spark - Practice Questions & Exercises

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Apache Spark's Core Purpose

What is the main purpose of Apache Spark in data processing?

A To provide a fast and general engine for large-scale data processing

B To store data permanently like a database

C To create visual dashboards for data analysis

D To replace all programming languages with a new one

💡 Hint

Think about what Spark does with big data quickly.

Apache Spark is designed to process large data sets quickly and efficiently across many computers.

❓ Predict Output

intermediate

Output of a Spark DataFrame Operation

What will be the output of the following Spark code snippet?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
df_filtered = df.filter(df.id > 1)
df_filtered.show()

A

+---+------+
| id| fruit|
+---+------+
|  1| apple|
|  2|banana|
|  3|cherry|
+---+------+

B

+---+------+
| id| fruit|
+---+------+
|  2|banana|
|  3|cherry|
+---+------+

C SyntaxError: invalid syntax

D RuntimeError: filter condition invalid

💡 Hint

Filter keeps rows where id is greater than 1.

The filter removes the row with id 1, so only rows with id 2 and 3 remain.

❓ data_output

advanced

Result of a Spark Aggregation

Given the following Spark DataFrame, what is the output of the aggregation?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import avg
spark = SparkSession.builder.appName('Test').getOrCreate()
data = [(1, 10), (2, 20), (3, 30), (4, 40)]
df = spark.createDataFrame(data, ['id', 'value'])
avg_df = df.agg(avg('value').alias('average_value'))
avg_df.show()

A

+-------------+
|average_value|
+-------------+
|         25.0|
+-------------+

B

+-----+-----+
|   id|value|
+-----+-----+
|    1|   10|
|    2|   20|
|    3|   30|
|    4|   40|
+-----+-----+

C TypeError: avg() missing 1 required positional argument

D RuntimeError: aggregation failed

💡 Hint

Average of 10, 20, 30, 40 is (10+20+30+40)/4.

The average value is 25.0, calculated by summing all values and dividing by count.

❓ visualization

advanced

Visualizing Data with Spark and Matplotlib

Which option correctly shows how to collect Spark DataFrame data and plot a bar chart of 'fruit' counts using matplotlib?

Apache Spark

from pyspark.sql import SparkSession
import matplotlib.pyplot as plt
spark = SparkSession.builder.appName('Test').getOrCreate()
data = [('apple', 4), ('banana', 2), ('cherry', 5)]
df = spark.createDataFrame(data, ['fruit', 'count'])

A

data = df.toPandas()
plt.bar(data.fruit, data['count'])
plt.show()

B

plt.bar(df['fruit'], df['count'])
plt.show()

C

data = df.collect()
fruits = [row['fruit'] for row in data]
counts = [row['count'] for row in data]
plt.bar(fruits, counts)
plt.show()

D

df.plot(kind='bar')
plt.show()

💡 Hint

You need to convert Spark DataFrame rows to Python lists before plotting.

Option C collects Spark rows into a list, then extracts columns for plotting. Other options misuse Spark DataFrame or pandas methods.

🔧 Debug

expert

Identifying the Error in Spark Code

What error will this Spark code raise?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
data = [(1, 'a'), (2, 'b')]
df = spark.createDataFrame(data, ['id', 'letter'])
result = df.groupBy('id').sum('letter').show()

A No error, outputs sum of letters

B

+---+-----------+
| id|sum(letter)|
+---+-----------+
|  1|          a|
|  2|          b|
+---+-----------+

C TypeError: sum() missing 1 required positional argument

D AnalysisException: cannot resolve 'sum(letter)' due to data type mismatch

💡 Hint

Sum aggregation expects numeric columns, but 'letter' is text.

Trying to sum a string column causes an AnalysisException because sum only works on numbers.