What is Apache Spark - Practice Questions & Exercises

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Apache Spark's Core Purpose

What is the main purpose of Apache Spark in data processing?

A To provide a fast and general engine for large-scale data processing

B To store data permanently like a database

C To create visual dashboards for data analysis

D To replace all programming languages with a new one

💡 Hint

Think about what Spark does with big data quickly.

Apache Spark is designed to process large data sets quickly and efficiently across many computers.

❓ Predict Output

intermediate

Output of a Spark DataFrame Operation

What will be the output of the following Spark code snippet?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
df_filtered = df.filter(df.id > 1)
df_filtered.show()

A

+---+------+
| id| fruit|
+---+------+
|  1| apple|
|  2|banana|
|  3|cherry|
+---+------+

B

+---+------+
| id| fruit|
+---+------+
|  2|banana|
|  3|cherry|
+---+------+

C SyntaxError: invalid syntax

D RuntimeError: filter condition invalid

💡 Hint

Filter keeps rows where id is greater than 1.

The filter removes the row with id 1, so only rows with id 2 and 3 remain.

❓ data_output

advanced

Result of a Spark Aggregation

Given the following Spark DataFrame, what is the output of the aggregation?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import avg
spark = SparkSession.builder.appName('Test').getOrCreate()
data = [(1, 10), (2, 20), (3, 30), (4, 40)]
df = spark.createDataFrame(data, ['id', 'value'])
avg_df = df.agg(avg('value').alias('average_value'))
avg_df.show()

A

+-------------+
|average_value|
+-------------+
|         25.0|
+-------------+

B

+-----+-----+
|   id|value|
+-----+-----+
|    1|   10|
|    2|   20|
|    3|   30|
|    4|   40|
+-----+-----+

C TypeError: avg() missing 1 required positional argument

D RuntimeError: aggregation failed

💡 Hint

Average of 10, 20, 30, 40 is (10+20+30+40)/4.

The average value is 25.0, calculated by summing all values and dividing by count.

❓ visualization

advanced

Visualizing Data with Spark and Matplotlib

Which option correctly shows how to collect Spark DataFrame data and plot a bar chart of 'fruit' counts using matplotlib?

Apache Spark

from pyspark.sql import SparkSession
import matplotlib.pyplot as plt
spark = SparkSession.builder.appName('Test').getOrCreate()
data = [('apple', 4), ('banana', 2), ('cherry', 5)]
df = spark.createDataFrame(data, ['fruit', 'count'])

A

data = df.toPandas()
plt.bar(data.fruit, data['count'])
plt.show()

B

plt.bar(df['fruit'], df['count'])
plt.show()

C

data = df.collect()
fruits = [row['fruit'] for row in data]
counts = [row['count'] for row in data]
plt.bar(fruits, counts)
plt.show()

D

df.plot(kind='bar')
plt.show()

💡 Hint

You need to convert Spark DataFrame rows to Python lists before plotting.

Option C collects Spark rows into a list, then extracts columns for plotting. Other options misuse Spark DataFrame or pandas methods.

🔧 Debug

expert

Identifying the Error in Spark Code

What error will this Spark code raise?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
data = [(1, 'a'), (2, 'b')]
df = spark.createDataFrame(data, ['id', 'letter'])
result = df.groupBy('id').sum('letter').show()

A No error, outputs sum of letters

B

+---+-----------+
| id|sum(letter)|
+---+-----------+
|  1|          a|
|  2|          b|
+---+-----------+

C TypeError: sum() missing 1 required positional argument

D AnalysisException: cannot resolve 'sum(letter)' due to data type mismatch

💡 Hint

Sum aggregation expects numeric columns, but 'letter' is text.

Trying to sum a string column causes an AnalysisException because sum only works on numbers.

Practice

1. What is Apache Spark primarily used for?

easy

A. Creating websites and web pages

B. Processing large amounts of data quickly using many computers

C. Designing computer hardware

D. Writing simple text documents

Solution

1. Step 1: Understand Apache Spark's purpose

   Apache Spark is designed to handle big data processing efficiently by using multiple computers working together.

2. Step 2: Compare options with Spark's use

   Options B, C, and D describe unrelated tasks like web design, hardware, or text editing, which Spark does not do.

3. Final Answer:

   Processing large amounts of data quickly using many computers -> Option B

4. Quick Check:

   Big data processing = A [OK]

Hint: Think big data and speed with many computers [OK]

Common Mistakes:

• Confusing Spark with web or hardware tools
• Thinking Spark is for simple text editing

2. Which of the following is the correct way to start a Spark session in Python?

easy

A. spark = SparkSession.builder.appName('App').getOrCreate()

B. spark = SparkSession.create('App')

C. spark = SparkSession.start('App')

D. spark = SparkSession.run('App')

Solution

1. Step 1: Recall Spark session creation syntax

   The correct syntax to create a Spark session is using builder, setting appName, then getOrCreate().

2. Step 2: Check each option's method

   Only spark = SparkSession.builder.appName('App').getOrCreate() uses builder.appName(...).getOrCreate(), others use invalid methods.

3. Final Answer:

   spark = SparkSession.builder.appName('App').getOrCreate() -> Option A

4. Quick Check:

   builder.appName().getOrCreate() = B [OK]

Hint: Remember builder.appName().getOrCreate() pattern [OK]

Common Mistakes:

• Using non-existent methods like create or start
• Missing getOrCreate() call

3. Given the following Spark code, what will be the output?

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
df.filter(df.id > 1).count()

medium

A. 3

B. 1

C. 0

D. 2

Solution

1. Step 1: Understand the DataFrame content

   The DataFrame has 3 rows with ids 1, 2, and 3.

2. Step 2: Apply the filter condition df.id > 1

   This keeps rows with id 2 and 3, so 2 rows remain.

3. Final Answer:

   2 -> Option D

4. Quick Check:

   Rows with id > 1 = 2 [OK]

Hint: Count rows after filter condition carefully [OK]

Common Mistakes:

• Counting all rows without filter
• Misunderstanding filter condition

4. Identify the error in this Spark code snippet:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Debug').getOrCreate()
data = [(1, 'x'), (2, 'y')]
df = spark.createDataFrame(data, ['num', 'char'])
df.select('number').show()

medium

A. DataFrame creation syntax is wrong

B. SparkSession not created properly

C. Column 'number' does not exist in DataFrame

D. show() method is not valid

Solution

1. Step 1: Check DataFrame columns

   The DataFrame columns are 'num' and 'char', not 'number'.

2. Step 2: Analyze the select statement

   Selecting 'number' column causes an error because it does not exist.

3. Final Answer:

   Column 'number' does not exist in DataFrame -> Option C

4. Quick Check:

   Wrong column name = A [OK]

Hint: Verify column names before selecting [OK]

Common Mistakes:

• Using wrong column names
• Assuming show() is invalid
• Thinking SparkSession creation failed

5. You have a large dataset split across many files. Which Apache Spark feature helps you process this data efficiently in one place?

hard

A. Spark SQL to query data using SQL syntax

B. Spark Streaming for real-time data only

C. Spark MLlib for machine learning only

D. Spark GraphX for graph data only

Solution

1. Step 1: Identify the need to process large datasets in one place

   Spark SQL allows querying structured data from many files as if it were one table.

2. Step 2: Compare other Spark components

   Spark Streaming is for live data, MLlib is for machine learning, GraphX is for graph data, so they don't fit the question.

3. Final Answer:

   Spark SQL to query data using SQL syntax -> Option A

4. Quick Check:

   Query large data with Spark SQL = D [OK]

Hint: Use Spark SQL for unified big data queries [OK]

Common Mistakes:

• Confusing Spark SQL with streaming or ML
• Thinking MLlib or GraphX handle all data types