Model evaluation and metrics in Apache Spark - Mini Project: Build & Apply

Model evaluation and metrics

📖 Scenario: You work as a data scientist at an online store. You have built a model to predict if a customer will buy a product (1) or not (0). Now, you want to check how good your model is by calculating some evaluation metrics.

🎯 Goal: Build a Spark DataFrame with actual and predicted labels, set a threshold for classification, calculate accuracy and precision, and print the results.

📋 What You'll Learn

Create a Spark DataFrame with columns actual and predicted with given values

Create a threshold variable called threshold with value 0.5

Calculate accuracy and precision using Spark DataFrame operations and filters

Print the accuracy and precision values

💡 Why This Matters

🌍 Real World

Model evaluation is essential in real-world projects to check if predictions are reliable before using them in business decisions.

💼 Career

Data scientists and machine learning engineers regularly calculate metrics like accuracy and precision to improve and validate their models.

1

Create the initial Spark DataFrame

Create a Spark DataFrame called df with two columns: actual and predicted. Use these exact values: actual = [1, 0, 1, 0, 1], predicted = [0.9, 0.4, 0.2, 0.3, 0.8].

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ModelEvaluation").getOrCreate()
# Create the DataFrame df with columns actual and predicted
# Your code here

Hint

Use spark.createDataFrame with a list of tuples and column names.

2

Set the classification threshold

Create a variable called threshold and set it to 0.5. This threshold will decide if predicted values are classified as 1 or 0.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ModelEvaluation").getOrCreate()
data = [(1, 0.9), (0, 0.4), (1, 0.2), (0, 0.3), (1, 0.8)]
columns = ["actual", "predicted"]
df = spark.createDataFrame(data, columns)
# Create threshold variable and set it to 0.5
# Your code here

Hint

Just assign 0.5 to a variable named threshold.

3

Calculate accuracy and precision

Using the DataFrame df and the threshold, create a new column predicted_label that is 1 if predicted >= threshold, else 0. Then calculate accuracy as the fraction of rows where actual equals predicted_label. Calculate precision as the fraction of rows where predicted_label is 1 and actual is also 1, divided by total rows where predicted_label is 1.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when

spark = SparkSession.builder.appName("ModelEvaluation").getOrCreate()
data = [(1, 0.9), (0, 0.4), (1, 0.2), (0, 0.3), (1, 0.8)]
columns = ["actual", "predicted"]
df = spark.createDataFrame(data, columns)
threshold = 0.5
# Create predicted_label column using threshold
# Calculate accuracy and precision
# Your code here

Hint

Use withColumn and when to create predicted_label. Use filter and count() to calculate counts for accuracy and precision.

4

Print the accuracy and precision

Print the values of accuracy and precision using print. Format the output exactly as: Accuracy: {accuracy} and Precision: {precision} where the values are decimals.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when

spark = SparkSession.builder.appName("ModelEvaluation").getOrCreate()
data = [(1, 0.9), (0, 0.4), (1, 0.2), (0, 0.3), (1, 0.8)]
columns = ["actual", "predicted"]
df = spark.createDataFrame(data, columns)
threshold = 0.5
df = df.withColumn("predicted_label", when(col("predicted") >= threshold, 1).otherwise(0))
total = df.count()
correct = df.filter(col("actual") == col("predicted_label")).count()
accuracy = correct / total
true_positives = df.filter((col("predicted_label") == 1) & (col("actual") == 1)).count()
predicted_positives = df.filter(col("predicted_label") == 1).count()
precision = true_positives / predicted_positives if predicted_positives != 0 else 0
# Print accuracy and precision
# Your code here

Hint

Use print(f"Accuracy: {accuracy}") and print(f"Precision: {precision}").

Practice

1. Which metric is commonly used to evaluate a classification model's accuracy in Apache Spark?

easy

A. R-squared

B. Root Mean Squared Error

C. Accuracy

D. Mean Absolute Error

Solution

1. Step 1: Understand classification metrics

   Classification models predict categories, so accuracy measures correct predictions over total predictions.

2. Step 2: Match metric to model type

   Accuracy is the standard metric for classification, while RMSE, MAE, and R-squared are for regression.

3. Final Answer:

   Accuracy -> Option C

4. Quick Check:

   Classification accuracy = Accuracy [OK]

Hint: Use accuracy for classification, RMSE for regression [OK]

Common Mistakes:

• Confusing regression metrics with classification
• Using R-squared for classification
• Choosing MAE for classification

2. Which of the following is the correct way to create a BinaryClassificationEvaluator in Spark to evaluate areaUnderROC?

easy

A. BinaryClassificationEvaluator(metric="areaUnderROC")

B. BinaryClassificationEvaluator(metricName="areaUnderROC")

C. BinaryClassificationEvaluator(metricName='accuracy')

D. BinaryClassificationEvaluator(metric='accuracy')

Solution

1. Step 1: Recall evaluator syntax

   BinaryClassificationEvaluator uses parameter metricName to specify the metric.

2. Step 2: Check metric value

   "areaUnderROC" is a valid metricName; accuracy is not valid for this evaluator.

3. Final Answer:

   BinaryClassificationEvaluator(metricName="areaUnderROC") -> Option B

4. Quick Check:

   Use metricName="areaUnderROC" in BinaryClassificationEvaluator [OK]

Hint: Use metricName, not metric, for evaluator metrics [OK]

Common Mistakes:

• Using 'metric' instead of 'metricName'
• Passing invalid metric names like 'accuracy'
• Using single quotes inconsistently

3. Given the following Spark code, what will be the output of the evaluator?

from pyspark.ml.evaluation import RegressionEvaluator

predictions = spark.createDataFrame([
  (3.0, 2.5),
  (5.0, 5.0),
  (2.5, 3.0),
  (7.0, 8.0)
], ["label", "prediction"])

evaluator = RegressionEvaluator(metricName="rmse")
rmse = evaluator.evaluate(predictions)
print(round(rmse, 2))

medium

A. 1.00

B. 0.50

C. 0.75

D. 0.61

Solution

1. Step 1: Calculate squared errors

   Errors: (3.0-2.5)^2=0.25, (5.0-5.0)^2=0, (2.5-3.0)^2=0.25, (7.0-8.0)^2=1.0

2. Step 2: Compute RMSE

   Mean squared error = (0.25+0+0.25+1.0)/4 = 0.375; RMSE = sqrt(0.375) ≈ 0.612

3. Final Answer:

   0.61 -> Option D

4. Quick Check:

   RMSE ≈ 0.61 [OK]

Hint: RMSE is sqrt of average squared error [OK]

Common Mistakes:

• Calculating mean absolute error instead of RMSE
• Forgetting to take square root
• Rounding errors

4. You run this Spark code but get an error:

from pyspark.ml.evaluation import RegressionEvaluator

evaluator = RegressionEvaluator(metricName="accuracy")

What is the cause of the error?

medium

A. "accuracy" is not a valid metric for RegressionEvaluator

B. RegressionEvaluator requires a DataFrame input

C. metricName parameter is misspelled

D. RegressionEvaluator cannot be imported from pyspark.ml.evaluation

Solution

1. Step 1: Check valid metrics for RegressionEvaluator

   RegressionEvaluator supports metrics like rmse, mse, mae, r2 but not accuracy.

2. Step 2: Identify error cause

   Using "accuracy" causes an error because it is for classification, not regression.

3. Final Answer:

   "accuracy" is not a valid metric for RegressionEvaluator -> Option A

4. Quick Check:

   RegressionEvaluator metrics exclude accuracy [OK]

Hint: Use classification evaluator for accuracy metric [OK]

Common Mistakes:

• Confusing metric names between evaluators
• Assuming metricName typo causes error
• Ignoring evaluator import errors

5. You have a Spark ML model for binary classification. You want to evaluate it using both areaUnderROC and F1 score. Which approach correctly evaluates both metrics on the same predictions DataFrame?

hard

A. Use BinaryClassificationEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1"

B. Use BinaryClassificationEvaluator twice with metricName="areaUnderROC" and metricName="f1" respectively

C. Use RegressionEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1"

D. Use MulticlassClassificationEvaluator twice with metricName="areaUnderROC" and metricName="f1"

Solution

1. Step 1: Identify evaluators for each metric

   areaUnderROC is supported by BinaryClassificationEvaluator; F1 score is supported by MulticlassClassificationEvaluator.

2. Step 2: Combine evaluators correctly

   Use BinaryClassificationEvaluator for areaUnderROC and MulticlassClassificationEvaluator for F1 on the same predictions DataFrame.

3. Final Answer:

   Use BinaryClassificationEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1" -> Option A

4. Quick Check:

   BinaryClassificationEvaluator + MulticlassClassificationEvaluator = correct combo [OK]

Hint: Use correct evaluator per metric type [OK]

Common Mistakes:

• Using RegressionEvaluator for classification metrics
• Using BinaryClassificationEvaluator for F1 score
• Using MulticlassClassificationEvaluator for areaUnderROC