Model evaluation and metrics in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Calculate accuracy of a binary classifier in Spark

Given the following Spark code that evaluates a binary classification model, what is the printed accuracy?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.linalg import Vectors
from pyspark.sql import Row

spark = SparkSession.builder.master('local').appName('Eval').getOrCreate()
data = [
    Row(label=1.0, features=Vectors.dense([0.5, 1.0])),
    Row(label=0.0, features=Vectors.dense([1.5, 2.0])),
    Row(label=1.0, features=Vectors.dense([1.0, 1.0])),
    Row(label=0.0, features=Vectors.dense([2.0, 3.0]))
]
df = spark.createDataFrame(data)

# Simulate predictions
predictions = df.withColumn('prediction', df.label)

evaluator = BinaryClassificationEvaluator(rawPredictionCol='prediction', labelCol='label', metricName='areaUnderROC')
accuracy = predictions.filter(predictions.label == predictions.prediction).count() / predictions.count()
print(f"Accuracy: {accuracy}")

A Accuracy: 1.0

B Accuracy: 0.5

C Accuracy: 0.75

D Accuracy: 0.25

💡 Hint

Check how many rows have prediction equal to label.

Since prediction column is set equal to label, all predictions match labels, so accuracy is 1.0.

❓ data_output

intermediate

Confusion matrix counts from Spark DataFrame

Given a Spark DataFrame with columns 'label' and 'prediction' as below, what is the count of true positives (TP)?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.master('local').appName('Confusion').getOrCreate()
data = [
    (1.0, 1.0),
    (0.0, 1.0),
    (1.0, 0.0),
    (1.0, 1.0),
    (0.0, 0.0)
]
df = spark.createDataFrame(data, ['label', 'prediction'])

# Count true positives
TP = df.filter((col('label') == 1.0) & (col('prediction') == 1.0)).count()
print(f"True Positives: {TP}")

A True Positives: 3

B True Positives: 2

C True Positives: 1

D True Positives: 0

💡 Hint

True positives are rows where label and prediction both are 1.0.

Rows 1 and 4 have label=1.0 and prediction=1.0, so TP=2.

❓ visualization

advanced

ROC curve plot from Spark model predictions

You have a Spark DataFrame with columns 'label' and 'rawPrediction' (score). Which code snippet correctly plots the ROC curve using matplotlib?

Apache Spark

import matplotlib.pyplot as plt
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.sql.functions import vector_to_array, col

# Assume df_predictions has columns 'label' and 'rawPrediction' (vector of two scores)
# Extract probability of positive class
prob_df = df_predictions.withColumn('probability', vector_to_array(col('rawPrediction'))[1])

# Collect label and probability to driver
labels_probs = prob_df.select('label', 'probability').collect()
labels = [row['label'] for row in labels_probs]
probs = [row['probability'] for row in labels_probs]

# Compute ROC points
from sklearn.metrics import roc_curve
fpr, tpr, _ = roc_curve(labels, probs)

# Plot ROC
plt.plot(fpr, tpr, label='ROC curve')
plt.plot([0, 1], [0, 1], 'k--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.show()

A Plots ROC using label and prediction columns without probability scores.

B Uses rawPrediction column directly as probabilities without conversion, then plots ROC.

C Correctly extracts positive class probability and plots ROC using sklearn and matplotlib.

D Uses Spark's BinaryClassificationEvaluator to plot ROC directly without collecting data.

💡 Hint

RawPrediction is a vector; extract positive class score before plotting.

RawPrediction is a vector of scores; we must extract the positive class score to compute ROC. Then use sklearn to compute ROC points and matplotlib to plot.

🧠 Conceptual

advanced

Understanding Precision-Recall tradeoff

In a binary classification task with imbalanced classes, which statement best explains why precision-recall curves are preferred over ROC curves?

A ROC curves always give higher scores than precision-recall curves.

B ROC curves ignore false positives and only show true positive rates.

C Precision-recall curves are easier to compute than ROC curves in Spark.

D Precision-recall curves focus on the positive class performance and are more informative when classes are imbalanced.

💡 Hint

Think about what happens when negative class is much larger than positive class.

ROC curves can be overly optimistic with imbalanced data because true negative rate dominates. Precision-recall curves highlight performance on positive class.

🔧 Debug

expert

Identify error in Spark model evaluation code

What error will the following Spark code raise when evaluating a regression model's RMSE?

Apache Spark

from pyspark.ml.evaluation import RegressionEvaluator

# Assume predictions DataFrame has columns 'label' and 'prediction'
evaluator = RegressionEvaluator(labelCol='label', predictionCol='prediction', metricName='rmse')
rmse = evaluator.evaluate(predictions)
print(f"RMSE: {rmse}")

# But predictions DataFrame is empty

A java.lang.IllegalArgumentException: Dataset is empty

B ValueError: metric rmse not supported

C TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

D No error, RMSE is 0.0

💡 Hint

Think about what happens if the DataFrame has no rows.

RegressionEvaluator requires non-empty DataFrame to compute metrics. Empty DataFrame causes IllegalArgumentException in Spark.

Practice

1. Which metric is commonly used to evaluate a classification model's accuracy in Apache Spark?

easy

A. R-squared

B. Root Mean Squared Error

C. Accuracy

D. Mean Absolute Error

Solution

1. Step 1: Understand classification metrics

   Classification models predict categories, so accuracy measures correct predictions over total predictions.

2. Step 2: Match metric to model type

   Accuracy is the standard metric for classification, while RMSE, MAE, and R-squared are for regression.

3. Final Answer:

   Accuracy -> Option C

4. Quick Check:

   Classification accuracy = Accuracy [OK]

Hint: Use accuracy for classification, RMSE for regression [OK]

Common Mistakes:

• Confusing regression metrics with classification
• Using R-squared for classification
• Choosing MAE for classification

2. Which of the following is the correct way to create a BinaryClassificationEvaluator in Spark to evaluate areaUnderROC?

easy

A. BinaryClassificationEvaluator(metric="areaUnderROC")

B. BinaryClassificationEvaluator(metricName="areaUnderROC")

C. BinaryClassificationEvaluator(metricName='accuracy')

D. BinaryClassificationEvaluator(metric='accuracy')

Solution

1. Step 1: Recall evaluator syntax

   BinaryClassificationEvaluator uses parameter metricName to specify the metric.

2. Step 2: Check metric value

   "areaUnderROC" is a valid metricName; accuracy is not valid for this evaluator.

3. Final Answer:

   BinaryClassificationEvaluator(metricName="areaUnderROC") -> Option B

4. Quick Check:

   Use metricName="areaUnderROC" in BinaryClassificationEvaluator [OK]

Hint: Use metricName, not metric, for evaluator metrics [OK]

Common Mistakes:

• Using 'metric' instead of 'metricName'
• Passing invalid metric names like 'accuracy'
• Using single quotes inconsistently

3. Given the following Spark code, what will be the output of the evaluator?

from pyspark.ml.evaluation import RegressionEvaluator

predictions = spark.createDataFrame([
  (3.0, 2.5),
  (5.0, 5.0),
  (2.5, 3.0),
  (7.0, 8.0)
], ["label", "prediction"])

evaluator = RegressionEvaluator(metricName="rmse")
rmse = evaluator.evaluate(predictions)
print(round(rmse, 2))

medium

A. 1.00

B. 0.50

C. 0.75

D. 0.61

Solution

1. Step 1: Calculate squared errors

   Errors: (3.0-2.5)^2=0.25, (5.0-5.0)^2=0, (2.5-3.0)^2=0.25, (7.0-8.0)^2=1.0

2. Step 2: Compute RMSE

   Mean squared error = (0.25+0+0.25+1.0)/4 = 0.375; RMSE = sqrt(0.375) ≈ 0.612

3. Final Answer:

   0.61 -> Option D

4. Quick Check:

   RMSE ≈ 0.61 [OK]

Hint: RMSE is sqrt of average squared error [OK]

Common Mistakes:

• Calculating mean absolute error instead of RMSE
• Forgetting to take square root
• Rounding errors

4. You run this Spark code but get an error:

from pyspark.ml.evaluation import RegressionEvaluator

evaluator = RegressionEvaluator(metricName="accuracy")

What is the cause of the error?

medium

A. "accuracy" is not a valid metric for RegressionEvaluator

B. RegressionEvaluator requires a DataFrame input

C. metricName parameter is misspelled

D. RegressionEvaluator cannot be imported from pyspark.ml.evaluation

Solution

1. Step 1: Check valid metrics for RegressionEvaluator

   RegressionEvaluator supports metrics like rmse, mse, mae, r2 but not accuracy.

2. Step 2: Identify error cause

   Using "accuracy" causes an error because it is for classification, not regression.

3. Final Answer:

   "accuracy" is not a valid metric for RegressionEvaluator -> Option A

4. Quick Check:

   RegressionEvaluator metrics exclude accuracy [OK]

Hint: Use classification evaluator for accuracy metric [OK]

Common Mistakes:

• Confusing metric names between evaluators
• Assuming metricName typo causes error
• Ignoring evaluator import errors

5. You have a Spark ML model for binary classification. You want to evaluate it using both areaUnderROC and F1 score. Which approach correctly evaluates both metrics on the same predictions DataFrame?

hard

A. Use BinaryClassificationEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1"

B. Use BinaryClassificationEvaluator twice with metricName="areaUnderROC" and metricName="f1" respectively

C. Use RegressionEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1"

D. Use MulticlassClassificationEvaluator twice with metricName="areaUnderROC" and metricName="f1"

Solution

1. Step 1: Identify evaluators for each metric

   areaUnderROC is supported by BinaryClassificationEvaluator; F1 score is supported by MulticlassClassificationEvaluator.

2. Step 2: Combine evaluators correctly

   Use BinaryClassificationEvaluator for areaUnderROC and MulticlassClassificationEvaluator for F1 on the same predictions DataFrame.

3. Final Answer:

   Use BinaryClassificationEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1" -> Option A

4. Quick Check:

   BinaryClassificationEvaluator + MulticlassClassificationEvaluator = correct combo [OK]

Hint: Use correct evaluator per metric type [OK]

Common Mistakes:

• Using RegressionEvaluator for classification metrics
• Using BinaryClassificationEvaluator for F1 score
• Using MulticlassClassificationEvaluator for areaUnderROC