Model evaluation and metrics in Apache Spark - Step-by-Step Execution

Concept Flow - Model evaluation and metrics

Train Model

↓

Make Predictions

↓

Compare Predictions with Actual

↓

Calculate Metrics

↓

Evaluate Model Performance

↓

Decide: Improve or Deploy?

↓

Improve

The flow shows training a model, making predictions, calculating metrics by comparing predictions to actual values, then deciding to improve or deploy the model.

Execution Sample

Apache Spark

from pyspark.ml.evaluation import BinaryClassificationEvaluator

predictions = model.transform(test_data)
evaluator = BinaryClassificationEvaluator(labelCol='label')
auc = evaluator.evaluate(predictions)
print(f'AUC: {auc}')

This code makes predictions on test data, evaluates using AUC metric, and prints the result.

Execution Table

Step	Action	Input Data	Output/Result	Notes
1	Train model	training dataset	trained model object	Model learns patterns from training data
2	Make predictions	test dataset + trained model	predictions DataFrame with 'prediction' column	Model predicts labels for test data
3	Initialize evaluator	BinaryClassificationEvaluator with labelCol='label'	evaluator object	Evaluator ready to compute metrics
4	Evaluate predictions	predictions DataFrame	AUC score (e.g. 0.85)	Calculates Area Under ROC Curve
5	Print metric	AUC score	Console output: 'AUC: 0.85'	Shows model performance
6	Decision	AUC score	Improve or deploy model	If AUC low, improve; else deploy

💡 Evaluation ends after metric calculation and decision step

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 4	Final
model	None	trained model object	trained model object	trained model object	trained model object
predictions	None	None	DataFrame with predictions	DataFrame with predictions	DataFrame with predictions
evaluator	None	None	evaluator object	evaluator object	evaluator object
auc	None	None	None	0.85	0.85

Key Moments - 3 Insights

Why do we need to separate training and test data before evaluation?

Because the model learns from training data, evaluating on test data shows how well it generalizes to new unseen data, as seen in execution_table step 1 and 2.

What does the AUC metric tell us about the model?

AUC measures how well the model distinguishes between classes; higher AUC means better performance. This is shown in step 4 where AUC is calculated.

Why can't we just look at predictions without metrics?

Predictions alone don't quantify accuracy or errors. Metrics like AUC summarize performance clearly, as shown in step 5 where the metric is printed.

Visual Quiz - 3 Questions

Look at the execution_table, what is stored in the 'predictions' variable after Step 2?

A A DataFrame with predicted labels

B The trained model object

C The raw test dataset without predictions

D The evaluation metric score

💡 Hint

Check the 'Output/Result' column for Step 2 in execution_table

At which step does the model's performance metric get calculated?

A Step 3

B Step 4

C Step 1

D Step 6

💡 Hint

Look for 'Calculate Metrics' or 'Evaluate predictions' in execution_table

If the AUC score is low, what is the likely next action according to the flow?

A Deploy the model immediately

B Ignore the metric and retrain

C Improve the model

D Delete the test data

💡 Hint

Refer to the decision step in concept_flow and execution_table step 6

Concept Snapshot

Model evaluation steps:
1. Train model on training data
2. Predict on test data
3. Use evaluator to calculate metrics (e.g., AUC)
4. Metrics show model quality
5. Decide to improve or deploy based on metrics

Full Transcript

In model evaluation, we first train a model using training data. Then, we use this model to predict labels on test data. Next, we compare these predictions with actual labels using evaluation metrics like AUC. The AUC metric tells us how well the model distinguishes between classes. Based on the metric value, we decide whether to improve the model or deploy it. This process ensures the model performs well on new, unseen data.

Practice

1. Which metric is commonly used to evaluate a classification model's accuracy in Apache Spark?

easy

A. R-squared

B. Root Mean Squared Error

C. Accuracy

D. Mean Absolute Error

Solution

1. Step 1: Understand classification metrics

   Classification models predict categories, so accuracy measures correct predictions over total predictions.

2. Step 2: Match metric to model type

   Accuracy is the standard metric for classification, while RMSE, MAE, and R-squared are for regression.

3. Final Answer:

   Accuracy -> Option C

4. Quick Check:

   Classification accuracy = Accuracy [OK]

Hint: Use accuracy for classification, RMSE for regression [OK]

Common Mistakes:

• Confusing regression metrics with classification
• Using R-squared for classification
• Choosing MAE for classification

2. Which of the following is the correct way to create a BinaryClassificationEvaluator in Spark to evaluate areaUnderROC?

easy

A. BinaryClassificationEvaluator(metric="areaUnderROC")

B. BinaryClassificationEvaluator(metricName="areaUnderROC")

C. BinaryClassificationEvaluator(metricName='accuracy')

D. BinaryClassificationEvaluator(metric='accuracy')

Solution

1. Step 1: Recall evaluator syntax

   BinaryClassificationEvaluator uses parameter metricName to specify the metric.

2. Step 2: Check metric value

   "areaUnderROC" is a valid metricName; accuracy is not valid for this evaluator.

3. Final Answer:

   BinaryClassificationEvaluator(metricName="areaUnderROC") -> Option B

4. Quick Check:

   Use metricName="areaUnderROC" in BinaryClassificationEvaluator [OK]

Hint: Use metricName, not metric, for evaluator metrics [OK]

Common Mistakes:

• Using 'metric' instead of 'metricName'
• Passing invalid metric names like 'accuracy'
• Using single quotes inconsistently

3. Given the following Spark code, what will be the output of the evaluator?

from pyspark.ml.evaluation import RegressionEvaluator

predictions = spark.createDataFrame([
  (3.0, 2.5),
  (5.0, 5.0),
  (2.5, 3.0),
  (7.0, 8.0)
], ["label", "prediction"])

evaluator = RegressionEvaluator(metricName="rmse")
rmse = evaluator.evaluate(predictions)
print(round(rmse, 2))

medium

A. 1.00

B. 0.50

C. 0.75

D. 0.61

Solution

1. Step 1: Calculate squared errors

   Errors: (3.0-2.5)^2=0.25, (5.0-5.0)^2=0, (2.5-3.0)^2=0.25, (7.0-8.0)^2=1.0

2. Step 2: Compute RMSE

   Mean squared error = (0.25+0+0.25+1.0)/4 = 0.375; RMSE = sqrt(0.375) ≈ 0.612

3. Final Answer:

   0.61 -> Option D

4. Quick Check:

   RMSE ≈ 0.61 [OK]

Hint: RMSE is sqrt of average squared error [OK]

Common Mistakes:

• Calculating mean absolute error instead of RMSE
• Forgetting to take square root
• Rounding errors

4. You run this Spark code but get an error:

from pyspark.ml.evaluation import RegressionEvaluator

evaluator = RegressionEvaluator(metricName="accuracy")

What is the cause of the error?

medium

A. "accuracy" is not a valid metric for RegressionEvaluator

B. RegressionEvaluator requires a DataFrame input

C. metricName parameter is misspelled

D. RegressionEvaluator cannot be imported from pyspark.ml.evaluation

Solution

1. Step 1: Check valid metrics for RegressionEvaluator

   RegressionEvaluator supports metrics like rmse, mse, mae, r2 but not accuracy.

2. Step 2: Identify error cause

   Using "accuracy" causes an error because it is for classification, not regression.

3. Final Answer:

   "accuracy" is not a valid metric for RegressionEvaluator -> Option A

4. Quick Check:

   RegressionEvaluator metrics exclude accuracy [OK]

Hint: Use classification evaluator for accuracy metric [OK]

Common Mistakes:

• Confusing metric names between evaluators
• Assuming metricName typo causes error
• Ignoring evaluator import errors

5. You have a Spark ML model for binary classification. You want to evaluate it using both areaUnderROC and F1 score. Which approach correctly evaluates both metrics on the same predictions DataFrame?

hard

A. Use BinaryClassificationEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1"

B. Use BinaryClassificationEvaluator twice with metricName="areaUnderROC" and metricName="f1" respectively

C. Use RegressionEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1"

D. Use MulticlassClassificationEvaluator twice with metricName="areaUnderROC" and metricName="f1"

Solution

1. Step 1: Identify evaluators for each metric

   areaUnderROC is supported by BinaryClassificationEvaluator; F1 score is supported by MulticlassClassificationEvaluator.

2. Step 2: Combine evaluators correctly

   Use BinaryClassificationEvaluator for areaUnderROC and MulticlassClassificationEvaluator for F1 on the same predictions DataFrame.

3. Final Answer:

   Use BinaryClassificationEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1" -> Option A

4. Quick Check:

   BinaryClassificationEvaluator + MulticlassClassificationEvaluator = correct combo [OK]

Hint: Use correct evaluator per metric type [OK]

Common Mistakes:

• Using RegressionEvaluator for classification metrics
• Using BinaryClassificationEvaluator for F1 score
• Using MulticlassClassificationEvaluator for areaUnderROC