Model evaluation and metrics in Apache Spark

Introduction

We use model evaluation and metrics to check how well a machine learning model works. It helps us know if the model makes good predictions or needs improvement.

After training a model to see how accurate it is on new data.

When comparing two or more models to pick the best one.

To measure errors and understand where the model makes mistakes.

To track model performance over time and detect if it gets worse.

Before deploying a model to make sure it meets quality standards.

Syntax

Apache Spark

from pyspark.ml.evaluation import RegressionEvaluator, BinaryClassificationEvaluator, MulticlassClassificationEvaluator

# Create evaluator object
# For regression:
evaluator = RegressionEvaluator(labelCol='label', predictionCol='prediction', metricName='rmse')

# For binary classification:
evaluator = BinaryClassificationEvaluator(labelCol='label', rawPredictionCol='rawPrediction', metricName='areaUnderROC')

# For multiclass classification:
evaluator = MulticlassClassificationEvaluator(labelCol='label', predictionCol='prediction', metricName='accuracy')

# Evaluate metric
metric_value = evaluator.evaluate(predictions)

Choose the evaluator based on your problem type: regression, binary classification, or multiclass classification.

Common metrics include RMSE for regression, area under ROC for binary classification, and accuracy for multiclass classification.

Examples

This example shows how to calculate RMSE for a regression model's predictions.

Apache Spark

from pyspark.ml.evaluation import RegressionEvaluator

evaluator = RegressionEvaluator(labelCol='label', predictionCol='prediction', metricName='rmse')
rmse = evaluator.evaluate(predictions)
print(f"Root Mean Squared Error (RMSE): {rmse}")

This example calculates the Area Under the ROC curve for a binary classification model.

Apache Spark

from pyspark.ml.evaluation import BinaryClassificationEvaluator

evaluator = BinaryClassificationEvaluator(labelCol='label', rawPredictionCol='rawPrediction', metricName='areaUnderROC')
auc = evaluator.evaluate(predictions)
print(f"Area Under ROC: {auc}")

This example calculates accuracy for a multiclass classification model.

Apache Spark

from pyspark.ml.evaluation import MulticlassClassificationEvaluator

evaluator = MulticlassClassificationEvaluator(labelCol='label', predictionCol='prediction', metricName='accuracy')
accuracy = evaluator.evaluate(predictions)
print(f"Accuracy: {accuracy}")

Sample Program

This program trains a simple logistic regression model on sample data, makes predictions, and evaluates the model using the Area Under ROC metric.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import BinaryClassificationEvaluator

# Start Spark session
spark = SparkSession.builder.appName('ModelEvaluationExample').getOrCreate()

# Create sample data
training = spark.createDataFrame([
    (1.0, [0.0, 1.1, 0.1]),
    (0.0, [2.0, 1.0, -1.0]),
    (0.0, [2.0, 1.3, 1.0]),
    (1.0, [0.0, 1.2, -0.5])
], ['label', 'features'])

# Train logistic regression model
lr = LogisticRegression(maxIter=10, regParam=0.01)
model = lr.fit(training)

# Make predictions
predictions = model.transform(training)

# Evaluate model using Area Under ROC
evaluator = BinaryClassificationEvaluator(labelCol='label', rawPredictionCol='rawPrediction', metricName='areaUnderROC')
auc = evaluator.evaluate(predictions)
print(f"Area Under ROC: {auc}")

spark.stop()

Important Notes

Always use a separate test dataset for evaluation to get an honest measure of model performance.

Different metrics tell different stories; choose the one that fits your problem goal.

For classification, rawPredictionCol is often used for metrics like areaUnderROC, not predictionCol.

Summary

Model evaluation helps check how well a model predicts new data.

Use the right evaluator and metric for your problem type.

Evaluate on test data to avoid overestimating model quality.

Practice

1. Which metric is commonly used to evaluate a classification model's accuracy in Apache Spark?

easy

A. R-squared

B. Root Mean Squared Error

C. Accuracy

D. Mean Absolute Error

Solution

1. Step 1: Understand classification metrics

   Classification models predict categories, so accuracy measures correct predictions over total predictions.

2. Step 2: Match metric to model type

   Accuracy is the standard metric for classification, while RMSE, MAE, and R-squared are for regression.

3. Final Answer:

   Accuracy -> Option C

4. Quick Check:

   Classification accuracy = Accuracy [OK]

Hint: Use accuracy for classification, RMSE for regression [OK]

Common Mistakes:

• Confusing regression metrics with classification
• Using R-squared for classification
• Choosing MAE for classification

2. Which of the following is the correct way to create a BinaryClassificationEvaluator in Spark to evaluate areaUnderROC?

easy

A. BinaryClassificationEvaluator(metric="areaUnderROC")

B. BinaryClassificationEvaluator(metricName="areaUnderROC")

C. BinaryClassificationEvaluator(metricName='accuracy')

D. BinaryClassificationEvaluator(metric='accuracy')

Solution

1. Step 1: Recall evaluator syntax

   BinaryClassificationEvaluator uses parameter metricName to specify the metric.

2. Step 2: Check metric value

   "areaUnderROC" is a valid metricName; accuracy is not valid for this evaluator.

3. Final Answer:

   BinaryClassificationEvaluator(metricName="areaUnderROC") -> Option B

4. Quick Check:

   Use metricName="areaUnderROC" in BinaryClassificationEvaluator [OK]

Hint: Use metricName, not metric, for evaluator metrics [OK]

Common Mistakes:

• Using 'metric' instead of 'metricName'
• Passing invalid metric names like 'accuracy'
• Using single quotes inconsistently

3. Given the following Spark code, what will be the output of the evaluator?

from pyspark.ml.evaluation import RegressionEvaluator

predictions = spark.createDataFrame([
  (3.0, 2.5),
  (5.0, 5.0),
  (2.5, 3.0),
  (7.0, 8.0)
], ["label", "prediction"])

evaluator = RegressionEvaluator(metricName="rmse")
rmse = evaluator.evaluate(predictions)
print(round(rmse, 2))

medium

A. 1.00

B. 0.50

C. 0.75

D. 0.61

Solution

1. Step 1: Calculate squared errors

   Errors: (3.0-2.5)^2=0.25, (5.0-5.0)^2=0, (2.5-3.0)^2=0.25, (7.0-8.0)^2=1.0

2. Step 2: Compute RMSE

   Mean squared error = (0.25+0+0.25+1.0)/4 = 0.375; RMSE = sqrt(0.375) ≈ 0.612

3. Final Answer:

   0.61 -> Option D

4. Quick Check:

   RMSE ≈ 0.61 [OK]

Hint: RMSE is sqrt of average squared error [OK]

Common Mistakes:

• Calculating mean absolute error instead of RMSE
• Forgetting to take square root
• Rounding errors

4. You run this Spark code but get an error:

from pyspark.ml.evaluation import RegressionEvaluator

evaluator = RegressionEvaluator(metricName="accuracy")

What is the cause of the error?

medium

A. "accuracy" is not a valid metric for RegressionEvaluator

B. RegressionEvaluator requires a DataFrame input

C. metricName parameter is misspelled

D. RegressionEvaluator cannot be imported from pyspark.ml.evaluation

Solution

1. Step 1: Check valid metrics for RegressionEvaluator

   RegressionEvaluator supports metrics like rmse, mse, mae, r2 but not accuracy.

2. Step 2: Identify error cause

   Using "accuracy" causes an error because it is for classification, not regression.

3. Final Answer:

   "accuracy" is not a valid metric for RegressionEvaluator -> Option A

4. Quick Check:

   RegressionEvaluator metrics exclude accuracy [OK]

Hint: Use classification evaluator for accuracy metric [OK]

Common Mistakes:

• Confusing metric names between evaluators
• Assuming metricName typo causes error
• Ignoring evaluator import errors

5. You have a Spark ML model for binary classification. You want to evaluate it using both areaUnderROC and F1 score. Which approach correctly evaluates both metrics on the same predictions DataFrame?

hard

A. Use BinaryClassificationEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1"

B. Use BinaryClassificationEvaluator twice with metricName="areaUnderROC" and metricName="f1" respectively

C. Use RegressionEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1"

D. Use MulticlassClassificationEvaluator twice with metricName="areaUnderROC" and metricName="f1"

Solution

1. Step 1: Identify evaluators for each metric

   areaUnderROC is supported by BinaryClassificationEvaluator; F1 score is supported by MulticlassClassificationEvaluator.

2. Step 2: Combine evaluators correctly

   Use BinaryClassificationEvaluator for areaUnderROC and MulticlassClassificationEvaluator for F1 on the same predictions DataFrame.

3. Final Answer:

   Use BinaryClassificationEvaluator with metricName="areaUnderROC" and MulticlassClassificationEvaluator with metricName="f1" -> Option A

4. Quick Check:

   BinaryClassificationEvaluator + MulticlassClassificationEvaluator = correct combo [OK]

Hint: Use correct evaluator per metric type [OK]

Common Mistakes:

• Using RegressionEvaluator for classification metrics
• Using BinaryClassificationEvaluator for F1 score
• Using MulticlassClassificationEvaluator for areaUnderROC