Hyperparameter tuning with CrossValidator in Apache Spark - Step-by-Step Execution

Concept Flow - Hyperparameter tuning with CrossValidator

Start: Define Model and ParamGrid

↓

Create CrossValidator with Estimator, ParamGrid, Evaluator

↓

Split Data into Folds

↓

For each Param Combination

↓

For each Fold

↓

Train Model

↓

Collect Metrics

↓

Average Metrics over Folds

↓

Select Best Param Set

↓

Train Final Model

↓

Done

This flow shows how CrossValidator tests different hyperparameter sets by training and validating models on data folds, then selects the best parameters.

Execution Sample

Apache Spark

from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import BinaryClassificationEvaluator

paramGrid = ParamGridBuilder().addGrid(lr.regParam, [0.1, 0.01]).build()
cv = CrossValidator(estimator=lr, estimatorParamMaps=paramGrid, evaluator=BinaryClassificationEvaluator(), numFolds=2)
cvModel = cv.fit(trainingData)

This code sets up a logistic regression model with two regularization parameters, uses 2-fold cross-validation to find the best parameter, and fits the model.

Execution Table

Step	Fold	Param regParam	Train Model	Validate Model	Metric (AUC)	Average Metric	Best Param?
1	Fold 1	0.1	Train on Fold 1	Validate on Fold 2	0.85
2	Fold 2	0.1	Train on Fold 2	Validate on Fold 1	0.80	0.825	No
3	Fold 1	0.01	Train on Fold 1	Validate on Fold 2	0.88
4	Fold 2	0.01	Train on Fold 2	Validate on Fold 1	0.83	0.855	Yes
5	-	-	Select param regParam=0.01	-	-	-	Best average AUC 0.855
6	-	-	Train final model on full data	-	-	-	-

💡 All parameter sets evaluated on all folds; best parameter selected based on highest average AUC.

Variable Tracker

Variable	Start	After Step 2	After Step 4	Final
regParam	None	0.1	0.01	0.01
Metric (AUC)	None	0.825	0.855	0.855
Best Param	None	No	Yes	Yes

Key Moments - 3 Insights

Why do we average metrics over folds instead of using just one fold's result?

Averaging metrics over folds (see execution_table rows 2 and 4) gives a more reliable estimate of model performance by reducing bias from any single fold.

How does CrossValidator decide which parameter is best?

It compares the average metric (AUC) across folds for each parameter set (rows 2 and 4) and selects the one with the highest average (row 5).

What happens after the best parameter is selected?

The final model is trained on the full dataset using the best parameter (row 6), ensuring it uses all available data.

Visual Quiz - 3 Questions

Look at the execution table, what is the average AUC for regParam=0.1 after both folds?

A 0.825

B 0.85

C 0.80

D 0.855

💡 Hint

Check the 'Average Metric' column after Step 2 in the execution_table.

At which step does the CrossValidator select the best parameter?

A Step 4

B Step 5

C Step 3

D Step 6

💡 Hint

Look for the step mentioning 'Select param' in the execution_table.

If we added a third regParam value, how would the execution_table change?

A No change in rows

B Fewer rows because folds reduce

C More rows for training and validation with the new parameter

D Only one fold would be used

💡 Hint

Each parameter value is tested on all folds, so adding one adds more rows (see current rows 1-4).

Concept Snapshot

Hyperparameter tuning with CrossValidator:
- Define model and parameter grid
- Use CrossValidator with estimator, param grid, evaluator, and folds
- Train and validate model on each fold for each param
- Average metrics to select best param
- Train final model on full data with best param

Full Transcript

Hyperparameter tuning with CrossValidator involves defining a model and a set of parameters to test. The CrossValidator splits data into folds and trains the model on each fold with each parameter set. It then validates on the remaining fold and collects performance metrics like AUC. These metrics are averaged across folds to find the best parameter. Finally, the model is retrained on the full dataset using the best parameter. This process helps find the best settings for the model to perform well on unseen data.

Practice

1. What is the main purpose of using CrossValidator in Apache Spark's machine learning pipeline?

easy

A. To visualize the model's performance metrics

B. To train the model on the entire dataset without validation

C. To clean and preprocess the input data

D. To find the best hyperparameters by testing multiple data splits

Solution

1. Step 1: Understand CrossValidator's role

   CrossValidator splits data into training and validation sets multiple times to evaluate model performance.

2. Step 2: Identify the purpose of hyperparameter tuning

   It tests different hyperparameter combinations to find the best model settings.

3. Final Answer:

   To find the best hyperparameters by testing multiple data splits -> Option D

4. Quick Check:

   CrossValidator = Best hyperparameters [OK]

Hint: CrossValidator tests parameters on data splits to find best model [OK]

Common Mistakes:

• Thinking CrossValidator trains on full data without validation
• Confusing CrossValidator with data cleaning tools
• Assuming CrossValidator is for visualization only

2. Which of the following is the correct way to create a parameter grid using ParamGridBuilder in Spark ML?

easy

A. paramGrid = ParamGridBuilder().addGrid(model.maxDepth, [2, 5]).build()

B. paramGrid = ParamGridBuilder.addGrid(model.maxDepth, [2, 5]).build()

C. paramGrid = ParamGridBuilder().addGrid(model.maxDepth = [2, 5]).build()

D. paramGrid = ParamGridBuilder().addGrid([2, 5], model.maxDepth).build()

Solution

1. Step 1: Recall ParamGridBuilder syntax

   ParamGridBuilder is instantiated, then addGrid is called with parameter and list of values.

2. Step 2: Check each option's syntax

   paramGrid = ParamGridBuilder().addGrid(model.maxDepth, [2, 5]).build() correctly calls ParamGridBuilder() then addGrid(model.maxDepth, [2, 5]) and finally build().

3. Final Answer:

   paramGrid = ParamGridBuilder().addGrid(model.maxDepth, [2, 5]).build() -> Option A

4. Quick Check:

   ParamGridBuilder() + addGrid(param, values) + build() [OK]

Hint: Use ParamGridBuilder() then addGrid(param, values) then build() [OK]

Common Mistakes:

• Calling addGrid without parentheses on ParamGridBuilder
• Using assignment inside addGrid instead of parameters
• Swapping parameter and values order in addGrid

3. Given the following code snippet, what will be the output of cvModel.bestModel.getMaxDepth()?

from pyspark.ml.classification import DecisionTreeClassifier
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import BinaryClassificationEvaluator

model = DecisionTreeClassifier()
paramGrid = ParamGridBuilder().addGrid(model.maxDepth, [2, 4]).build()
evaluator = BinaryClassificationEvaluator()

cv = CrossValidator(estimator=model, estimatorParamMaps=paramGrid, evaluator=evaluator, numFolds=2)
cvModel = cv.fit(trainingData)

best_depth = cvModel.bestModel.getMaxDepth()

medium

A. The average of 2 and 4, which is 3

B. 2 or 4 depending on which depth gave better validation metric

C. An error because getMaxDepth() is not a valid method

D. Always 2 because it is the first value in the grid

Solution

1. Step 1: Understand CrossValidator fitting

   CrossValidator tries all parameter combinations and selects the best model based on evaluator metric.

2. Step 2: Interpret bestModel.getMaxDepth()

   The bestModel will have maxDepth set to the value that gave the best validation score, either 2 or 4.

3. Final Answer:

   2 or 4 depending on which depth gave better validation metric -> Option B

4. Quick Check:

   bestModel maxDepth = best param tested [OK]

Hint: bestModel has hyperparameter with best validation score [OK]

Common Mistakes:

• Assuming bestModel always uses first grid value
• Thinking getMaxDepth() method does not exist
• Calculating average of grid values instead of selecting best

4. Identify the error in this code snippet for hyperparameter tuning with CrossValidator:

from pyspark.ml.classification import LogisticRegression
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import BinaryClassificationEvaluator

lr = LogisticRegression()
paramGrid = ParamGridBuilder().addGrid(lr.regParam, [0.1, 0.01])

evaluator = BinaryClassificationEvaluator()
cv = CrossValidator(estimator=lr, estimatorParamMaps=paramGrid, evaluator=evaluator, numFolds=3)
cvModel = cv.fit(trainingData)

medium

A. Missing .build() call after ParamGridBuilder()

B. BinaryClassificationEvaluator cannot be used with LogisticRegression

C. numFolds must be at least 5 for CrossValidator

D. regParam values must be strings, not floats

Solution

1. Step 1: Check ParamGridBuilder usage

   ParamGridBuilder requires calling .build() to create the parameter grid list.

2. Step 2: Identify missing method call

   The code misses .build() after addGrid, so paramGrid is not a valid parameter map list.

3. Final Answer:

   Missing .build() call after ParamGridBuilder() -> Option A

4. Quick Check:

   ParamGridBuilder().addGrid(...).build() required [OK]

Hint: Always end ParamGridBuilder with .build() to create grid [OK]

Common Mistakes:

• Forgetting to call .build() after addGrid
• Using wrong evaluator for classification
• Setting numFolds less than 2 (valid but uncommon)

5. You want to tune two hyperparameters of a RandomForestClassifier: numTrees with values [10, 20] and maxDepth with values [5, 10]. How many total models will CrossValidator train if numFolds=3?

hard

A. 30 models

B. 6 models

C. 12 models

D. 60 models

Solution

1. Step 1: Calculate total hyperparameter combinations

   numTrees has 2 values, maxDepth has 2 values, so total combinations = 2 * 2 = 4.

2. Step 2: Multiply by number of folds

   CrossValidator trains each combination on each fold, so total models = 4 * 3 = 12.

3. Step 3: Re-check calculation

   Wait, the question asks how many total models trained. Each fold trains one model per combination, so total = combinations * folds = 4 * 3 = 12.

4. Final Answer:

   12 models -> Option C

5. Quick Check:

   Combinations * folds = 12 [OK]

Hint: Multiply parameter combos by folds for total models trained [OK]

Common Mistakes:

• Multiplying number of values incorrectly
• Forgetting to multiply by number of folds
• Confusing total models with total parameter combos