Regression models in Spark in Apache Spark - Step-by-Step Execution

Concept Flow - Regression models in Spark

Load Data

↓

Prepare Features

↓

Split Data

↓

Create Regression Model

↓

Train Model on Training Data

↓

Make Predictions on Test Data

↓

Evaluate Model Performance

↓

Use Model for Insights or Future Predictions

This flow shows how to load data, prepare it, train a regression model in Spark, then evaluate and use it.

Execution Sample

Apache Spark

from pyspark.ml.regression import LinearRegression

# Load and prepare data
lr = LinearRegression(featuresCol='features', labelCol='label')
model = lr.fit(trainingData)
predictions = model.transform(testData)

This code trains a linear regression model on training data and makes predictions on test data.

Execution Table

Step	Action	Input Data	Model State	Output
1	Load Data	Raw CSV file	No model	DataFrame with raw data
2	Prepare Features	Raw DataFrame	No model	DataFrame with 'features' vector and 'label' column
3	Split Data	Feature DataFrame	No model	trainingData and testData DataFrames
4	Create Model	trainingData	LinearRegression object created	Model ready to train
5	Train Model	trainingData	Model trained with coefficients and intercept	Trained LinearRegressionModel
6	Make Predictions	testData	Trained model	DataFrame with predictions column
7	Evaluate Model	Predictions DataFrame	Trained model	Metrics like RMSE, R2
8	Use Model	New data or predictions	Trained model	Predicted values or insights
9	End	N/A	Model trained and evaluated	Process complete

💡 Process ends after model is trained, predictions made, and evaluation completed.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 5	After Step 6	Final
data	None	DataFrame with features and label	Split into trainingData and testData	Training complete, model object created	Predictions DataFrame created	Model trained and predictions ready
model	None	LinearRegression object created	Same	Trained LinearRegressionModel	Same	Same

Key Moments - 3 Insights

Why do we split the data into training and test sets before training the model?

Splitting data (see execution_table step 3) helps us train the model on one part (trainingData) and test its accuracy on unseen data (testData), preventing overfitting.

What does the 'features' column represent in Spark regression?

The 'features' column (step 2) is a vector of numbers representing input variables for the model; Spark ML requires features in this vector format.

How do we know if the model is good after training?

After making predictions (step 6), we evaluate metrics like RMSE or R2 (step 7) to measure how well the model fits the data.

Visual Quiz - 3 Questions

Look at the execution table, at which step is the model actually trained on data?

A Step 4

B Step 5

C Step 6

D Step 3

💡 Hint

Check the 'Action' column for the step where training happens.

According to the variable tracker, what is the state of 'model' after step 5?

A No model created yet

B LinearRegression object created but not trained

C Trained LinearRegressionModel

D Predictions DataFrame

💡 Hint

Look at the 'model' row under 'After Step 5' in variable_tracker.

If we skip splitting data (step 3), what is likely to happen?

A Model trains and tests on same data, risking overfitting

B Model cannot be created

C Predictions will be more accurate

D Evaluation metrics will be zero

💡 Hint

Refer to key_moments about why splitting data is important.

Concept Snapshot

Regression models in Spark:
- Prepare data with features vector and label
- Split data into training and test sets
- Create regression model (e.g., LinearRegression)
- Train model on training data
- Predict on test data
- Evaluate with metrics like RMSE, R2
- Use model for predictions or insights

Full Transcript

This visual execution shows how to build regression models in Apache Spark. First, data is loaded and prepared by creating a features vector and label column. Then, data is split into training and test sets to avoid overfitting. A regression model object is created and trained on the training data. After training, the model makes predictions on the test data. Finally, the model's performance is evaluated using metrics such as RMSE and R2. This process helps us understand how well the model fits and how to use it for future predictions.

Practice

1. What is the main purpose of regression models in Apache Spark ML?

easy

A. To predict continuous numerical values from input data

B. To classify data into categories

C. To cluster data points into groups

D. To reduce the dimensionality of data

Solution

1. Step 1: Understand regression model goal

   Regression models are designed to predict continuous numerical outcomes based on input features.

2. Step 2: Compare with other model types

   Classification predicts categories, clustering groups data, and dimensionality reduction simplifies features, so these are not regression tasks.

3. Final Answer:

   To predict continuous numerical values from input data -> Option A

4. Quick Check:

   Regression = Predict numbers [OK]

Hint: Regression predicts numbers, classification predicts categories [OK]

Common Mistakes:

• Confusing regression with classification
• Thinking regression clusters data
• Mixing regression with dimensionality reduction

2. Which of the following is the correct way to create a Linear Regression model in Spark ML using Python?

easy

A. from pyspark.ml.classification import LinearRegression; lr = LinearRegression()

B. from pyspark.ml.regression import LinearRegression; lr = LinearRegression()

C. from pyspark.ml.regression import LogisticRegression; lr = LogisticRegression()

D. from pyspark.ml.clustering import LinearRegression; lr = LinearRegression()

Solution

1. Step 1: Identify correct import for Linear Regression

   LinearRegression is in pyspark.ml.regression module, so import must be from there.

2. Step 2: Check class name and module match

   from pyspark.ml.regression import LinearRegression; lr = LinearRegression() correctly imports LinearRegression from regression module and creates an instance.

3. Final Answer:

   from pyspark.ml.regression import LinearRegression; lr = LinearRegression() -> Option B

4. Quick Check:

   LinearRegression import from regression module [OK]

Hint: Regression models import from pyspark.ml.regression [OK]

Common Mistakes:

• Importing LinearRegression from classification or clustering modules
• Using LogisticRegression instead of LinearRegression
• Wrong module import causing runtime errors

3. Given the following Spark code, what will be the output of predictions.select('prediction').collect()?

from pyspark.ml.regression import LinearRegression
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data = spark.createDataFrame([
  (1.0, 2.0),
  (2.0, 4.1),
  (3.0, 6.0)
], ['feature', 'label'])
lr = LinearRegression(featuresCol='feature', labelCol='label')
model = lr.fit(data)
predictions = model.transform(data)

medium

A. [Row(prediction=2.0), Row(prediction=4.1), Row(prediction=6.0)]

B. [Row(prediction=2.05), Row(prediction=4.1), Row(prediction=6.15)]

C. Error due to incorrect features column type

D. [Row(prediction=1.0), Row(prediction=2.0), Row(prediction=3.0)]

Solution

1. Step 1: Check features column type

   LinearRegression expects featuresCol to be a vector column, but here 'feature' is a single float, causing an error.

2. Step 2: Understand Spark ML input requirements

   Features must be Vector type; passing a float causes runtime error during fit or transform.

3. Final Answer:

   Error due to incorrect features column type -> Option C

4. Quick Check:

   Features must be Vector, not float [OK]

Hint: Features must be Vector type, not single float [OK]

Common Mistakes:

• Passing float instead of Vector for features
• Expecting predictions without proper feature vector
• Ignoring Spark ML input data format requirements

4. You have this Spark ML code snippet that throws an error:

lr = LinearRegression(featuresCol='features', labelCol='label')
model = lr.fit(trainingData)

The error says: "java.lang.IllegalArgumentException: requirement failed: Column features must be of type vector." What is the likely cause and fix?

medium

A. The trainingData is empty; load data before fitting

B. The 'label' column is missing; add a label column to the data

C. LinearRegression does not support 'features' as input; rename the column

D. The 'features' column is not a vector; fix by using VectorAssembler to create it

Solution

1. Step 1: Analyze error message

   The error says 'features' column must be a vector, so current 'features' column is not Vector type.

2. Step 2: Fix by creating vector features

   Use VectorAssembler to combine feature columns into a single vector column named 'features'.

3. Final Answer:

   The 'features' column is not a vector; fix by using VectorAssembler to create it -> Option D

4. Quick Check:

   Features must be Vector; use VectorAssembler [OK]

Hint: Use VectorAssembler to create vector features column [OK]

Common Mistakes:

• Ignoring the need for vector type in features column
• Renaming columns without fixing data type
• Assuming label column causes this error

5. You want to predict house prices using Spark ML Linear Regression. Your dataset has columns: size (square feet), bedrooms, and price</>. Which is the correct way to prepare features and train the model?

hard

A. Use VectorAssembler to combine 'size' and 'bedrooms' into 'features', then fit LinearRegression with labelCol='price'

B. Directly use 'size' as featuresCol and 'price' as labelCol in LinearRegression

C. Use 'price' as featuresCol and 'size' as labelCol in LinearRegression

D. Train LinearRegression without feature transformation on 'bedrooms' only

Solution

1. Step 1: Combine input features

   Multiple input columns ('size', 'bedrooms') must be combined into a single vector column using VectorAssembler.

2. Step 2: Train LinearRegression model

   Use the combined 'features' vector column as featuresCol and 'price' as labelCol to fit the model.

3. Final Answer:

   Use VectorAssembler to combine 'size' and 'bedrooms' into 'features', then fit LinearRegression with labelCol='price' -> Option A

4. Quick Check:

   Combine features vector, label is target [OK]

Hint: Combine all input features into vector before training [OK]

Common Mistakes:

• Using single feature without vector assembly
• Swapping label and features columns
• Ignoring feature vector requirement