Regression models in Spark in Apache Spark - Mini Project: Build & Apply

Regression models in Spark

📖 Scenario: You work as a data analyst for a real estate company. You want to predict house prices based on the size of the house using a simple regression model in Apache Spark.

🎯 Goal: Build a simple linear regression model in Spark to predict house prices from house sizes.

📋 What You'll Learn

Create a Spark DataFrame with house sizes and prices

Set up a LinearRegression model with default parameters

Fit the model to the data

Print the model coefficients and intercept

💡 Why This Matters

🌍 Real World

Real estate companies use regression models to estimate house prices based on features like size, location, and condition.

💼 Career

Data scientists and analysts build regression models in Spark to handle large datasets and make predictions for business decisions.

1

Create the Spark DataFrame with house data

Create a Spark DataFrame called data with two columns: size and price. Use these exact rows: (1000, 300000), (1500, 400000), (2000, 500000), (2500, 600000), (3000, 700000).

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("HousePricePrediction").getOrCreate()
# Create the DataFrame called data with columns size and price
# Your code here

Hint

Use spark.createDataFrame with a list of tuples and specify the column names as a list.

2

Prepare features column for regression

Create a new DataFrame called df by transforming data to have a features column with the size values as vectors, and keep the price column. Use VectorAssembler from pyspark.ml.feature.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("HousePricePrediction").getOrCreate()
data = spark.createDataFrame([
    (1000, 300000),
    (1500, 400000),
    (2000, 500000),
    (2500, 600000),
    (3000, 700000)
], ["size", "price"])

from pyspark.ml.feature import VectorAssembler
# Create VectorAssembler to transform size into features
# Your code here

Hint

Use VectorAssembler to convert the size column into a features vector column.

3

Create and fit the Linear Regression model

Import LinearRegression from pyspark.ml.regression. Create a LinearRegression object called lr with default parameters. Fit the model on df and save the result in lr_model.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("HousePricePrediction").getOrCreate()
data = spark.createDataFrame([
    (1000, 300000),
    (1500, 400000),
    (2000, 500000),
    (2500, 600000),
    (3000, 700000)
], ["size", "price"])

from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(inputCols=["size"], outputCol="features")
df = assembler.transform(data).select("features", "price")

from pyspark.ml.regression import LinearRegression
# Create LinearRegression object lr
# Fit lr on df and save to lr_model
# Your code here

Hint

Create the LinearRegression object with featuresCol="features" and labelCol="price". Then call fit on df.

4

Print the model coefficients and intercept

Print the coefficient and intercept of lr_model using print. Use lr_model.coefficients and lr_model.intercept.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("HousePricePrediction").getOrCreate()
data = spark.createDataFrame([
    (1000, 300000),
    (1500, 400000),
    (2000, 500000),
    (2500, 600000),
    (3000, 700000)
], ["size", "price"])

from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(inputCols=["size"], outputCol="features")
df = assembler.transform(data).select("features", "price")

from pyspark.ml.regression import LinearRegression
lr = LinearRegression(featuresCol="features", labelCol="price")
lr_model = lr.fit(df)
# Print the coefficient and intercept of lr_model
# Your code here

Hint

Use print(f"Coefficient: {lr_model.coefficients[0]}") and print(f"Intercept: {lr_model.intercept}") to display the results.

Practice

1. What is the main purpose of regression models in Apache Spark ML?

easy

A. To predict continuous numerical values from input data

B. To classify data into categories

C. To cluster data points into groups

D. To reduce the dimensionality of data

Solution

1. Step 1: Understand regression model goal

   Regression models are designed to predict continuous numerical outcomes based on input features.

2. Step 2: Compare with other model types

   Classification predicts categories, clustering groups data, and dimensionality reduction simplifies features, so these are not regression tasks.

3. Final Answer:

   To predict continuous numerical values from input data -> Option A

4. Quick Check:

   Regression = Predict numbers [OK]

Hint: Regression predicts numbers, classification predicts categories [OK]

Common Mistakes:

• Confusing regression with classification
• Thinking regression clusters data
• Mixing regression with dimensionality reduction

2. Which of the following is the correct way to create a Linear Regression model in Spark ML using Python?

easy

A. from pyspark.ml.classification import LinearRegression; lr = LinearRegression()

B. from pyspark.ml.regression import LinearRegression; lr = LinearRegression()

C. from pyspark.ml.regression import LogisticRegression; lr = LogisticRegression()

D. from pyspark.ml.clustering import LinearRegression; lr = LinearRegression()

Solution

1. Step 1: Identify correct import for Linear Regression

   LinearRegression is in pyspark.ml.regression module, so import must be from there.

2. Step 2: Check class name and module match

   from pyspark.ml.regression import LinearRegression; lr = LinearRegression() correctly imports LinearRegression from regression module and creates an instance.

3. Final Answer:

   from pyspark.ml.regression import LinearRegression; lr = LinearRegression() -> Option B

4. Quick Check:

   LinearRegression import from regression module [OK]

Hint: Regression models import from pyspark.ml.regression [OK]

Common Mistakes:

• Importing LinearRegression from classification or clustering modules
• Using LogisticRegression instead of LinearRegression
• Wrong module import causing runtime errors

3. Given the following Spark code, what will be the output of predictions.select('prediction').collect()?

from pyspark.ml.regression import LinearRegression
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data = spark.createDataFrame([
  (1.0, 2.0),
  (2.0, 4.1),
  (3.0, 6.0)
], ['feature', 'label'])
lr = LinearRegression(featuresCol='feature', labelCol='label')
model = lr.fit(data)
predictions = model.transform(data)

medium

A. [Row(prediction=2.0), Row(prediction=4.1), Row(prediction=6.0)]

B. [Row(prediction=2.05), Row(prediction=4.1), Row(prediction=6.15)]

C. Error due to incorrect features column type

D. [Row(prediction=1.0), Row(prediction=2.0), Row(prediction=3.0)]

Solution

1. Step 1: Check features column type

   LinearRegression expects featuresCol to be a vector column, but here 'feature' is a single float, causing an error.

2. Step 2: Understand Spark ML input requirements

   Features must be Vector type; passing a float causes runtime error during fit or transform.

3. Final Answer:

   Error due to incorrect features column type -> Option C

4. Quick Check:

   Features must be Vector, not float [OK]

Hint: Features must be Vector type, not single float [OK]

Common Mistakes:

• Passing float instead of Vector for features
• Expecting predictions without proper feature vector
• Ignoring Spark ML input data format requirements

4. You have this Spark ML code snippet that throws an error:

lr = LinearRegression(featuresCol='features', labelCol='label')
model = lr.fit(trainingData)

The error says: "java.lang.IllegalArgumentException: requirement failed: Column features must be of type vector." What is the likely cause and fix?

medium

A. The trainingData is empty; load data before fitting

B. The 'label' column is missing; add a label column to the data

C. LinearRegression does not support 'features' as input; rename the column

D. The 'features' column is not a vector; fix by using VectorAssembler to create it

Solution

1. Step 1: Analyze error message

   The error says 'features' column must be a vector, so current 'features' column is not Vector type.

2. Step 2: Fix by creating vector features

   Use VectorAssembler to combine feature columns into a single vector column named 'features'.

3. Final Answer:

   The 'features' column is not a vector; fix by using VectorAssembler to create it -> Option D

4. Quick Check:

   Features must be Vector; use VectorAssembler [OK]

Hint: Use VectorAssembler to create vector features column [OK]

Common Mistakes:

• Ignoring the need for vector type in features column
• Renaming columns without fixing data type
• Assuming label column causes this error

5. You want to predict house prices using Spark ML Linear Regression. Your dataset has columns: size (square feet), bedrooms, and price</>. Which is the correct way to prepare features and train the model?

hard

A. Use VectorAssembler to combine 'size' and 'bedrooms' into 'features', then fit LinearRegression with labelCol='price'

B. Directly use 'size' as featuresCol and 'price' as labelCol in LinearRegression

C. Use 'price' as featuresCol and 'size' as labelCol in LinearRegression

D. Train LinearRegression without feature transformation on 'bedrooms' only

Solution

1. Step 1: Combine input features

   Multiple input columns ('size', 'bedrooms') must be combined into a single vector column using VectorAssembler.

2. Step 2: Train LinearRegression model

   Use the combined 'features' vector column as featuresCol and 'price' as labelCol to fit the model.

3. Final Answer:

   Use VectorAssembler to combine 'size' and 'bedrooms' into 'features', then fit LinearRegression with labelCol='price' -> Option A

4. Quick Check:

   Combine features vector, label is target [OK]

Hint: Combine all input features into vector before training [OK]

Common Mistakes:

• Using single feature without vector assembly
• Swapping label and features columns
• Ignoring feature vector requirement