Classification with Spark (LogisticRegression, RandomForest) in Apache Spark - Mini Project: Build & Apply

Classification with Spark (LogisticRegression, RandomForest)

📖 Scenario: You work as a data scientist for a retail company. They want to predict if a customer will buy a product based on their age and salary. You will use Spark to build two simple classification models: Logistic Regression and Random Forest.

🎯 Goal: Build a Spark DataFrame with customer data, set up a feature vector, train Logistic Regression and Random Forest classifiers, and print their prediction results.

📋 What You'll Learn

Create a Spark DataFrame with given customer data

Use VectorAssembler to combine features into a single vector column

Train a LogisticRegression model on the data

Train a RandomForestClassifier model on the data

Print the predictions from both models

💡 Why This Matters

🌍 Real World

Retail companies use classification models to predict customer buying behavior based on demographics and other features.

💼 Career

Data scientists often build and compare classification models using Spark to handle large datasets efficiently.

1

Create Spark DataFrame with customer data

Create a Spark DataFrame called data with these exact rows and columns: age (integer), salary (integer), and bought (integer label). Use these rows: (25, 50000, 0), (35, 60000, 1), (45, 80000, 1), (20, 20000, 0), (50, 90000, 1).

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CustomerClassification").getOrCreate()
# Create the DataFrame called data with columns age, salary, bought and the exact rows
# Your code here

Hint

Use spark.createDataFrame with a list of tuples and specify the column names as a list.

2

Set up feature vector with VectorAssembler

Create a VectorAssembler called assembler that combines the columns age and salary into a new column called features. Then create a new DataFrame called assembled_data by transforming data with assembler.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CustomerClassification").getOrCreate()
data = spark.createDataFrame([
    (25, 50000, 0),
    (35, 60000, 1),
    (45, 80000, 1),
    (20, 20000, 0),
    (50, 90000, 1)
], ["age", "salary", "bought"])

from pyspark.ml.feature import VectorAssembler

# Create VectorAssembler called assembler with inputCols age and salary, outputCol features
# Create assembled_data by transforming data with assembler
# Your code here

Hint

Use VectorAssembler(inputCols=["age", "salary"], outputCol="features") and then call transform(data).

3

Train LogisticRegression and RandomForestClassifier models

Import LogisticRegression and RandomForestClassifier from pyspark.ml.classification. Create a LogisticRegression model called lr with label column bought and features column features. Create a RandomForestClassifier model called rf with the same label and features columns. Fit both models on assembled_data and save the results as lr_model and rf_model.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CustomerClassification").getOrCreate()
data = spark.createDataFrame([
    (25, 50000, 0),
    (35, 60000, 1),
    (45, 80000, 1),
    (20, 20000, 0),
    (50, 90000, 1)
], ["age", "salary", "bought"])

from pyspark.ml.feature import VectorAssembler

assembler = VectorAssembler(inputCols=["age", "salary"], outputCol="features")
assembled_data = assembler.transform(data)

from pyspark.ml.classification import LogisticRegression, RandomForestClassifier

# Create LogisticRegression model lr with labelCol bought and featuresCol features
# Create RandomForestClassifier model rf with labelCol bought and featuresCol features
# Fit lr on assembled_data to get lr_model
# Fit rf on assembled_data to get rf_model
# Your code here

Hint

Use LogisticRegression(labelCol="bought", featuresCol="features") and RandomForestClassifier(labelCol="bought", featuresCol="features"). Then call fit(assembled_data) on each.

4

Print predictions from both models

Use lr_model.transform(assembled_data) to get predictions from LogisticRegression and save as lr_predictions. Use rf_model.transform(assembled_data) to get predictions from RandomForestClassifier and save as rf_predictions. Then print the prediction column from both lr_predictions and rf_predictions DataFrames.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CustomerClassification").getOrCreate()
data = spark.createDataFrame([
    (25, 50000, 0),
    (35, 60000, 1),
    (45, 80000, 1),
    (20, 20000, 0),
    (50, 90000, 1)
], ["age", "salary", "bought"])

from pyspark.ml.feature import VectorAssembler

assembler = VectorAssembler(inputCols=["age", "salary"], outputCol="features")
assembled_data = assembler.transform(data)

from pyspark.ml.classification import LogisticRegression, RandomForestClassifier

lr = LogisticRegression(labelCol="bought", featuresCol="features")
rf = RandomForestClassifier(labelCol="bought", featuresCol="features")

lr_model = lr.fit(assembled_data)
rf_model = rf.fit(assembled_data)

# Get predictions from lr_model and rf_model on assembled_data
# Print the prediction column from lr_predictions
# Print the prediction column from rf_predictions
# Your code here

Hint

Use model.transform(assembled_data) to get predictions. Then use select("prediction").show() to print predictions.

Practice

1. What is the main purpose of Logistic Regression in Spark MLlib classification?

easy

A. To predict the probability of a binary outcome

B. To cluster data into groups without labels

C. To reduce the number of features in the dataset

D. To generate random decision trees

Solution

1. Step 1: Understand Logistic Regression's goal

   Logistic Regression predicts the probability that an input belongs to a class, usually binary (yes/no).

2. Step 2: Compare with other options

   Clustering is unsupervised, feature reduction is different, and random trees relate to Random Forest, not Logistic Regression.

3. Final Answer:

   To predict the probability of a binary outcome -> Option A

4. Quick Check:

   Logistic Regression = Binary probability prediction [OK]

Hint: Logistic Regression predicts yes/no probabilities [OK]

Common Mistakes:

• Confusing classification with clustering
• Thinking Logistic Regression reduces features
• Mixing Logistic Regression with Random Forest

2. Which of the following is the correct way to import LogisticRegression from Spark MLlib in Python?

easy

A. import LogisticRegression from pyspark.ml

B. from pyspark.ml.classification import LogisticRegression

C. from pyspark.ml import LogisticRegression

D. from pyspark.ml.classification import logisticregression

Solution

1. Step 1: Recall correct import syntax

   The correct import uses 'from pyspark.ml.classification import LogisticRegression' with exact casing.

2. Step 2: Check other options for errors

   import LogisticRegression from pyspark.ml has wrong import order, C misses classification submodule, D uses wrong case for class name.

3. Final Answer:

   from pyspark.ml.classification import LogisticRegression -> Option B

4. Quick Check:

   Correct import syntax = from pyspark.ml.classification import LogisticRegression [OK]

Hint: Use exact module and class names with correct case [OK]

Common Mistakes:

• Wrong import order or missing submodule
• Incorrect capitalization of class names
• Assuming all classes are in pyspark.ml directly

3. Given the following Spark code snippet, what will be the output of predictions.select('prediction').show()?

from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import VectorAssembler

# Sample data
data = spark.createDataFrame([
  (0, 1.0, 3.0),
  (1, 2.0, 1.0),
  (0, 1.5, 2.5)
], ['label', 'feature1', 'feature2'])

assembler = VectorAssembler(inputCols=['feature1', 'feature2'], outputCol='features')
data = assembler.transform(data)

lr = LogisticRegression(featuresCol='features', labelCol='label')
model = lr.fit(data)
predictions = model.transform(data)

medium

A. Shows predicted labels as 0.0 or 1.0 for each row

B. Shows raw feature vectors for each row

C. Throws an error because 'prediction' column is missing

D. Shows only the label column values

Solution

1. Step 1: Understand model.transform output

   LogisticRegression's transform adds a 'prediction' column with predicted labels (0.0 or 1.0).

2. Step 2: Check what predictions.select('prediction').show() does

   This command displays the predicted labels for each row, not features or original labels.

3. Final Answer:

   Shows predicted labels as 0.0 or 1.0 for each row -> Option A

4. Quick Check:

   Prediction column = predicted labels [OK]

Hint: Model.transform adds 'prediction' column with labels [OK]

Common Mistakes:

• Expecting raw features instead of predictions
• Assuming 'prediction' column is missing
• Confusing label column with prediction column

4. Identify the error in this Spark RandomForest classification code snippet:

from pyspark.ml.classification import RandomForestClassifier

rf = RandomForestClassifier(labelCol='label', featuresCol='features')
model = rf.fit(data)
predictions = model.transform(data)
predictions.select('prediction').show()

Assuming data has columns 'label' and 'features'.

medium

A. RandomForestClassifier requires setting numTrees parameter

B. Missing import for VectorAssembler

C. Features column must be assembled before fitting

D. No error; code runs correctly

Solution

1. Step 1: Check if data has required columns

   The question states data already has 'label' and 'features' columns, so no need to assemble features here.

2. Step 2: Verify RandomForestClassifier usage

   RandomForestClassifier can run with default parameters; numTrees is optional. The code fits and predicts correctly.

3. Final Answer:

   No error; code runs correctly -> Option D

4. Quick Check:

   Data with 'features' and 'label' = code runs [OK]

Hint: Check if data has features before assuming errors [OK]

Common Mistakes:

• Assuming numTrees is mandatory
• Forgetting features must be assembled before fitting
• Confusing import errors with runtime errors

5. You want to improve classification accuracy on a Spark dataset with many features and possible overfitting. Which approach is best to try first?

hard

A. Train LogisticRegression with only one feature

B. Use LogisticRegression without feature scaling

C. Use RandomForest with more trees and limit tree depth

D. Remove the label column before training

Solution

1. Step 1: Understand overfitting and feature impact

   Overfitting happens when model is too complex; limiting tree depth in RandomForest helps reduce it.

2. Step 2: Evaluate options for improving accuracy

   Increasing trees and limiting depth balances bias and variance. LogisticRegression without scaling or removing label harms model. Using one feature loses information.

3. Final Answer:

   Use RandomForest with more trees and limit tree depth -> Option C

4. Quick Check:

   Control complexity to reduce overfitting [OK]

Hint: Limit tree depth in RandomForest to reduce overfitting [OK]

Common Mistakes:

• Ignoring feature scaling when needed
• Removing label column by mistake
• Using too few features reduces accuracy