Classification with Spark (LogisticRegression, RandomForest) in Apache Spark - Step-by-Step Execution

Concept Flow - Classification with Spark (LogisticRegression, RandomForest)

Load Data

↓

Prepare Features

↓

Split Data: Train/Test

↓

Choose Model

↓

LogisticRegression

↓

Train Model

↓

Make Predictions on Test Data

↓

Evaluate Model Accuracy

↓

End

The flow starts with loading data, preparing features, splitting into train/test sets, choosing a classification model (Logistic Regression or Random Forest), training it, making predictions, and evaluating accuracy.

Execution Sample

Apache Spark

from pyspark.ml.classification import LogisticRegression, RandomForestClassifier
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Load and prepare data
# Split data
# Train Logistic Regression
# Predict and evaluate

This code loads data, trains a Logistic Regression model, makes predictions, and evaluates accuracy.

Execution Table

Step	Action	Model	Data Split	Training Status	Prediction Output	Evaluation Metric
1	Load data	-	-	-	-	-
2	Prepare features	-	-	-	-	-
3	Split data	-	Train: 70%, Test: 30%	-	-	-
4	Initialize LogisticRegression	LogisticRegression	Train/Test split ready	Not started	-	-
5	Train LogisticRegression model	LogisticRegression	Train/Test split ready	Completed	-	-
6	Make predictions on test data	LogisticRegression	Train/Test split ready	Completed	Predictions DataFrame with labels and probabilities	-
7	Evaluate accuracy	LogisticRegression	Train/Test split ready	Completed	Predictions DataFrame	Accuracy: 0.85
8	Initialize RandomForestClassifier	RandomForest	Train/Test split ready	Not started	-	-
9	Train RandomForest model	RandomForest	Train/Test split ready	Completed	-	-
10	Make predictions on test data	RandomForest	Train/Test split ready	Completed	Predictions DataFrame with labels and probabilities	-
11	Evaluate accuracy	RandomForest	Train/Test split ready	Completed	Predictions DataFrame	Accuracy: 0.88
12	End process	-	-	-	-	-

💡 Both models trained and evaluated; process ends after accuracy comparison.

Variable Tracker

Variable	Start	After Data Load	After Feature Prep	After Split	After Training LR	After Prediction LR	After Training RF	After Prediction RF	Final
data	None	Raw DataFrame	Features Prepared DataFrame	Train/Test DataFrames	Train/Test DataFrames	Train/Test DataFrames	Train/Test DataFrames	Train/Test DataFrames	Train/Test DataFrames
model	None	None	None	None	LogisticRegression Model	LogisticRegression Model	RandomForest Model	RandomForest Model	RandomForest Model
predictions	None	None	None	None	None	LR Predictions DataFrame	LR Predictions DataFrame	RF Predictions DataFrame	RF Predictions DataFrame
accuracy	None	None	None	None	None	None	0.85	0.85	0.88

Key Moments - 3 Insights

Why do we split the data into training and test sets before training?

Splitting data (see execution_table row 3) ensures the model learns on one part (training) and is tested on unseen data (test) to fairly evaluate performance.

What is the difference between LogisticRegression and RandomForest in this flow?

They are two different classification models trained separately (rows 4-7 for LogisticRegression and 8-11 for RandomForest) to compare which predicts better.

Why do we evaluate accuracy after predictions?

Accuracy (rows 7 and 11) measures how well the model predicts the correct labels on test data, showing model effectiveness.

Visual Quiz - 3 Questions

Look at the execution_table at step 7, what is the accuracy of the LogisticRegression model?

A 0.75

B 0.85

C 0.88

D 0.90

💡 Hint

Check the Evaluation Metric column at step 7 in the execution_table.

At which step does the RandomForest model finish training?

A Step 6

B Step 11

C Step 9

D Step 4

💡 Hint

Look at the Training Status and Model columns in execution_table rows.

If we did not split the data before training, how would the execution_table change?

A There would be no Train/Test split step and evaluation would be unreliable

B Accuracy would be higher automatically

C Models would not train at all

D Predictions would be skipped

💡 Hint

Refer to the importance of data splitting in key_moments and execution_table step 3.

Concept Snapshot

Classification with Spark:
- Load and prepare data
- Split data into train/test sets
- Choose model: LogisticRegression or RandomForest
- Train model on training data
- Predict on test data
- Evaluate accuracy to compare models

Full Transcript

This visual execution shows how to perform classification using Apache Spark's LogisticRegression and RandomForest models. First, data is loaded and features are prepared. Then, data is split into training and testing sets to ensure fair evaluation. Next, LogisticRegression is initialized and trained on the training data, followed by making predictions on the test data and evaluating accuracy. The same steps are repeated for RandomForest. Accuracy scores help compare which model performs better. Key moments include understanding why data splitting is essential, the difference between the two models, and the purpose of evaluation. The execution table tracks each step, model status, predictions, and accuracy values. This step-by-step trace helps beginners see how classification models are built and assessed in Spark.

Practice

1. What is the main purpose of Logistic Regression in Spark MLlib classification?

easy

A. To predict the probability of a binary outcome

B. To cluster data into groups without labels

C. To reduce the number of features in the dataset

D. To generate random decision trees

Solution

1. Step 1: Understand Logistic Regression's goal

   Logistic Regression predicts the probability that an input belongs to a class, usually binary (yes/no).

2. Step 2: Compare with other options

   Clustering is unsupervised, feature reduction is different, and random trees relate to Random Forest, not Logistic Regression.

3. Final Answer:

   To predict the probability of a binary outcome -> Option A

4. Quick Check:

   Logistic Regression = Binary probability prediction [OK]

Hint: Logistic Regression predicts yes/no probabilities [OK]

Common Mistakes:

• Confusing classification with clustering
• Thinking Logistic Regression reduces features
• Mixing Logistic Regression with Random Forest

2. Which of the following is the correct way to import LogisticRegression from Spark MLlib in Python?

easy

A. import LogisticRegression from pyspark.ml

B. from pyspark.ml.classification import LogisticRegression

C. from pyspark.ml import LogisticRegression

D. from pyspark.ml.classification import logisticregression

Solution

1. Step 1: Recall correct import syntax

   The correct import uses 'from pyspark.ml.classification import LogisticRegression' with exact casing.

2. Step 2: Check other options for errors

   import LogisticRegression from pyspark.ml has wrong import order, C misses classification submodule, D uses wrong case for class name.

3. Final Answer:

   from pyspark.ml.classification import LogisticRegression -> Option B

4. Quick Check:

   Correct import syntax = from pyspark.ml.classification import LogisticRegression [OK]

Hint: Use exact module and class names with correct case [OK]

Common Mistakes:

• Wrong import order or missing submodule
• Incorrect capitalization of class names
• Assuming all classes are in pyspark.ml directly

3. Given the following Spark code snippet, what will be the output of predictions.select('prediction').show()?

from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import VectorAssembler

# Sample data
data = spark.createDataFrame([
  (0, 1.0, 3.0),
  (1, 2.0, 1.0),
  (0, 1.5, 2.5)
], ['label', 'feature1', 'feature2'])

assembler = VectorAssembler(inputCols=['feature1', 'feature2'], outputCol='features')
data = assembler.transform(data)

lr = LogisticRegression(featuresCol='features', labelCol='label')
model = lr.fit(data)
predictions = model.transform(data)

medium

A. Shows predicted labels as 0.0 or 1.0 for each row

B. Shows raw feature vectors for each row

C. Throws an error because 'prediction' column is missing

D. Shows only the label column values

Solution

1. Step 1: Understand model.transform output

   LogisticRegression's transform adds a 'prediction' column with predicted labels (0.0 or 1.0).

2. Step 2: Check what predictions.select('prediction').show() does

   This command displays the predicted labels for each row, not features or original labels.

3. Final Answer:

   Shows predicted labels as 0.0 or 1.0 for each row -> Option A

4. Quick Check:

   Prediction column = predicted labels [OK]

Hint: Model.transform adds 'prediction' column with labels [OK]

Common Mistakes:

• Expecting raw features instead of predictions
• Assuming 'prediction' column is missing
• Confusing label column with prediction column

4. Identify the error in this Spark RandomForest classification code snippet:

from pyspark.ml.classification import RandomForestClassifier

rf = RandomForestClassifier(labelCol='label', featuresCol='features')
model = rf.fit(data)
predictions = model.transform(data)
predictions.select('prediction').show()

Assuming data has columns 'label' and 'features'.

medium

A. RandomForestClassifier requires setting numTrees parameter

B. Missing import for VectorAssembler

C. Features column must be assembled before fitting

D. No error; code runs correctly

Solution

1. Step 1: Check if data has required columns

   The question states data already has 'label' and 'features' columns, so no need to assemble features here.

2. Step 2: Verify RandomForestClassifier usage

   RandomForestClassifier can run with default parameters; numTrees is optional. The code fits and predicts correctly.

3. Final Answer:

   No error; code runs correctly -> Option D

4. Quick Check:

   Data with 'features' and 'label' = code runs [OK]

Hint: Check if data has features before assuming errors [OK]

Common Mistakes:

• Assuming numTrees is mandatory
• Forgetting features must be assembled before fitting
• Confusing import errors with runtime errors

5. You want to improve classification accuracy on a Spark dataset with many features and possible overfitting. Which approach is best to try first?

hard

A. Train LogisticRegression with only one feature

B. Use LogisticRegression without feature scaling

C. Use RandomForest with more trees and limit tree depth

D. Remove the label column before training

Solution

1. Step 1: Understand overfitting and feature impact

   Overfitting happens when model is too complex; limiting tree depth in RandomForest helps reduce it.

2. Step 2: Evaluate options for improving accuracy

   Increasing trees and limiting depth balances bias and variance. LogisticRegression without scaling or removing label harms model. Using one feature loses information.

3. Final Answer:

   Use RandomForest with more trees and limit tree depth -> Option C

4. Quick Check:

   Control complexity to reduce overfitting [OK]

Hint: Limit tree depth in RandomForest to reduce overfitting [OK]

Common Mistakes:

• Ignoring feature scaling when needed
• Removing label column by mistake
• Using too few features reduces accuracy