Classification with Spark (LogisticRegression, RandomForest) in Apache Spark

Introduction

Classification helps us sort data into groups. Spark makes it easy to do this fast on big data using models like Logistic Regression and Random Forest.

When you want to predict if an email is spam or not.

When you need to decide if a customer will buy a product.

When sorting images into categories like cats or dogs.

When detecting fraud in bank transactions.

When grouping patients by disease type based on symptoms.

Syntax

Apache Spark

from pyspark.ml.classification import LogisticRegression, RandomForestClassifier

# Create Logistic Regression model
lr = LogisticRegression(featuresCol='features', labelCol='label')

# Create Random Forest model
rf = RandomForestClassifier(featuresCol='features', labelCol='label', numTrees=10)

featuresCol is the column with input data features.

labelCol is the column with the target group labels.

Examples

This creates a Logistic Regression model for binary classification.

Apache Spark

lr = LogisticRegression(featuresCol='features', labelCol='label')

This creates a Random Forest model with 20 trees for better accuracy.

Apache Spark

rf = RandomForestClassifier(featuresCol='features', labelCol='label', numTrees=20)

Sample Program

This program creates a small dataset, prepares features, trains Logistic Regression and Random Forest models, and shows their predictions.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.ml.classification import LogisticRegression, RandomForestClassifier
from pyspark.ml.feature import VectorAssembler

# Start Spark session
spark = SparkSession.builder.appName('ClassificationExample').getOrCreate()

# Sample data: features and label
data = [
    (0, 1.0, 0.1, 0.2),
    (1, 2.0, 1.1, 1.2),
    (0, 1.3, 0.2, 0.3),
    (1, 3.0, 1.5, 1.3),
    (0, 0.8, 0.1, 0.1),
    (1, 2.5, 1.3, 1.1)
]

# Create DataFrame with columns: label, feature1, feature2, feature3
df = spark.createDataFrame(data, ['label', 'f1', 'f2', 'f3'])

# Combine features into one vector column
assembler = VectorAssembler(inputCols=['f1', 'f2', 'f3'], outputCol='features')
assembled_df = assembler.transform(df)

# Select only label and features
final_df = assembled_df.select('label', 'features')

# Logistic Regression model
lr = LogisticRegression(featuresCol='features', labelCol='label')
lr_model = lr.fit(final_df)
lr_predictions = lr_model.transform(final_df)

print('Logistic Regression Predictions:')
lr_predictions.select('label', 'prediction', 'probability').show()

# Random Forest model
rf = RandomForestClassifier(featuresCol='features', labelCol='label', numTrees=5)
rf_model = rf.fit(final_df)
rf_predictions = rf_model.transform(final_df)

print('Random Forest Predictions:')
rf_predictions.select('label', 'prediction', 'probability').show()

spark.stop()

Important Notes

Make sure your features are combined into a single vector column before training.

Logistic Regression works well for simple binary problems.

Random Forest can handle more complex data and usually gives better accuracy.

Summary

Classification sorts data into groups using models.

Spark MLlib offers Logistic Regression and Random Forest for classification.

Prepare data by combining features, then train and predict with these models.

Practice

1. What is the main purpose of Logistic Regression in Spark MLlib classification?

easy

A. To predict the probability of a binary outcome

B. To cluster data into groups without labels

C. To reduce the number of features in the dataset

D. To generate random decision trees

Solution

1. Step 1: Understand Logistic Regression's goal

   Logistic Regression predicts the probability that an input belongs to a class, usually binary (yes/no).

2. Step 2: Compare with other options

   Clustering is unsupervised, feature reduction is different, and random trees relate to Random Forest, not Logistic Regression.

3. Final Answer:

   To predict the probability of a binary outcome -> Option A

4. Quick Check:

   Logistic Regression = Binary probability prediction [OK]

Hint: Logistic Regression predicts yes/no probabilities [OK]

Common Mistakes:

• Confusing classification with clustering
• Thinking Logistic Regression reduces features
• Mixing Logistic Regression with Random Forest

2. Which of the following is the correct way to import LogisticRegression from Spark MLlib in Python?

easy

A. import LogisticRegression from pyspark.ml

B. from pyspark.ml.classification import LogisticRegression

C. from pyspark.ml import LogisticRegression

D. from pyspark.ml.classification import logisticregression

Solution

1. Step 1: Recall correct import syntax

   The correct import uses 'from pyspark.ml.classification import LogisticRegression' with exact casing.

2. Step 2: Check other options for errors

   import LogisticRegression from pyspark.ml has wrong import order, C misses classification submodule, D uses wrong case for class name.

3. Final Answer:

   from pyspark.ml.classification import LogisticRegression -> Option B

4. Quick Check:

   Correct import syntax = from pyspark.ml.classification import LogisticRegression [OK]

Hint: Use exact module and class names with correct case [OK]

Common Mistakes:

• Wrong import order or missing submodule
• Incorrect capitalization of class names
• Assuming all classes are in pyspark.ml directly

3. Given the following Spark code snippet, what will be the output of predictions.select('prediction').show()?

from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import VectorAssembler

# Sample data
data = spark.createDataFrame([
  (0, 1.0, 3.0),
  (1, 2.0, 1.0),
  (0, 1.5, 2.5)
], ['label', 'feature1', 'feature2'])

assembler = VectorAssembler(inputCols=['feature1', 'feature2'], outputCol='features')
data = assembler.transform(data)

lr = LogisticRegression(featuresCol='features', labelCol='label')
model = lr.fit(data)
predictions = model.transform(data)

medium

A. Shows predicted labels as 0.0 or 1.0 for each row

B. Shows raw feature vectors for each row

C. Throws an error because 'prediction' column is missing

D. Shows only the label column values

Solution

1. Step 1: Understand model.transform output

   LogisticRegression's transform adds a 'prediction' column with predicted labels (0.0 or 1.0).

2. Step 2: Check what predictions.select('prediction').show() does

   This command displays the predicted labels for each row, not features or original labels.

3. Final Answer:

   Shows predicted labels as 0.0 or 1.0 for each row -> Option A

4. Quick Check:

   Prediction column = predicted labels [OK]

Hint: Model.transform adds 'prediction' column with labels [OK]

Common Mistakes:

• Expecting raw features instead of predictions
• Assuming 'prediction' column is missing
• Confusing label column with prediction column

4. Identify the error in this Spark RandomForest classification code snippet:

from pyspark.ml.classification import RandomForestClassifier

rf = RandomForestClassifier(labelCol='label', featuresCol='features')
model = rf.fit(data)
predictions = model.transform(data)
predictions.select('prediction').show()

Assuming data has columns 'label' and 'features'.

medium

A. RandomForestClassifier requires setting numTrees parameter

B. Missing import for VectorAssembler

C. Features column must be assembled before fitting

D. No error; code runs correctly

Solution

1. Step 1: Check if data has required columns

   The question states data already has 'label' and 'features' columns, so no need to assemble features here.

2. Step 2: Verify RandomForestClassifier usage

   RandomForestClassifier can run with default parameters; numTrees is optional. The code fits and predicts correctly.

3. Final Answer:

   No error; code runs correctly -> Option D

4. Quick Check:

   Data with 'features' and 'label' = code runs [OK]

Hint: Check if data has features before assuming errors [OK]

Common Mistakes:

• Assuming numTrees is mandatory
• Forgetting features must be assembled before fitting
• Confusing import errors with runtime errors

5. You want to improve classification accuracy on a Spark dataset with many features and possible overfitting. Which approach is best to try first?

hard

A. Train LogisticRegression with only one feature

B. Use LogisticRegression without feature scaling

C. Use RandomForest with more trees and limit tree depth

D. Remove the label column before training

Solution

1. Step 1: Understand overfitting and feature impact

   Overfitting happens when model is too complex; limiting tree depth in RandomForest helps reduce it.

2. Step 2: Evaluate options for improving accuracy

   Increasing trees and limiting depth balances bias and variance. LogisticRegression without scaling or removing label harms model. Using one feature loses information.

3. Final Answer:

   Use RandomForest with more trees and limit tree depth -> Option C

4. Quick Check:

   Control complexity to reduce overfitting [OK]

Hint: Limit tree depth in RandomForest to reduce overfitting [OK]

Common Mistakes:

• Ignoring feature scaling when needed
• Removing label column by mistake
• Using too few features reduces accuracy