Classification with Spark (LogisticRegression, RandomForest) in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of LogisticRegression Model Training

What is the output of the following Spark code snippet that trains a LogisticRegression model and prints the coefficients?

Apache Spark

from pyspark.ml.classification import LogisticRegression
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('TestLR').getOrCreate()
data = spark.createDataFrame([
  (1.0, [0.0, 1.1, 0.1]),
  (0.0, [2.0, 1.0, -1.0]),
  (0.0, [2.0, 1.3, 1.0]),
  (1.0, [0.0, 1.2, -0.5])
], ["label", "features"])
lr = LogisticRegression(maxIter=10, regParam=0.01)
model = lr.fit(data)
print(model.coefficients)

A [0.0, 0.0, 0.0]

B [0.0, 0.5, -0.3]

C DenseVector([0.0, 0.5, -0.3])

D DenseVector([0.0, 0.0, 0.0])

💡 Hint

Look at the type of the coefficients printed by Spark ML LogisticRegression model.

The LogisticRegression model in Spark ML returns coefficients as a DenseVector object, not a plain list.

❓ data_output

intermediate

Number of Trees in RandomForest Model

After training a RandomForestClassifier with numTrees=20 on a dataset, what is the value of the 'numTrees' attribute in the trained model?

Apache Spark

from pyspark.ml.classification import RandomForestClassifier
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('TestRF').getOrCreate()
data = spark.createDataFrame([
  (1.0, [0.0, 1.1]),
  (0.0, [2.0, 1.0]),
  (0.0, [2.0, 1.3]),
  (1.0, [0.0, 1.2])
], ["label", "features"])
rf = RandomForestClassifier(numTrees=20, maxDepth=5)
model = rf.fit(data)
print(model.getNumTrees())

A 1

B 20

C 5

D 0

💡 Hint

The numTrees parameter controls how many trees are built in the forest.

The getNumTrees method returns the number of trees in the trained RandomForest model, which matches the numTrees parameter.

🔧 Debug

advanced

Identify the Error in LogisticRegression Prediction Code

What error will the following code raise when trying to predict with a LogisticRegression model?

Apache Spark

from pyspark.ml.classification import LogisticRegression
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('TestLR').getOrCreate()
data = spark.createDataFrame([
  (1.0, [0.0, 1.1, 0.1])
], ["label", "features"])
lr = LogisticRegression(maxIter=10)
model = lr.fit(data)
predictions = model.transform([0.0, 1.0])
print(predictions.collect())

A TypeError: 'list' object is not a DataFrame

B AttributeError: 'LogisticRegressionModel' object has no attribute 'transform'

C ValueError: Features vector size mismatch

D No error, prints predictions

💡 Hint

Check the input type passed to model.transform method.

The transform method expects a DataFrame, but a list is passed, causing a TypeError.

❓ visualization

advanced

Interpreting Feature Importances from RandomForest

Given a trained RandomForestClassifier model, which option correctly describes the output of model.featureImportances?

Apache Spark

from pyspark.ml.classification import RandomForestClassifier
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('TestRF').getOrCreate()
data = spark.createDataFrame([
  (1.0, [0.0, 1.1, 0.1]),
  (0.0, [2.0, 1.0, -1.0]),
  (0.0, [2.0, 1.3, 1.0]),
  (1.0, [0.0, 1.2, -0.5])
], ["label", "features"])
rf = RandomForestClassifier(numTrees=10)
model = rf.fit(data)
print(model.featureImportances)

A A DenseVector with importance scores summing to 1.0

B A SparseVector with importance scores for each feature

C A list of integers representing feature indices

D A dictionary mapping feature names to importance scores

💡 Hint

Check the type and properties of featureImportances in Spark ML RandomForest model.

The featureImportances attribute returns a DenseVector with scores for each feature that sum to 1.0.

🚀 Application

expert

Choosing the Best Model Based on Evaluation Metrics

You trained both LogisticRegression and RandomForest models on the same dataset. The LogisticRegression model has an accuracy of 0.85 and AUC of 0.88. The RandomForest model has an accuracy of 0.83 and AUC of 0.90. Which model should you choose if your priority is to maximize the ability to rank positive cases higher than negative ones?

A Choose LogisticRegression because it has higher accuracy

B Choose RandomForest because it has higher accuracy

C Choose LogisticRegression because it has both higher accuracy and AUC

D Choose RandomForest because it has higher AUC

💡 Hint

AUC measures ranking quality, accuracy measures overall correctness.

AUC (Area Under ROC Curve) measures how well the model ranks positive cases higher than negative ones. Since RandomForest has higher AUC, it is better for ranking.

Practice

1. What is the main purpose of Logistic Regression in Spark MLlib classification?

easy

A. To predict the probability of a binary outcome

B. To cluster data into groups without labels

C. To reduce the number of features in the dataset

D. To generate random decision trees

Solution

1. Step 1: Understand Logistic Regression's goal

   Logistic Regression predicts the probability that an input belongs to a class, usually binary (yes/no).

2. Step 2: Compare with other options

   Clustering is unsupervised, feature reduction is different, and random trees relate to Random Forest, not Logistic Regression.

3. Final Answer:

   To predict the probability of a binary outcome -> Option A

4. Quick Check:

   Logistic Regression = Binary probability prediction [OK]

Hint: Logistic Regression predicts yes/no probabilities [OK]

Common Mistakes:

• Confusing classification with clustering
• Thinking Logistic Regression reduces features
• Mixing Logistic Regression with Random Forest

2. Which of the following is the correct way to import LogisticRegression from Spark MLlib in Python?

easy

A. import LogisticRegression from pyspark.ml

B. from pyspark.ml.classification import LogisticRegression

C. from pyspark.ml import LogisticRegression

D. from pyspark.ml.classification import logisticregression

Solution

1. Step 1: Recall correct import syntax

   The correct import uses 'from pyspark.ml.classification import LogisticRegression' with exact casing.

2. Step 2: Check other options for errors

   import LogisticRegression from pyspark.ml has wrong import order, C misses classification submodule, D uses wrong case for class name.

3. Final Answer:

   from pyspark.ml.classification import LogisticRegression -> Option B

4. Quick Check:

   Correct import syntax = from pyspark.ml.classification import LogisticRegression [OK]

Hint: Use exact module and class names with correct case [OK]

Common Mistakes:

• Wrong import order or missing submodule
• Incorrect capitalization of class names
• Assuming all classes are in pyspark.ml directly

3. Given the following Spark code snippet, what will be the output of predictions.select('prediction').show()?

from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import VectorAssembler

# Sample data
data = spark.createDataFrame([
  (0, 1.0, 3.0),
  (1, 2.0, 1.0),
  (0, 1.5, 2.5)
], ['label', 'feature1', 'feature2'])

assembler = VectorAssembler(inputCols=['feature1', 'feature2'], outputCol='features')
data = assembler.transform(data)

lr = LogisticRegression(featuresCol='features', labelCol='label')
model = lr.fit(data)
predictions = model.transform(data)

medium

A. Shows predicted labels as 0.0 or 1.0 for each row

B. Shows raw feature vectors for each row

C. Throws an error because 'prediction' column is missing

D. Shows only the label column values

Solution

1. Step 1: Understand model.transform output

   LogisticRegression's transform adds a 'prediction' column with predicted labels (0.0 or 1.0).

2. Step 2: Check what predictions.select('prediction').show() does

   This command displays the predicted labels for each row, not features or original labels.

3. Final Answer:

   Shows predicted labels as 0.0 or 1.0 for each row -> Option A

4. Quick Check:

   Prediction column = predicted labels [OK]

Hint: Model.transform adds 'prediction' column with labels [OK]

Common Mistakes:

• Expecting raw features instead of predictions
• Assuming 'prediction' column is missing
• Confusing label column with prediction column

4. Identify the error in this Spark RandomForest classification code snippet:

from pyspark.ml.classification import RandomForestClassifier

rf = RandomForestClassifier(labelCol='label', featuresCol='features')
model = rf.fit(data)
predictions = model.transform(data)
predictions.select('prediction').show()

Assuming data has columns 'label' and 'features'.

medium

A. RandomForestClassifier requires setting numTrees parameter

B. Missing import for VectorAssembler

C. Features column must be assembled before fitting

D. No error; code runs correctly

Solution

1. Step 1: Check if data has required columns

   The question states data already has 'label' and 'features' columns, so no need to assemble features here.

2. Step 2: Verify RandomForestClassifier usage

   RandomForestClassifier can run with default parameters; numTrees is optional. The code fits and predicts correctly.

3. Final Answer:

   No error; code runs correctly -> Option D

4. Quick Check:

   Data with 'features' and 'label' = code runs [OK]

Hint: Check if data has features before assuming errors [OK]

Common Mistakes:

• Assuming numTrees is mandatory
• Forgetting features must be assembled before fitting
• Confusing import errors with runtime errors

5. You want to improve classification accuracy on a Spark dataset with many features and possible overfitting. Which approach is best to try first?

hard

A. Train LogisticRegression with only one feature

B. Use LogisticRegression without feature scaling

C. Use RandomForest with more trees and limit tree depth

D. Remove the label column before training

Solution

1. Step 1: Understand overfitting and feature impact

   Overfitting happens when model is too complex; limiting tree depth in RandomForest helps reduce it.

2. Step 2: Evaluate options for improving accuracy

   Increasing trees and limiting depth balances bias and variance. LogisticRegression without scaling or removing label harms model. Using one feature loses information.

3. Final Answer:

   Use RandomForest with more trees and limit tree depth -> Option C

4. Quick Check:

   Control complexity to reduce overfitting [OK]

Hint: Limit tree depth in RandomForest to reduce overfitting [OK]

Common Mistakes:

• Ignoring feature scaling when needed
• Removing label column by mistake
• Using too few features reduces accuracy