ML Pipelines in Spark in Apache Spark - Mini Project: Build & Apply

ML Pipelines in Spark

📖 Scenario: You work as a data scientist at a retail company. You want to build a machine learning model to predict if a customer will buy a product based on their age and salary. To keep your work organized and reusable, you will use ML Pipelines in Apache Spark.

🎯 Goal: Build a simple ML Pipeline in Spark that includes data setup, configuration of a feature assembler, training a logistic regression model, and displaying the predictions.

📋 What You'll Learn

Create a Spark DataFrame with customer data

Configure a VectorAssembler to combine features

Train a LogisticRegression model using a Pipeline

Show the prediction results

💡 Why This Matters

🌍 Real World

ML Pipelines help data scientists organize and automate machine learning workflows, making it easier to manage data preparation, model training, and prediction steps.

💼 Career

Understanding ML Pipelines in Spark is essential for roles like Data Scientist and Machine Learning Engineer working with big data and scalable machine learning solutions.

1

Create the initial Spark DataFrame

Create a Spark DataFrame called data with these exact rows and columns: age (integer), salary (integer), and purchased (integer). The rows are: (25, 50000, 0), (35, 65000, 1), (45, 80000, 1), (20, 40000, 0), (50, 90000, 1).

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MLPipelineExample").getOrCreate()
# Create the Spark DataFrame called data with the specified rows and columns
# Your code here

Hint

Use spark.createDataFrame with a list of tuples and a list of column names.

2

Configure the VectorAssembler

Create a VectorAssembler called assembler that combines the columns age and salary into a new column called features.

Apache Spark

from pyspark.ml.feature import VectorAssembler

# Create the VectorAssembler called assembler
# Your code here

Hint

Use VectorAssembler from pyspark.ml.feature with inputCols and outputCol.

3

Build and train the ML Pipeline

Create a LogisticRegression model called lr with label column purchased. Then create a Pipeline called pipeline with stages assembler and lr. Fit the pipeline on data and save the result as model.

Apache Spark

from pyspark.ml.classification import LogisticRegression
from pyspark.ml import Pipeline

# Create LogisticRegression model lr with labelCol 'purchased'
# Create Pipeline pipeline with stages assembler and lr
# Fit pipeline on data and save as model
# Your code here

Hint

Import LogisticRegression and Pipeline. Use Pipeline(stages=[...]) and fit().

4

Show predictions from the model

Use model.transform(data) to get predictions and save as predictions. Then use predictions.select("age", "salary", "purchased", "prediction") and show() to display the results.

Apache Spark

# Use model.transform(data) to get predictions
# Select columns age, salary, purchased, prediction and show the results
# Your code here

Hint

Use model.transform(data) to get predictions, then select the columns and call show().

Practice

1. What is the main purpose of using ML Pipelines in Apache Spark?

easy

A. To visualize data in charts and graphs

B. To chain multiple data processing and machine learning steps into one workflow

C. To store data permanently in a database

D. To write SQL queries for data extraction

Solution

1. Step 1: Understand ML Pipelines concept

   ML Pipelines are designed to combine multiple steps like data cleaning, feature extraction, and model training into a single workflow.

2. Step 2: Identify the main purpose

   This chaining helps automate the process and apply the same steps easily to new data.

3. Final Answer:

   To chain multiple data processing and machine learning steps into one workflow -> Option B

4. Quick Check:

   ML Pipelines = chaining steps [OK]

Hint: Remember: Pipelines automate and chain ML steps [OK]

Common Mistakes:

• Confusing pipelines with data visualization tools
• Thinking pipelines store data permanently
• Assuming pipelines are only for SQL queries

2. Which of the following is the correct way to create a Pipeline in Spark ML?

easy

A. pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

B. pipeline = Pipeline(steps=[tokenizer, hashingTF, lr])

C. pipeline = Pipeline(chain=[tokenizer, hashingTF, lr])

D. pipeline = Pipeline(process=[tokenizer, hashingTF, lr])

Solution

1. Step 1: Recall Spark ML Pipeline syntax

   In Spark ML, the Pipeline constructor expects a parameter named stages which is a list of transformers and estimators.

2. Step 2: Match the correct parameter name

   Only pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) uses stages correctly; others use incorrect parameter names.

3. Final Answer:

   pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) -> Option A

4. Quick Check:

   Use 'stages' to list pipeline steps [OK]

Hint: Use 'stages' parameter to add steps in Pipeline [OK]

Common Mistakes:

• Using 'steps' instead of 'stages'
• Using 'chain' or 'process' which are invalid
• Forgetting to pass a list of stages

3. Given the following Spark ML Pipeline code, what will be the output type of model.transform(testData)?

from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF
from pyspark.ml.classification import LogisticRegression

# Define stages
tokenizer = Tokenizer(inputCol='text', outputCol='words')
hashingTF = HashingTF(inputCol='words', outputCol='features')
lr = LogisticRegression(maxIter=10, regParam=0.01)

pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

# Fit the pipeline
model = pipeline.fit(trainingData)

# Transform test data
result = model.transform(testData)
print(type(result))

medium

A.

B.

C.

D.

Solution

1. Step 1: Understand what model.transform returns

   In Spark ML, calling transform() on a PipelineModel returns a DataFrame with new columns added by the pipeline stages.

2. Step 2: Identify the output type

   The output is a Spark DataFrame, not a list, dict, or PipelineModel.

3. Final Answer:

   <class 'pyspark.sql.dataframe.DataFrame'> -> Option D

4. Quick Check:

   model.transform() returns DataFrame [OK]

Hint: transform() always outputs a DataFrame in Spark ML [OK]

Common Mistakes:

• Thinking transform returns the model itself
• Expecting a Python list or dict instead of DataFrame
• Confusing fit() output with transform() output

4. Identify the error in the following Spark ML Pipeline code snippet:

from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF

tokenizer = Tokenizer(inputCol='text', outputCol='words')
hashingTF = HashingTF(inputCol='words', outputCol='features')

pipeline = Pipeline(stages=[tokenizer, hashingTF])
model = pipeline.fit(trainingData)
result = model.transform(testData)
print(result.select('prediction').show())

medium

A. Missing an estimator stage like LogisticRegression before fitting

B. Tokenizer inputCol should be 'words' instead of 'text'

C. Pipeline stages should be passed as a tuple, not a list

D. transform() cannot be called on the pipeline model

Solution

1. Step 1: Check pipeline stages

   The pipeline only has transformers (Tokenizer and HashingTF) but no estimator (like LogisticRegression) to train a model.

2. Step 2: Understand fit() behavior

   Fitting a pipeline without an estimator returns a PipelineModel but no trained model for prediction; this is valid but usually incomplete for ML tasks.

3. Step 3: Identify the error

   For supervised learning, missing an estimator means no model is trained, so the pipeline is incomplete.

4. Final Answer:

   Missing an estimator stage like LogisticRegression before fitting -> Option A

5. Quick Check:

   Estimator needed for training model [OK]

Hint: Always include an estimator for training in pipeline stages [OK]

Common Mistakes:

• Confusing inputCol names
• Thinking stages must be tuples
• Believing transform() can't be called on PipelineModel

5. You want to create a Spark ML Pipeline that preprocesses text data, trains a logistic regression model, and then applies the same pipeline to new streaming data. Which approach is best to ensure the pipeline works correctly on both batch and streaming data?

hard

A. Use only Tokenizer and HashingTF in the pipeline; train LogisticRegression on streaming data directly

B. Train LogisticRegression separately on batch data; apply Tokenizer and HashingTF manually on streaming data

C. Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform

D. Fit the pipeline on streaming data first, then apply it to batch data

Solution

1. Step 1: Understand pipeline reuse for batch and streaming

   To apply the same transformations and model to streaming data, you must save the trained PipelineModel from batch training.

2. Step 2: Best practice for streaming

   Fit the pipeline on batch data, save the PipelineModel, then load it in streaming jobs to transform streaming data consistently.

3. Step 3: Evaluate options

   Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform follows this best practice. Other options either separate steps manually or train on streaming data first, which is less reliable.

4. Final Answer:

   Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform -> Option C

5. Quick Check:

   Save and reuse PipelineModel for streaming [OK]

Hint: Train pipeline on batch, save model, reuse for streaming [OK]

Common Mistakes:

• Training model separately from preprocessing
• Trying to train on streaming data first
• Not saving/loading PipelineModel for reuse