ML Pipelines in Spark in Apache Spark - Step-by-Step Execution

Concept Flow - ML Pipelines in Spark

Start: Raw Data

↓

Data Cleaning & Preparation

↓

Feature Engineering

↓

Model Training

↓

Model Evaluation

↓

Model Deployment or Prediction

↓

End

ML Pipelines in Spark flow from raw data through preparation, feature creation, training, evaluation, and finally prediction.

Execution Sample

Apache Spark

from pyspark.ml import Pipeline
from pyspark.ml.feature import StringIndexer, VectorAssembler
from pyspark.ml.classification import LogisticRegression

indexer = StringIndexer(inputCol='category', outputCol='categoryIndex')
assembler = VectorAssembler(inputCols=['feature1', 'feature2', 'categoryIndex'], outputCol='features')

lr = LogisticRegression(featuresCol='features', labelCol='label')

pipeline = Pipeline(stages=[indexer, assembler, lr])
model = pipeline.fit(trainingData)
predictions = model.transform(testData)

This code builds and runs a Spark ML pipeline with indexing, assembling features, and logistic regression.

Execution Table

Step	Action	Input Data	Output Data	Notes
1	StringIndexer fit	Raw data with 'category' column	Data with 'categoryIndex' column	Converts categories to numeric indices
2	VectorAssembler transform	Data with 'categoryIndex', 'feature1', 'feature2'	Data with 'features' vector column	Combines features into one vector
3	LogisticRegression fit	Data with 'features' and 'label'	Trained logistic regression model	Model learns from features and labels
4	Pipeline fit	Training data	PipelineModel with all stages	Pipeline chains all steps
5	PipelineModel transform	Test data	Predictions with probabilities and labels	Model applies all stages to test data
6	End	Predictions DataFrame	Final output with predictions	Pipeline execution complete

💡 All pipeline stages executed sequentially; model trained and predictions generated.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	Final
data	Raw DataFrame	With 'categoryIndex'	With 'features' vector	Used for training model	PipelineModel created	Predictions DataFrame	Predictions DataFrame

Key Moments - 3 Insights

Why do we need StringIndexer before VectorAssembler?

StringIndexer converts text categories into numbers, which VectorAssembler requires to create a numeric feature vector. See execution_table step 1 and 2.

What does the Pipeline do with multiple stages?

Pipeline chains all stages so they run in order automatically. This avoids manual step-by-step calls. See execution_table step 4.

How does the model use the 'features' column?

The model trains on the 'features' vector and 'label' column to learn patterns. See execution_table step 3.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output after Step 2?

A Trained logistic regression model

B Data with 'categoryIndex' column

C Data with 'features' vector column

D Predictions DataFrame

💡 Hint

Check the 'Output Data' column in execution_table row for Step 2.

At which step does the pipeline get created?

A Step 1

B Step 4

C Step 3

D Step 5

💡 Hint

Look for 'PipelineModel created' in the 'Output Data' column.

If we skip StringIndexer, what happens to VectorAssembler?

A It fails because it needs numeric inputs

B It works normally with text columns

C It converts text automatically

D It creates predictions directly

💡 Hint

Refer to key_moments about why StringIndexer is needed before VectorAssembler.

Concept Snapshot

ML Pipelines in Spark:
- Chain data prep, feature engineering, and model training
- Use stages like StringIndexer, VectorAssembler, and Estimators
- Fit pipeline on training data to get PipelineModel
- Transform test data to get predictions
- Automates sequential ML steps cleanly

Full Transcript

ML Pipelines in Spark help automate machine learning workflows by chaining data preparation, feature engineering, and model training steps. First, raw data is cleaned and prepared. Then, features are created, for example by converting categorical text to numeric indices using StringIndexer and combining features into a vector with VectorAssembler. Next, a model like LogisticRegression is trained on these features. All these steps are combined into a Pipeline, which is fit on training data to produce a PipelineModel. This model can then transform test data to generate predictions. This process ensures all steps run in order without manual intervention, making ML workflows easier and less error-prone.

Practice

1. What is the main purpose of using ML Pipelines in Apache Spark?

easy

A. To visualize data in charts and graphs

B. To chain multiple data processing and machine learning steps into one workflow

C. To store data permanently in a database

D. To write SQL queries for data extraction

Solution

1. Step 1: Understand ML Pipelines concept

   ML Pipelines are designed to combine multiple steps like data cleaning, feature extraction, and model training into a single workflow.

2. Step 2: Identify the main purpose

   This chaining helps automate the process and apply the same steps easily to new data.

3. Final Answer:

   To chain multiple data processing and machine learning steps into one workflow -> Option B

4. Quick Check:

   ML Pipelines = chaining steps [OK]

Hint: Remember: Pipelines automate and chain ML steps [OK]

Common Mistakes:

• Confusing pipelines with data visualization tools
• Thinking pipelines store data permanently
• Assuming pipelines are only for SQL queries

2. Which of the following is the correct way to create a Pipeline in Spark ML?

easy

A. pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

B. pipeline = Pipeline(steps=[tokenizer, hashingTF, lr])

C. pipeline = Pipeline(chain=[tokenizer, hashingTF, lr])

D. pipeline = Pipeline(process=[tokenizer, hashingTF, lr])

Solution

1. Step 1: Recall Spark ML Pipeline syntax

   In Spark ML, the Pipeline constructor expects a parameter named stages which is a list of transformers and estimators.

2. Step 2: Match the correct parameter name

   Only pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) uses stages correctly; others use incorrect parameter names.

3. Final Answer:

   pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) -> Option A

4. Quick Check:

   Use 'stages' to list pipeline steps [OK]

Hint: Use 'stages' parameter to add steps in Pipeline [OK]

Common Mistakes:

• Using 'steps' instead of 'stages'
• Using 'chain' or 'process' which are invalid
• Forgetting to pass a list of stages

3. Given the following Spark ML Pipeline code, what will be the output type of model.transform(testData)?

from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF
from pyspark.ml.classification import LogisticRegression

# Define stages
tokenizer = Tokenizer(inputCol='text', outputCol='words')
hashingTF = HashingTF(inputCol='words', outputCol='features')
lr = LogisticRegression(maxIter=10, regParam=0.01)

pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

# Fit the pipeline
model = pipeline.fit(trainingData)

# Transform test data
result = model.transform(testData)
print(type(result))

medium

A.

B.

C.

D.

Solution

1. Step 1: Understand what model.transform returns

   In Spark ML, calling transform() on a PipelineModel returns a DataFrame with new columns added by the pipeline stages.

2. Step 2: Identify the output type

   The output is a Spark DataFrame, not a list, dict, or PipelineModel.

3. Final Answer:

   <class 'pyspark.sql.dataframe.DataFrame'> -> Option D

4. Quick Check:

   model.transform() returns DataFrame [OK]

Hint: transform() always outputs a DataFrame in Spark ML [OK]

Common Mistakes:

• Thinking transform returns the model itself
• Expecting a Python list or dict instead of DataFrame
• Confusing fit() output with transform() output

4. Identify the error in the following Spark ML Pipeline code snippet:

from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF

tokenizer = Tokenizer(inputCol='text', outputCol='words')
hashingTF = HashingTF(inputCol='words', outputCol='features')

pipeline = Pipeline(stages=[tokenizer, hashingTF])
model = pipeline.fit(trainingData)
result = model.transform(testData)
print(result.select('prediction').show())

medium

A. Missing an estimator stage like LogisticRegression before fitting

B. Tokenizer inputCol should be 'words' instead of 'text'

C. Pipeline stages should be passed as a tuple, not a list

D. transform() cannot be called on the pipeline model

Solution

1. Step 1: Check pipeline stages

   The pipeline only has transformers (Tokenizer and HashingTF) but no estimator (like LogisticRegression) to train a model.

2. Step 2: Understand fit() behavior

   Fitting a pipeline without an estimator returns a PipelineModel but no trained model for prediction; this is valid but usually incomplete for ML tasks.

3. Step 3: Identify the error

   For supervised learning, missing an estimator means no model is trained, so the pipeline is incomplete.

4. Final Answer:

   Missing an estimator stage like LogisticRegression before fitting -> Option A

5. Quick Check:

   Estimator needed for training model [OK]

Hint: Always include an estimator for training in pipeline stages [OK]

Common Mistakes:

• Confusing inputCol names
• Thinking stages must be tuples
• Believing transform() can't be called on PipelineModel

5. You want to create a Spark ML Pipeline that preprocesses text data, trains a logistic regression model, and then applies the same pipeline to new streaming data. Which approach is best to ensure the pipeline works correctly on both batch and streaming data?

hard

A. Use only Tokenizer and HashingTF in the pipeline; train LogisticRegression on streaming data directly

B. Train LogisticRegression separately on batch data; apply Tokenizer and HashingTF manually on streaming data

C. Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform

D. Fit the pipeline on streaming data first, then apply it to batch data

Solution

1. Step 1: Understand pipeline reuse for batch and streaming

   To apply the same transformations and model to streaming data, you must save the trained PipelineModel from batch training.

2. Step 2: Best practice for streaming

   Fit the pipeline on batch data, save the PipelineModel, then load it in streaming jobs to transform streaming data consistently.

3. Step 3: Evaluate options

   Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform follows this best practice. Other options either separate steps manually or train on streaming data first, which is less reliable.

4. Final Answer:

   Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform -> Option C

5. Quick Check:

   Save and reuse PipelineModel for streaming [OK]

Hint: Train pipeline on batch, save model, reuse for streaming [OK]

Common Mistakes:

• Training model separately from preprocessing
• Trying to train on streaming data first
• Not saving/loading PipelineModel for reuse