ML Pipelines in Spark in Apache Spark

Introduction

ML Pipelines help organize and automate machine learning steps in Spark. They make building and testing models easier and faster.

When you want to prepare data and train a model in one flow.

When you need to repeat the same steps on new data easily.

When you want to combine multiple data transformations and model training.

When you want to avoid manual errors by automating ML workflows.

When you want to save and reuse your machine learning process.

Syntax

Apache Spark

from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF
from pyspark.ml.classification import LogisticRegression

# Define stages
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol="words", outputCol="features")
logReg = LogisticRegression(featuresCol="features", labelCol="label")

# Create pipeline
pipeline = Pipeline(stages=[tokenizer, hashingTF, logReg])

A pipeline is a sequence of stages, each stage is a transformer or an estimator.

Transformers change data, estimators train models.

Examples

This example shows a pipeline with one stage that converts categories to numbers.

Apache Spark

from pyspark.ml.feature import StringIndexer

indexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
pipeline = Pipeline(stages=[indexer])

This pipeline stage combines multiple columns into a single features column.

Apache Spark

from pyspark.ml.feature import VectorAssembler

assembler = VectorAssembler(inputCols=["col1", "col2"], outputCol="features")
pipeline = Pipeline(stages=[assembler])

Sample Program

This program creates a simple ML pipeline in Spark that tokenizes text, converts words to features, and trains a logistic regression model. Then it predicts labels on new text data.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF
from pyspark.ml.classification import LogisticRegression

# Start Spark session
spark = SparkSession.builder.appName("MLPipelineExample").getOrCreate()

# Sample data
training = spark.createDataFrame([
    (0, "spark is great", 1.0),
    (1, "spark is fast", 1.0),
    (2, "hadoop is slow", 0.0),
    (3, "hadoop is old", 0.0)
], ["id", "text", "label"])

# Define stages
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol="words", outputCol="features")
logReg = LogisticRegression(featuresCol="features", labelCol="label")

# Create pipeline
pipeline = Pipeline(stages=[tokenizer, hashingTF, logReg])

# Train model
model = pipeline.fit(training)

# Test data
test = spark.createDataFrame([
    (4, "spark is awesome"),
    (5, "hadoop is outdated")
], ["id", "text"])

# Make predictions
predictions = model.transform(test)

# Show results
predictions.select("id", "text", "probability", "prediction").show()

spark.stop()

Important Notes

Always define clear input and output columns for each stage.

Use Pipeline.fit() to train and PipelineModel.transform() to apply the model.

ML Pipelines help keep your code clean and reusable.

Summary

ML Pipelines chain data preparation and model training steps.

They automate workflows and reduce errors.

Use pipelines to easily apply the same process to new data.

Practice

1. What is the main purpose of using ML Pipelines in Apache Spark?

easy

A. To visualize data in charts and graphs

B. To chain multiple data processing and machine learning steps into one workflow

C. To store data permanently in a database

D. To write SQL queries for data extraction

Solution

1. Step 1: Understand ML Pipelines concept

   ML Pipelines are designed to combine multiple steps like data cleaning, feature extraction, and model training into a single workflow.

2. Step 2: Identify the main purpose

   This chaining helps automate the process and apply the same steps easily to new data.

3. Final Answer:

   To chain multiple data processing and machine learning steps into one workflow -> Option B

4. Quick Check:

   ML Pipelines = chaining steps [OK]

Hint: Remember: Pipelines automate and chain ML steps [OK]

Common Mistakes:

• Confusing pipelines with data visualization tools
• Thinking pipelines store data permanently
• Assuming pipelines are only for SQL queries

2. Which of the following is the correct way to create a Pipeline in Spark ML?

easy

A. pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

B. pipeline = Pipeline(steps=[tokenizer, hashingTF, lr])

C. pipeline = Pipeline(chain=[tokenizer, hashingTF, lr])

D. pipeline = Pipeline(process=[tokenizer, hashingTF, lr])

Solution

1. Step 1: Recall Spark ML Pipeline syntax

   In Spark ML, the Pipeline constructor expects a parameter named stages which is a list of transformers and estimators.

2. Step 2: Match the correct parameter name

   Only pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) uses stages correctly; others use incorrect parameter names.

3. Final Answer:

   pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) -> Option A

4. Quick Check:

   Use 'stages' to list pipeline steps [OK]

Hint: Use 'stages' parameter to add steps in Pipeline [OK]

Common Mistakes:

• Using 'steps' instead of 'stages'
• Using 'chain' or 'process' which are invalid
• Forgetting to pass a list of stages

3. Given the following Spark ML Pipeline code, what will be the output type of model.transform(testData)?

from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF
from pyspark.ml.classification import LogisticRegression

# Define stages
tokenizer = Tokenizer(inputCol='text', outputCol='words')
hashingTF = HashingTF(inputCol='words', outputCol='features')
lr = LogisticRegression(maxIter=10, regParam=0.01)

pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

# Fit the pipeline
model = pipeline.fit(trainingData)

# Transform test data
result = model.transform(testData)
print(type(result))

medium

A.

B.

C.

D.

Solution

1. Step 1: Understand what model.transform returns

   In Spark ML, calling transform() on a PipelineModel returns a DataFrame with new columns added by the pipeline stages.

2. Step 2: Identify the output type

   The output is a Spark DataFrame, not a list, dict, or PipelineModel.

3. Final Answer:

   <class 'pyspark.sql.dataframe.DataFrame'> -> Option D

4. Quick Check:

   model.transform() returns DataFrame [OK]

Hint: transform() always outputs a DataFrame in Spark ML [OK]

Common Mistakes:

• Thinking transform returns the model itself
• Expecting a Python list or dict instead of DataFrame
• Confusing fit() output with transform() output

4. Identify the error in the following Spark ML Pipeline code snippet:

from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF

tokenizer = Tokenizer(inputCol='text', outputCol='words')
hashingTF = HashingTF(inputCol='words', outputCol='features')

pipeline = Pipeline(stages=[tokenizer, hashingTF])
model = pipeline.fit(trainingData)
result = model.transform(testData)
print(result.select('prediction').show())

medium

A. Missing an estimator stage like LogisticRegression before fitting

B. Tokenizer inputCol should be 'words' instead of 'text'

C. Pipeline stages should be passed as a tuple, not a list

D. transform() cannot be called on the pipeline model

Solution

1. Step 1: Check pipeline stages

   The pipeline only has transformers (Tokenizer and HashingTF) but no estimator (like LogisticRegression) to train a model.

2. Step 2: Understand fit() behavior

   Fitting a pipeline without an estimator returns a PipelineModel but no trained model for prediction; this is valid but usually incomplete for ML tasks.

3. Step 3: Identify the error

   For supervised learning, missing an estimator means no model is trained, so the pipeline is incomplete.

4. Final Answer:

   Missing an estimator stage like LogisticRegression before fitting -> Option A

5. Quick Check:

   Estimator needed for training model [OK]

Hint: Always include an estimator for training in pipeline stages [OK]

Common Mistakes:

• Confusing inputCol names
• Thinking stages must be tuples
• Believing transform() can't be called on PipelineModel

5. You want to create a Spark ML Pipeline that preprocesses text data, trains a logistic regression model, and then applies the same pipeline to new streaming data. Which approach is best to ensure the pipeline works correctly on both batch and streaming data?

hard

A. Use only Tokenizer and HashingTF in the pipeline; train LogisticRegression on streaming data directly

B. Train LogisticRegression separately on batch data; apply Tokenizer and HashingTF manually on streaming data

C. Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform

D. Fit the pipeline on streaming data first, then apply it to batch data

Solution

1. Step 1: Understand pipeline reuse for batch and streaming

   To apply the same transformations and model to streaming data, you must save the trained PipelineModel from batch training.

2. Step 2: Best practice for streaming

   Fit the pipeline on batch data, save the PipelineModel, then load it in streaming jobs to transform streaming data consistently.

3. Step 3: Evaluate options

   Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform follows this best practice. Other options either separate steps manually or train on streaming data first, which is less reliable.

4. Final Answer:

   Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform -> Option C

5. Quick Check:

   Save and reuse PipelineModel for streaming [OK]

Hint: Train pipeline on batch, save model, reuse for streaming [OK]

Common Mistakes:

• Training model separately from preprocessing
• Trying to train on streaming data first
• Not saving/loading PipelineModel for reuse