ML Pipelines in Spark in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the Pipeline class from Spark ML.

Apache Spark

from pyspark.ml import [1]

Drag options to blanks, or click blank then click option'

APipeline

BPipelineModel

CTransformer

DEstimator

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing Pipeline with PipelineModel or Transformer.

Trying to import from pyspark.ml.pipeline instead of pyspark.ml.

The Pipeline class is used to create ML pipelines in Spark.

2fill in blank

medium

Complete the code to create a pipeline with stages stored in a list called 'stages'.

Apache Spark

pipeline = Pipeline([1]=stages)

Drag options to blanks, or click blank then click option'

Asteps

Bstages

Ctransformers

Destimators

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'steps' instead of 'stages'.

Passing a single stage instead of a list.

The Pipeline constructor expects a parameter named stages which is a list of pipeline stages.

3fill in blank

hard

Fix the error in the code to fit the pipeline on a DataFrame called 'df'.

Apache Spark

model = pipeline.[1](df)

Drag options to blanks, or click blank then click option'

Atransform

Bpredict

Cfit

Devaluate

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'transform' instead of 'fit' to train the pipeline.

Trying to use 'predict' which is not a Pipeline method.

To train the pipeline, use the fit method on the DataFrame.

4fill in blank

hard

Fill both blanks to create a pipeline stage that converts a string column 'category' into numeric indices.

Apache Spark

indexer = StringIndexer(inputCol=[1], outputCol=[2])

Drag options to blanks, or click blank then click option'

A'category'

B'categoryIndex'

C'label'

D'features'

Attempts:

3 left

💡 Hint

Common Mistakes

Swapping inputCol and outputCol values.

Using 'label' or 'features' instead of the correct column names.

The inputCol is the original string column, and outputCol is the new numeric column name.

5fill in blank

hard

Fill all three blanks to create a pipeline stage that assembles features from columns 'col1' and 'col2' into a vector column 'features'.

Apache Spark

assembler = VectorAssembler(inputCols=[1], outputCol=[2], handleInvalid=[3])

Drag options to blanks, or click blank then click option'

A['col1', 'col2']

B'features'

C'skip'

D'ignore'

Attempts:

3 left

💡 Hint

Common Mistakes

Passing inputCols as a string instead of a list.

Using 'ignore' instead of 'skip' for handleInvalid.

Misnaming the output column.

The inputCols is a list of columns to combine, outputCol is the name of the vector column, and handleInvalid defines how to handle invalid data.

Practice

1. What is the main purpose of using ML Pipelines in Apache Spark?

easy

A. To visualize data in charts and graphs

B. To chain multiple data processing and machine learning steps into one workflow

C. To store data permanently in a database

D. To write SQL queries for data extraction

Solution

1. Step 1: Understand ML Pipelines concept

   ML Pipelines are designed to combine multiple steps like data cleaning, feature extraction, and model training into a single workflow.

2. Step 2: Identify the main purpose

   This chaining helps automate the process and apply the same steps easily to new data.

3. Final Answer:

   To chain multiple data processing and machine learning steps into one workflow -> Option B

4. Quick Check:

   ML Pipelines = chaining steps [OK]

Hint: Remember: Pipelines automate and chain ML steps [OK]

Common Mistakes:

• Confusing pipelines with data visualization tools
• Thinking pipelines store data permanently
• Assuming pipelines are only for SQL queries

2. Which of the following is the correct way to create a Pipeline in Spark ML?

easy

A. pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

B. pipeline = Pipeline(steps=[tokenizer, hashingTF, lr])

C. pipeline = Pipeline(chain=[tokenizer, hashingTF, lr])

D. pipeline = Pipeline(process=[tokenizer, hashingTF, lr])

Solution

1. Step 1: Recall Spark ML Pipeline syntax

   In Spark ML, the Pipeline constructor expects a parameter named stages which is a list of transformers and estimators.

2. Step 2: Match the correct parameter name

   Only pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) uses stages correctly; others use incorrect parameter names.

3. Final Answer:

   pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) -> Option A

4. Quick Check:

   Use 'stages' to list pipeline steps [OK]

Hint: Use 'stages' parameter to add steps in Pipeline [OK]

Common Mistakes:

• Using 'steps' instead of 'stages'
• Using 'chain' or 'process' which are invalid
• Forgetting to pass a list of stages

3. Given the following Spark ML Pipeline code, what will be the output type of model.transform(testData)?

from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF
from pyspark.ml.classification import LogisticRegression

# Define stages
tokenizer = Tokenizer(inputCol='text', outputCol='words')
hashingTF = HashingTF(inputCol='words', outputCol='features')
lr = LogisticRegression(maxIter=10, regParam=0.01)

pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

# Fit the pipeline
model = pipeline.fit(trainingData)

# Transform test data
result = model.transform(testData)
print(type(result))

medium

A.

B.

C.

D.

Solution

1. Step 1: Understand what model.transform returns

   In Spark ML, calling transform() on a PipelineModel returns a DataFrame with new columns added by the pipeline stages.

2. Step 2: Identify the output type

   The output is a Spark DataFrame, not a list, dict, or PipelineModel.

3. Final Answer:

   <class 'pyspark.sql.dataframe.DataFrame'> -> Option D

4. Quick Check:

   model.transform() returns DataFrame [OK]

Hint: transform() always outputs a DataFrame in Spark ML [OK]

Common Mistakes:

• Thinking transform returns the model itself
• Expecting a Python list or dict instead of DataFrame
• Confusing fit() output with transform() output

4. Identify the error in the following Spark ML Pipeline code snippet:

from pyspark.ml import Pipeline
from pyspark.ml.feature import Tokenizer, HashingTF

tokenizer = Tokenizer(inputCol='text', outputCol='words')
hashingTF = HashingTF(inputCol='words', outputCol='features')

pipeline = Pipeline(stages=[tokenizer, hashingTF])
model = pipeline.fit(trainingData)
result = model.transform(testData)
print(result.select('prediction').show())

medium

A. Missing an estimator stage like LogisticRegression before fitting

B. Tokenizer inputCol should be 'words' instead of 'text'

C. Pipeline stages should be passed as a tuple, not a list

D. transform() cannot be called on the pipeline model

Solution

1. Step 1: Check pipeline stages

   The pipeline only has transformers (Tokenizer and HashingTF) but no estimator (like LogisticRegression) to train a model.

2. Step 2: Understand fit() behavior

   Fitting a pipeline without an estimator returns a PipelineModel but no trained model for prediction; this is valid but usually incomplete for ML tasks.

3. Step 3: Identify the error

   For supervised learning, missing an estimator means no model is trained, so the pipeline is incomplete.

4. Final Answer:

   Missing an estimator stage like LogisticRegression before fitting -> Option A

5. Quick Check:

   Estimator needed for training model [OK]

Hint: Always include an estimator for training in pipeline stages [OK]

Common Mistakes:

• Confusing inputCol names
• Thinking stages must be tuples
• Believing transform() can't be called on PipelineModel

5. You want to create a Spark ML Pipeline that preprocesses text data, trains a logistic regression model, and then applies the same pipeline to new streaming data. Which approach is best to ensure the pipeline works correctly on both batch and streaming data?

hard

A. Use only Tokenizer and HashingTF in the pipeline; train LogisticRegression on streaming data directly

B. Train LogisticRegression separately on batch data; apply Tokenizer and HashingTF manually on streaming data

C. Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform

D. Fit the pipeline on streaming data first, then apply it to batch data

Solution

1. Step 1: Understand pipeline reuse for batch and streaming

   To apply the same transformations and model to streaming data, you must save the trained PipelineModel from batch training.

2. Step 2: Best practice for streaming

   Fit the pipeline on batch data, save the PipelineModel, then load it in streaming jobs to transform streaming data consistently.

3. Step 3: Evaluate options

   Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform follows this best practice. Other options either separate steps manually or train on streaming data first, which is less reliable.

4. Final Answer:

   Build a Pipeline with Tokenizer, HashingTF, LogisticRegression; fit on batch data; save and load the PipelineModel for streaming transform -> Option C

5. Quick Check:

   Save and reuse PipelineModel for streaming [OK]

Hint: Train pipeline on batch, save model, reuse for streaming [OK]

Common Mistakes:

• Training model separately from preprocessing
• Trying to train on streaming data first
• Not saving/loading PipelineModel for reuse