SciPy with scikit-learn pipeline - Step-by-Step Execution

Concept Flow - scikit-learn pipeline

Load Data

↓

Define Pipeline Steps

↓

Create Pipeline Object

↓

Fit Pipeline on Training Data

↓

Predict or Transform Data

↓

Evaluate or Use Results

This flow shows how data is loaded, a pipeline is created with steps, then fitted and used for prediction or transformation.

Execution Sample

SciPy

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
  ('scaler', StandardScaler()),
  ('logreg', LogisticRegression())
])

pipe.fit(X_train, y_train)
preds = pipe.predict(X_test)

This code creates a pipeline that scales data then fits a logistic regression model, then predicts on test data.

Execution Table

Step	Action	Input Data Shape	Output Data Shape	Notes
1	Load X_train, y_train	(100, 4), (100,)	(100, 4), (100,)	Data loaded with 100 samples, 4 features
2	Create Pipeline with scaler and logistic regression	N/A	Pipeline object created	Pipeline ready with two steps
3	Fit pipeline on X_train, y_train	(100, 4), (100,)	Model fitted	Scaler fit and applied, logistic regression trained
4	Predict on X_test	(20, 4)	(20,)	Predictions generated for 20 test samples
5	Output predictions	(20,)	(20,)	Final predicted labels array
6	End	N/A	N/A	Pipeline execution complete

💡 All steps completed; pipeline fit and predictions done

Variable Tracker

Variable	Start	After Step 1	After Step 3	After Step 4	Final
X_train	undefined	(100, 4)	(100, 4)	(100, 4)	(100, 4)
y_train	undefined	(100,)	(100,)	(100,)	(100,)
pipe	undefined	Pipeline object	Fitted pipeline	Fitted pipeline	Fitted pipeline
preds	undefined	undefined	undefined	(20,)	(20,)

Key Moments - 3 Insights

Why do we fit the pipeline instead of fitting scaler and model separately?

Fitting the pipeline (see execution_table step 3) ensures the scaler is fit only on training data and the same scaling is applied during prediction, avoiding data leakage.

What shape does the data have after scaling inside the pipeline?

The shape remains the same (100 samples, 4 features) as scaling changes values but not dimensions, shown in variable_tracker for X_train after step 3.

Why do predictions have shape (20,) after step 4?

Because predictions are labels for each test sample, one per sample, so 20 samples produce 20 predicted labels, as shown in execution_table step 4.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what happens during pipeline fitting?

A Scaler is fit on test data

B Only logistic regression is fit, scaler is ignored

C Scaler and logistic regression are both fit on training data

D Pipeline predicts without fitting

💡 Hint

Refer to execution_table row with Step 3 describing fitting process

According to variable_tracker, what is the shape of preds after step 4?

A (20,)

B (100, 4)

C (100,)

D (20, 4)

💡 Hint

Check preds variable column 'After Step 4' in variable_tracker

If we skip scaling in the pipeline, how would execution_table step 3 change?

A Pipeline would fail to fit

B Scaler fit step would be missing, model fit remains same

C Predictions would be shape (100, 4)

D Data shape would change to (20,)

💡 Hint

Think about what happens if scaler step is removed from pipeline steps

Concept Snapshot

scikit-learn pipeline:
- Use Pipeline to chain steps like scaling and modeling
- Fit pipeline on training data to avoid data leakage
- Predict or transform data using pipeline
- Keeps code clean and reproducible
- Data shape stays consistent through pipeline steps

Full Transcript

This visual execution shows how to use a scikit-learn pipeline. First, data is loaded with 100 samples and 4 features. Then a pipeline is created with two steps: StandardScaler and LogisticRegression. The pipeline is fit on training data, which fits the scaler and model in order. After fitting, predictions are made on test data with 20 samples. Variables like X_train, y_train, pipeline object, and predictions change state through the steps. Key moments clarify why fitting the pipeline is important to avoid data leakage, how data shape remains the same after scaling, and why predictions have shape matching test samples. The quiz tests understanding of pipeline fitting, prediction shapes, and effects of skipping scaling. The snapshot summarizes pipeline usage for clean, reproducible modeling.

Practice

1. What is the main benefit of using a Pipeline in scikit-learn when combined with SciPy functions?

easy

A. It organizes data processing and modeling steps into one repeatable workflow.

B. It automatically improves model accuracy without tuning.

C. It replaces the need for any data cleaning.

D. It allows running code without importing any libraries.

Solution

1. Step 1: Understand the purpose of Pipeline

   A Pipeline in scikit-learn is designed to chain multiple steps like data transformation and modeling into a single object.

2. Step 2: Recognize the benefit of combining SciPy functions

   Using SciPy functions inside a Pipeline via FunctionTransformer keeps the workflow organized and repeatable.

3. Final Answer:

   It organizes data processing and modeling steps into one repeatable workflow. -> Option A

4. Quick Check:

   Pipeline = Organized workflow [OK]

Hint: Pipelines bundle steps for easy reuse and clarity [OK]

Common Mistakes:

• Thinking Pipeline improves accuracy automatically
• Assuming Pipeline removes need for data cleaning
• Believing Pipeline runs without imports

2. Which of the following is the correct way to include a SciPy function scipy_func inside a scikit-learn pipeline using FunctionTransformer?

easy

A. Pipeline([('transform', scipy_func), ('model', LogisticRegression())])

B. Pipeline([('transform', FunctionTransformer(scipy_func)), ('model', LogisticRegression())])

C. Pipeline([('transform', FunctionTransformer()), ('model', LogisticRegression())])

D. Pipeline([('transform', FunctionTransformer(scipy_func())), ('model', LogisticRegression())])

Solution

1. Step 1: Understand FunctionTransformer usage

   FunctionTransformer takes a function as an argument without calling it (no parentheses).

2. Step 2: Identify correct pipeline syntax

   The pipeline step should be ('transform', FunctionTransformer(scipy_func)) to wrap the function properly.

3. Final Answer:

   Pipeline([('transform', FunctionTransformer(scipy_func)), ('model', LogisticRegression())]) -> Option B

4. Quick Check:

   FunctionTransformer(function) no parentheses [OK]

Hint: Pass function name, not call, to FunctionTransformer [OK]

Common Mistakes:

• Calling the function inside FunctionTransformer
• Passing function directly without FunctionTransformer
• Using FunctionTransformer without function argument

3. What will be the output of the following code snippet?

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
import numpy as np

def add_one(X):
    return X + 1

pipe = Pipeline([
    ('add', FunctionTransformer(add_one)),
])

X = np.array([1, 2, 3])
result = pipe.transform(X)
print(result)

medium

A. Error: Pipeline has no transform method

B. [1 2 3]

C. [0 1 2]

D. [2 3 4]

Solution

1. Step 1: Understand FunctionTransformer behavior

   FunctionTransformer applies the function add_one to input data during transform.

2. Step 2: Apply the function to input array

   Input array [1, 2, 3] plus 1 becomes [2, 3, 4].

3. Final Answer:

   [2 3 4] -> Option D

4. Quick Check:

   Input + 1 = Output [OK]

Hint: FunctionTransformer applies function on transform call [OK]

Common Mistakes:

• Assuming pipeline has no transform method
• Forgetting function adds 1
• Confusing fit and transform methods

4. Identify the error in this pipeline code using a SciPy function inside FunctionTransformer:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
import numpy as np

def multiply_by_two(X):
    return X * 2

pipe = Pipeline([
    ('mult', FunctionTransformer(multiply_by_two())),
])

X = np.array([1, 2, 3])
result = pipe.transform(X)
print(result)

medium

A. Using transform instead of fit_transform

B. Missing import for numpy

C. Calling multiply_by_two() instead of passing the function

D. Pipeline missing a model step

Solution

1. Step 1: Check FunctionTransformer argument

   FunctionTransformer expects a function, not the result of a function call.

2. Step 2: Identify the error in code

   Code calls multiply_by_two() immediately, which causes an error because it returns an array, not a function.

3. Final Answer:

   Calling multiply_by_two() instead of passing the function -> Option C

4. Quick Check:

   Pass function, don't call it [OK]

Hint: Pass function name, avoid parentheses in FunctionTransformer [OK]

Common Mistakes:

• Calling function instead of passing it
• Assuming pipeline needs a model step
• Confusing transform with fit_transform

5. You want to build a pipeline that first applies a SciPy function to normalize data, then fits a logistic regression model. Which of the following code snippets correctly implements this?

hard

A. from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.linear_model import LogisticRegression
import scipy.stats as stats

pipe = Pipeline([
('normalize', FunctionTransformer(stats.zscore)),
('model', LogisticRegression())
])

B. from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.linear_model import LogisticRegression
import scipy.stats as stats

pipe = Pipeline([
('normalize', stats.zscore()),
('model', LogisticRegression())
])

C. from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.linear_model import LogisticRegression
import scipy.stats as stats

pipe = Pipeline([
('normalize', FunctionTransformer(stats.zscore())),
('model', LogisticRegression())
])

D. from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.linear_model import LogisticRegression
import scipy.stats as stats

pipe = Pipeline([
('normalize', FunctionTransformer(stats.zscore)),
('model', LogisticRegression)
])

Solution

1. Step 1: Use FunctionTransformer correctly with SciPy function

   Pass the function stats.zscore without calling it, wrapped by FunctionTransformer.

2. Step 2: Ensure LogisticRegression is instantiated

   Use LogisticRegression() with parentheses to create the model instance.

3. Final Answer:

   Code snippet with FunctionTransformer(stats.zscore) and LogisticRegression() -> Option A

4. Quick Check:

   FunctionTransformer(function) + model instance [OK]

Hint: Wrap function, instantiate model with parentheses [OK]

Common Mistakes:

• Calling SciPy function instead of passing it
• Not instantiating LogisticRegression
• Passing function call to FunctionTransformer