Feature transformers (VectorAssembler, StandardScaler) in Apache Spark - Step-by-Step Execution

Concept Flow - Feature transformers (VectorAssembler, StandardScaler)

Start with raw columns

↓

VectorAssembler combines columns

↓

Output: single vector column

↓

StandardScaler fits on vector column

↓

Output: scaled vector column

↓

Use scaled features for ML model

Raw feature columns are combined into one vector column using VectorAssembler. Then StandardScaler scales this vector to have mean 0 and variance 1 for better model training.

Execution Sample

Apache Spark

from pyspark.ml.feature import VectorAssembler, StandardScaler
assembler = VectorAssembler(inputCols=["height", "weight"], outputCol="features")
assembled = assembler.transform(df)
scaler = StandardScaler(inputCol="features", outputCol="scaledFeatures", withMean=True, withStd=True)
scalerModel = scaler.fit(assembled)
scaledData = scalerModel.transform(assembled)

Combine height and weight into a features vector, then scale it to zero mean and unit variance.

Execution Table

Step	Action	Input Data	Output Data	Notes
1	VectorAssembler transform	{"height": 170, "weight": 70}	{"features": [170.0, 70.0]}	Combine columns into vector
2	VectorAssembler transform	{"height": 160, "weight": 60}	{"features": [160.0, 60.0]}	Combine columns into vector
3	VectorAssembler transform	{"height": 180, "weight": 80}	{"features": [180.0, 80.0]}	Combine columns into vector
4	StandardScaler fit	[[170.0, 70.0], [160.0, 60.0], [180.0, 80.0]]	Compute mean=[170,70], std=[8.16,8.16]	Calculate mean and std dev
5	StandardScaler transform	[170.0, 70.0]	[0.0, 0.0]	Center and scale first vector
6	StandardScaler transform	[160.0, 60.0]	[-1.22, -1.22]	Center and scale second vector
7	StandardScaler transform	[180.0, 80.0]	[1.22, 1.22]	Center and scale third vector
8	End	All rows processed	Data with scaledFeatures column	Transformation complete

💡 All input rows transformed and scaled vectors created

Variable Tracker

Variable	Start	After 1	After 2	After 3	Final
features	N/A	[170.0, 70.0]	[160.0, 60.0]	[180.0, 80.0]	Vector column for all rows
mean	N/A	N/A	N/A	N/A	[170.0, 70.0]
std	N/A	N/A	N/A	N/A	[8.16, 8.16]
scaledFeatures	N/A	[0.0, 0.0]	[-1.22, -1.22]	[1.22, 1.22]	Scaled vector column

Key Moments - 3 Insights

Why do we need VectorAssembler before StandardScaler?

VectorAssembler combines multiple columns into a single vector column, which StandardScaler requires as input. Without this, StandardScaler cannot process separate columns directly (see execution_table steps 1-3).

What does StandardScaler withMean=True and withStd=True do?

It centers each feature by subtracting the mean and scales by dividing by the standard deviation, resulting in features with mean 0 and variance 1 (see execution_table steps 4-7).

Why are scaled values like 0.0 or -1.22 instead of original values?

Because StandardScaler transforms original features to have zero mean and unit variance, so values represent how many standard deviations they are from the mean (see execution_table steps 5-7).

Visual Quiz - 3 Questions

Look at the execution_table at step 5. What is the scaledFeatures value for the first row?

A [-1.22, -1.22]

B [0.0, 0.0]

C [1.22, 1.22]

D [170.0, 70.0]

💡 Hint

Check the Output Data column at step 5 in execution_table

At which step does the StandardScaler compute the mean and standard deviation?

A Step 4

B Step 3

C Step 6

D Step 7

💡 Hint

Look for the step where mean and std are calculated in execution_table

If we remove VectorAssembler and try to fit StandardScaler directly on height and weight columns, what happens?

A StandardScaler works normally

B StandardScaler scales only the first column

C StandardScaler throws an error because input is not a vector

D StandardScaler ignores the columns

💡 Hint

Recall why VectorAssembler is needed before StandardScaler from key_moments

Concept Snapshot

Feature transformers combine and scale features:
- VectorAssembler merges multiple columns into one vector column
- StandardScaler centers and scales this vector to mean=0, std=1
- Use withMean=True and withStd=True for standard scaling
- Output is a scaled vector column ready for ML models

Full Transcript

We start with raw columns like height and weight. VectorAssembler combines these into a single vector column called features. Then StandardScaler calculates the mean and standard deviation of these features. It transforms each vector by subtracting the mean and dividing by the standard deviation, producing scaledFeatures with mean zero and variance one. This scaling helps machine learning models learn better. The process stops after all rows are transformed and scaled. Key points: VectorAssembler is required to combine columns before scaling. StandardScaler needs vector input. The scaled values show how far each feature is from the mean in standard deviations.

Practice

1. What is the main purpose of the VectorAssembler in Apache Spark's feature transformers?

easy

A. To remove missing values from the dataset

B. To scale features to have zero mean and unit variance

C. To combine multiple columns into a single vector column

D. To split a vector column into multiple columns

Solution

1. Step 1: Understand VectorAssembler's role

   VectorAssembler is used to combine several feature columns into one vector column, which is needed for ML algorithms.

2. Step 2: Differentiate from other transformers

   StandardScaler scales features, but VectorAssembler only combines columns into a vector.

3. Final Answer:

   To combine multiple columns into a single vector column -> Option C

4. Quick Check:

   VectorAssembler = combine columns [OK]

Hint: VectorAssembler joins columns into one vector column [OK]

Common Mistakes:

• Confusing VectorAssembler with StandardScaler
• Thinking it splits vectors instead of combining
• Assuming it handles missing values

2. Which of the following is the correct way to create a VectorAssembler in PySpark to combine columns 'age' and 'income' into 'features'?

easy

A. VectorAssembler(inputCols=['age', 'income'], outputCol='features')

B. VectorAssembler(outputCols=['age', 'income'], inputCol='features')

C. VectorAssembler(inputCol=['age', 'income'], outputCols='features')

D. VectorAssembler(columns=['age', 'income'], output='features')

Solution

1. Step 1: Recall VectorAssembler syntax

   The correct parameters are inputCols (list of input columns) and outputCol (name of output vector column).

2. Step 2: Match parameters with options

   VectorAssembler(inputCols=['age', 'income'], outputCol='features') uses inputCols and outputCol correctly; others misuse parameter names or types.

3. Final Answer:

   VectorAssembler(inputCols=['age', 'income'], outputCol='features') -> Option A

4. Quick Check:

   inputCols and outputCol are correct parameters [OK]

Hint: Use inputCols and outputCol parameters exactly [OK]

Common Mistakes:

• Swapping inputCols and outputCol names
• Using singular inputCol instead of inputCols
• Using wrong parameter names like columns or output

3. Given the following PySpark code, what will be the output of scaled_features.show()?

from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data = [(1, 10.0, 100.0), (2, 20.0, 200.0), (3, 30.0, 300.0)]
df = spark.createDataFrame(data, ['id', 'feature1', 'feature2'])

assembler = VectorAssembler(inputCols=['feature1', 'feature2'], outputCol='features')
assembled = assembler.transform(df)

scaler = StandardScaler(inputCol='features', outputCol='scaledFeatures', withMean=True, withStd=True)
scalerModel = scaler.fit(assembled)
scaled_features = scalerModel.transform(assembled).select('id', 'scaledFeatures')

medium

A. A DataFrame with 'id' and 'scaledFeatures' columns where features are scaled to zero mean and unit variance

B. A DataFrame with 'id' and 'features' columns unchanged

C. An error because StandardScaler cannot be applied after VectorAssembler

D. A DataFrame with only 'scaledFeatures' column without 'id'

Solution

1. Step 1: Understand VectorAssembler output

   VectorAssembler creates a 'features' vector column combining 'feature1' and 'feature2'.

2. Step 2: Apply StandardScaler

   StandardScaler scales 'features' to zero mean and unit variance, outputting 'scaledFeatures'.

3. Final Answer:

   A DataFrame with 'id' and 'scaledFeatures' columns where features are scaled to zero mean and unit variance -> Option A

4. Quick Check:

   VectorAssembler + StandardScaler = scaled features [OK]

Hint: StandardScaler scales VectorAssembler output vector [OK]

Common Mistakes:

• Expecting original columns unchanged after scaling
• Thinking StandardScaler causes error after VectorAssembler
• Missing 'id' column in output

4. Identify the error in the following PySpark code snippet using StandardScaler:

assembler = VectorAssembler(inputCols=['f1', 'f2'], outputCol='features')
df2 = assembler.transform(df)
scaler = StandardScaler(inputCol='features', outputCol='scaled', withMean=False, withStd=True)
scaled_df = scaler.transform(df2)

medium

A. Wrong inputCol name in StandardScaler

B. Missing call to fit() before transform() on StandardScaler

C. VectorAssembler outputCol should be 'scaled' not 'features'

D. withMean must be True for StandardScaler to work

Solution

1. Step 1: Check StandardScaler usage

   StandardScaler requires calling fit() on data before transform().

2. Step 2: Identify missing fit() call

   The code calls transform() directly on scaler without fit(), causing error.

3. Final Answer:

   Missing call to fit() before transform() on StandardScaler -> Option B

4. Quick Check:

   StandardScaler needs fit() before transform() [OK]

Hint: Always fit StandardScaler before transform [OK]

Common Mistakes:

• Calling transform() without fit() on StandardScaler
• Confusing inputCol and outputCol names
• Assuming withMean must be True

5. You have a dataset with columns 'height', 'weight', and 'age'. You want to prepare features for a machine learning model by combining these columns into a single vector and scaling them to zero mean and unit variance. Which sequence of steps correctly achieves this in PySpark?

hard

A. Use StandardScaler first on raw columns, then combine with VectorAssembler

B. Apply StandardScaler separately on each column, then combine with VectorAssembler

C. Combine columns with VectorAssembler and use StandardScaler without fitting

D. Use VectorAssembler to combine columns, then fit and transform with StandardScaler on the assembled vector

Solution

1. Step 1: Combine columns using VectorAssembler

   VectorAssembler merges 'height', 'weight', and 'age' into one vector column for ML models.

2. Step 2: Fit and transform the assembled vector with StandardScaler

   StandardScaler requires fitting on the vector column to scale features properly.

3. Final Answer:

   Use VectorAssembler to combine columns, then fit and transform with StandardScaler on the assembled vector -> Option D

4. Quick Check:

   Combine first, then scale with fit() [OK]

Hint: Combine columns first, then fit StandardScaler [OK]

Common Mistakes:

• Scaling columns before combining
• Skipping fit() on StandardScaler
• Combining after scaling