Feature transformers (VectorAssembler, StandardScaler) in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of VectorAssembler on simple DataFrame

Given the following Spark DataFrame and VectorAssembler code, what is the output of the transformed DataFrame's 'features' column?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler

spark = SparkSession.builder.getOrCreate()
data = [(1, 2.0, 3.0), (4, 5.0, 6.0)]
columns = ['id', 'f1', 'f2']
df = spark.createDataFrame(data, columns)
assembler = VectorAssembler(inputCols=['f1', 'f2'], outputCol='features')
output = assembler.transform(df).select('features').collect()

A [DenseVector([2.0, 3.0]), DenseVector([5.0, 6.0])]

B [DenseVector([1.0, 2.0, 3.0]), DenseVector([4.0, 5.0, 6.0])]

C [DenseVector([1.0]), DenseVector([4.0])]

D [DenseVector([3.0, 2.0]), DenseVector([6.0, 5.0])]

💡 Hint

VectorAssembler combines specified columns into a single vector column in the order given.

VectorAssembler takes the columns 'f1' and 'f2' and combines them into a vector. The 'id' column is not included. So the output vectors are [2.0, 3.0] and [5.0, 6.0].

❓ data_output

intermediate

Effect of StandardScaler on feature vectors

After applying StandardScaler with mean centering and scaling to unit variance on the following feature vectors, what is the resulting scaled vector for the first row?

Apache Spark

from pyspark.ml.feature import StandardScaler
from pyspark.ml.linalg import Vectors
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data = [(Vectors.dense([1.0, 0.0, -1.0]),), (Vectors.dense([2.0, 1.0, 1.0]),)]
df = spark.createDataFrame(data, ['features'])
scaler = StandardScaler(inputCol='features', outputCol='scaledFeatures', withMean=True, withStd=True)
scalerModel = scaler.fit(df)
scaledData = scalerModel.transform(df).select('scaledFeatures').collect()

A [DenseVector([-1.0, -0.7071, -1.0])]

B [DenseVector([-0.7071, -0.7071, -0.7071])]

C [DenseVector([-1.0, -0.7071, -0.7071])]

D [DenseVector([-1.0, -0.7071, -1.4142])]

💡 Hint

StandardScaler subtracts the mean and divides by standard deviation for each feature.

The mean vector is [1.5, 0.5, 0.0] and std deviation vector is [0.7071, 0.7071, 1.4142] (sample std dev, divide by n-1). The first vector scaled is ((1-1.5)/0.7071, (0-0.5)/0.7071, (-1-0)/1.4142) ≈ (-0.7071, -0.7071, -0.7071).

🔧 Debug

advanced

Identify the error in VectorAssembler usage

What error will this Spark code raise when trying to assemble features?

Apache Spark

from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(inputCols=['a', 'b'], outputCol='features')
data = [(1, 2), (3, 4)]
df = spark.createDataFrame(data, ['a', 'c'])
assembler.transform(df).show()

A AnalysisException: Column b does not exist.

B TypeError: inputCols must be a list of strings.

C No error, outputs features column with nulls.

D ValueError: outputCol must be a string.

💡 Hint

Check if all inputCols exist in the DataFrame.

The DataFrame has columns 'a' and 'c', but VectorAssembler expects 'a' and 'b'. Missing 'b' causes AnalysisException.

🚀 Application

advanced

Choosing correct parameters for StandardScaler

You have a dataset with features on very different scales and some features centered around zero. Which StandardScaler parameters should you use to ensure all features have zero mean and unit variance?

A withMean=True, withStd=False

B withMean=False, withStd=True

C withMean=True, withStd=True

D withMean=False, withStd=False

💡 Hint

Zero mean means centering, unit variance means scaling by std deviation.

Setting withMean=True centers data to zero mean. Setting withStd=True scales data to unit variance. Both are needed for zero mean and unit variance.

🧠 Conceptual

expert

Understanding VectorAssembler output schema

After applying VectorAssembler on a DataFrame with columns ['x', 'y', 'z'], what is the data type of the output column 'features'?

A StringType representing vector as text

B ArrayType(DoubleType)

C StructType with fields x, y, z

D VectorUDT (a Spark MLlib vector type)

💡 Hint

VectorAssembler outputs a single vector column compatible with MLlib.

VectorAssembler outputs a column of type VectorUDT, which is Spark MLlib's vector type used for machine learning pipelines.

Practice

1. What is the main purpose of the VectorAssembler in Apache Spark's feature transformers?

easy

A. To remove missing values from the dataset

B. To scale features to have zero mean and unit variance

C. To combine multiple columns into a single vector column

D. To split a vector column into multiple columns

Solution

1. Step 1: Understand VectorAssembler's role

   VectorAssembler is used to combine several feature columns into one vector column, which is needed for ML algorithms.

2. Step 2: Differentiate from other transformers

   StandardScaler scales features, but VectorAssembler only combines columns into a vector.

3. Final Answer:

   To combine multiple columns into a single vector column -> Option C

4. Quick Check:

   VectorAssembler = combine columns [OK]

Hint: VectorAssembler joins columns into one vector column [OK]

Common Mistakes:

• Confusing VectorAssembler with StandardScaler
• Thinking it splits vectors instead of combining
• Assuming it handles missing values

2. Which of the following is the correct way to create a VectorAssembler in PySpark to combine columns 'age' and 'income' into 'features'?

easy

A. VectorAssembler(inputCols=['age', 'income'], outputCol='features')

B. VectorAssembler(outputCols=['age', 'income'], inputCol='features')

C. VectorAssembler(inputCol=['age', 'income'], outputCols='features')

D. VectorAssembler(columns=['age', 'income'], output='features')

Solution

1. Step 1: Recall VectorAssembler syntax

   The correct parameters are inputCols (list of input columns) and outputCol (name of output vector column).

2. Step 2: Match parameters with options

   VectorAssembler(inputCols=['age', 'income'], outputCol='features') uses inputCols and outputCol correctly; others misuse parameter names or types.

3. Final Answer:

   VectorAssembler(inputCols=['age', 'income'], outputCol='features') -> Option A

4. Quick Check:

   inputCols and outputCol are correct parameters [OK]

Hint: Use inputCols and outputCol parameters exactly [OK]

Common Mistakes:

• Swapping inputCols and outputCol names
• Using singular inputCol instead of inputCols
• Using wrong parameter names like columns or output

3. Given the following PySpark code, what will be the output of scaled_features.show()?

from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data = [(1, 10.0, 100.0), (2, 20.0, 200.0), (3, 30.0, 300.0)]
df = spark.createDataFrame(data, ['id', 'feature1', 'feature2'])

assembler = VectorAssembler(inputCols=['feature1', 'feature2'], outputCol='features')
assembled = assembler.transform(df)

scaler = StandardScaler(inputCol='features', outputCol='scaledFeatures', withMean=True, withStd=True)
scalerModel = scaler.fit(assembled)
scaled_features = scalerModel.transform(assembled).select('id', 'scaledFeatures')

medium

A. A DataFrame with 'id' and 'scaledFeatures' columns where features are scaled to zero mean and unit variance

B. A DataFrame with 'id' and 'features' columns unchanged

C. An error because StandardScaler cannot be applied after VectorAssembler

D. A DataFrame with only 'scaledFeatures' column without 'id'

Solution

1. Step 1: Understand VectorAssembler output

   VectorAssembler creates a 'features' vector column combining 'feature1' and 'feature2'.

2. Step 2: Apply StandardScaler

   StandardScaler scales 'features' to zero mean and unit variance, outputting 'scaledFeatures'.

3. Final Answer:

   A DataFrame with 'id' and 'scaledFeatures' columns where features are scaled to zero mean and unit variance -> Option A

4. Quick Check:

   VectorAssembler + StandardScaler = scaled features [OK]

Hint: StandardScaler scales VectorAssembler output vector [OK]

Common Mistakes:

• Expecting original columns unchanged after scaling
• Thinking StandardScaler causes error after VectorAssembler
• Missing 'id' column in output

4. Identify the error in the following PySpark code snippet using StandardScaler:

assembler = VectorAssembler(inputCols=['f1', 'f2'], outputCol='features')
df2 = assembler.transform(df)
scaler = StandardScaler(inputCol='features', outputCol='scaled', withMean=False, withStd=True)
scaled_df = scaler.transform(df2)

medium

A. Wrong inputCol name in StandardScaler

B. Missing call to fit() before transform() on StandardScaler

C. VectorAssembler outputCol should be 'scaled' not 'features'

D. withMean must be True for StandardScaler to work

Solution

1. Step 1: Check StandardScaler usage

   StandardScaler requires calling fit() on data before transform().

2. Step 2: Identify missing fit() call

   The code calls transform() directly on scaler without fit(), causing error.

3. Final Answer:

   Missing call to fit() before transform() on StandardScaler -> Option B

4. Quick Check:

   StandardScaler needs fit() before transform() [OK]

Hint: Always fit StandardScaler before transform [OK]

Common Mistakes:

• Calling transform() without fit() on StandardScaler
• Confusing inputCol and outputCol names
• Assuming withMean must be True

5. You have a dataset with columns 'height', 'weight', and 'age'. You want to prepare features for a machine learning model by combining these columns into a single vector and scaling them to zero mean and unit variance. Which sequence of steps correctly achieves this in PySpark?

hard

A. Use StandardScaler first on raw columns, then combine with VectorAssembler

B. Apply StandardScaler separately on each column, then combine with VectorAssembler

C. Combine columns with VectorAssembler and use StandardScaler without fitting

D. Use VectorAssembler to combine columns, then fit and transform with StandardScaler on the assembled vector

Solution

1. Step 1: Combine columns using VectorAssembler

   VectorAssembler merges 'height', 'weight', and 'age' into one vector column for ML models.

2. Step 2: Fit and transform the assembled vector with StandardScaler

   StandardScaler requires fitting on the vector column to scale features properly.

3. Final Answer:

   Use VectorAssembler to combine columns, then fit and transform with StandardScaler on the assembled vector -> Option D

4. Quick Check:

   Combine first, then scale with fit() [OK]

Hint: Combine columns first, then fit StandardScaler [OK]

Common Mistakes:

• Scaling columns before combining
• Skipping fit() on StandardScaler
• Combining after scaling