Why data preparation consumes most ML time in ML Python - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why does data cleaning take so much time in ML projects?

In machine learning, data cleaning is a major part of data preparation. Why does it usually take the most time?

A Because raw data often has errors, missing values, and inconsistencies that need fixing before training.

B Because training models requires a lot of computing power and time.

C Because data cleaning involves writing complex algorithms to improve model accuracy.

D Because data cleaning is automated and runs slowly on large datasets.

💡 Hint

Think about the quality of data collected from real-world sources.

Raw data is often messy with errors and missing parts. Fixing these issues manually or with scripts takes time, which is why data cleaning dominates the preparation phase.

🧠 Conceptual

intermediate

What is the main reason feature engineering is time-consuming?

Feature engineering is a key step in data preparation. Why does it often consume a lot of time?

A Because it only works on small datasets and needs manual scaling.

B Because it involves training multiple models to select features.

C Because it requires domain knowledge to create meaningful features from raw data.

D Because it is fully automated and requires tuning many parameters.

💡 Hint

Think about how features relate to the problem you want to solve.

Feature engineering needs understanding of the data and problem to create useful inputs for models. This manual process takes time and effort.

❓ Metrics

advanced

Measuring data quality impact on model accuracy

You have two datasets: Dataset A is clean and Dataset B has 20% missing values. You train the same model on both. Which metric difference best shows the impact of data quality?

A Dataset A precision: 0.5, Dataset B precision: 0.5

B Dataset A loss: 0.2, Dataset B loss: 0.2

C Dataset A training time: 10 minutes, Dataset B training time: 15 minutes

D Dataset A accuracy: 90%, Dataset B accuracy: 70%

💡 Hint

Look for a metric that directly reflects model performance difference.

Accuracy difference shows how missing data reduces model performance. Same loss or precision values do not reflect this clearly.

🔧 Debug

advanced

Why does this data scaling code cause poor model results?

Consider this Python snippet for scaling features before training:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.fit_transform(X_test)

Why might this cause poor model performance?

A Because the scaler is fit separately on test data, causing data leakage and inconsistent scaling.

B Because StandardScaler cannot be used on numeric data.

C Because scaling is not needed before training models.

D Because fit_transform should be replaced with transform on training data.

💡 Hint

Think about how scaling should be applied to training and test sets.

The scaler must be fit only on training data to learn scaling parameters. Fitting again on test data causes different scaling and leaks information.

❓ Model Choice

expert

Choosing the best approach to handle missing data in a large dataset

You have a large dataset with 30% missing values scattered randomly. Which approach is best to prepare data for a machine learning model?

A Remove all rows with missing values to keep only complete data.

B Fill missing values with the mean or median of each feature before training.

C Use a model that can handle missing values natively without imputation.

D Replace missing values with zeros regardless of feature meaning.

💡 Hint

Consider balancing data loss and preserving information.

Removing 30% of data loses too much information. Filling missing values with mean/median preserves data and is simple. Some models handle missing data but are less common. Zero filling can distort features.

Practice

1. Why does data preparation usually take the most time in a machine learning project?

easy

A. Because testing the model takes a long time

B. Because training the model requires many calculations

C. Because raw data is often messy and needs cleaning before use

D. Because deploying the model to production is complex

Solution

1. Step 1: Understand the nature of raw data

   Raw data collected from real sources often contains errors, missing values, and inconsistencies.

2. Step 2: Recognize the need for cleaning and organizing

   Before training, data must be cleaned and formatted properly to ensure the model learns correctly.

3. Final Answer:

   Because raw data is often messy and needs cleaning before use -> Option C

4. Quick Check:

   Messy data = More preparation time [OK]

Hint: Messy raw data means more cleaning time [OK]

Common Mistakes:

• Confusing model training time with data cleaning time
• Thinking deployment takes the most time
• Assuming testing is the longest step

2. Which of the following is the correct Python code snippet to check for missing values in a pandas DataFrame named df?

easy

A. df.isnull().sum()

B. df.missing_values()

C. df.check_na()

D. df.nulls_count()

Solution

1. Step 1: Recall pandas method for missing values

   The method isnull() returns a DataFrame of booleans indicating missing values.

2. Step 2: Summing missing values per column

   Using sum() on the boolean DataFrame counts missing values per column.

3. Final Answer:

   df.isnull().sum() -> Option A

4. Quick Check:

   pandas isnull + sum = count missing [OK]

Hint: Use df.isnull().sum() to count missing data [OK]

Common Mistakes:

• Using non-existent pandas methods
• Confusing isnull() with isna() (both work but only isnull() shown)
• Trying to call methods that don't exist on DataFrame

3. Consider this Python code for data preparation:

import pandas as pd

data = {'age': [25, None, 30], 'salary': [50000, 60000, None]}
df = pd.DataFrame(data)
df_filled = df.fillna(df.mean())
print(df_filled)

What will be the output?

medium

A. DataFrame with missing values replaced by zeros

B. DataFrame unchanged with null values still present

C. Error because df.mean() cannot be used with fillna

D. DataFrame with missing values replaced by column means

Solution

1. Step 1: Understand df.mean()

   Calculates mean of each numeric column ignoring null (NaN) values.

2. Step 2: fillna with means

   fillna replaces missing values with the corresponding column mean.

3. Final Answer:

   DataFrame with missing values replaced by column means -> Option D

4. Quick Check:

   fillna(df.mean()) fills missing with mean [OK]

Hint: fillna(df.mean()) replaces missing with average [OK]

Common Mistakes:

• Thinking fillna(df.mean()) causes error
• Assuming missing values remain unchanged
• Believing missing values become zero

4. You have this code snippet to drop rows with missing values:

import pandas as pd
data = {'height': [170, None, 160], 'weight': [70, 65, None]}
df = pd.DataFrame(data)
df_clean = df.dropna(axis=1)
print(df_clean)

What is wrong with this code if the goal is to remove rows with missing data?

medium

A. dropna() requires a threshold parameter to work

B. dropna(axis=1) removes columns, not rows

C. dropna() cannot be used on DataFrames

D. dropna(axis=1) removes rows, not columns

Solution

1. Step 1: Understand dropna axis parameter

   axis=0 drops rows; axis=1 drops columns with missing values.

2. Step 2: Identify the mistake

   Using axis=1 removes columns, but the goal is to remove rows.

3. Final Answer:

   dropna(axis=1) removes columns, not rows -> Option B

4. Quick Check:

   axis=1 means columns, not rows [OK]

Hint: Use dropna(axis=0) to remove rows with missing data [OK]

Common Mistakes:

• Confusing axis=1 with rows
• Thinking dropna always removes rows by default
• Believing dropna needs threshold parameter

5. You have a dataset with many missing values and inconsistent formats. Which two data preparation steps combined will most improve model quality and reduce errors?

hard

A. Filling missing values with mean and converting all text to lowercase

B. Randomly shuffling data and dropping duplicate rows

C. Ignoring missing values and training the model directly

D. Removing all rows with any missing data and normalizing numeric columns

Solution

1. Step 1: Consider filling missing values

   Replacing missing data with mean keeps data size and reduces bias from dropping rows.

2. Step 2: Standardize text format

   Converting text to lowercase fixes inconsistencies and helps model learn better.

3. Final Answer:

   Filling missing values with mean and converting all text to lowercase -> Option A

4. Quick Check:

   Fill missing + text clean = better data [OK]

Hint: Fill missing + clean text improves data quality [OK]

Common Mistakes:

• Dropping all rows loses too much data
• Ignoring missing values causes errors
• Shuffling or dropping duplicates alone won't fix missing data