Why data preparation consumes most ML time in ML Python

Introduction

Data preparation takes most of the time because real-world data is messy and needs cleaning before it can be used. This step makes sure the machine learning model learns from good, useful information.

When you get data from different sources and need to combine it.

When your data has missing or wrong values that need fixing.

When you want to change data into a format the model can understand.

When you need to remove noise or errors from the data.

When you want to select only the important parts of the data for training.

Syntax

ML Python

No fixed code syntax because data preparation involves many steps like cleaning, transforming, and selecting data.

Data preparation is not a single command but a set of tasks done before training.

Common tools include pandas for cleaning and sklearn for transforming data.

Examples

Example of removing missing data using pandas.

ML Python

import pandas as pd

df = pd.read_csv('data.csv')
df = df.dropna()  # Remove rows with missing values

Example of scaling data to have mean 0 and variance 1.

ML Python

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
data_scaled = scaler.fit_transform(data)

Example of converting text categories into numbers.

ML Python

df['category'] = df['category'].astype('category').cat.codes

Sample Model

This example shows cleaning missing data, converting categories, training a simple model, and checking accuracy.

ML Python

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Sample data with missing values and categories
data = {'age': [25, 30, None, 22, 40],
        'income': [50000, 60000, 55000, None, 65000],
        'gender': ['M', 'F', 'F', 'M', 'F'],
        'purchased': [0, 1, 0, 1, 1]}

df = pd.DataFrame(data)

# Data preparation
# 1. Remove rows with missing values
clean_df = df.dropna()

# 2. Convert gender to numeric
clean_df['gender'] = clean_df['gender'].astype('category').cat.codes

# 3. Split features and target
X = clean_df[['age', 'income', 'gender']]
y = clean_df['purchased']

# 4. Split into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

# 5. Train model
model = LogisticRegression()
model.fit(X_train, y_train)

# 6. Predict and evaluate
predictions = model.predict(X_test)
accuracy = accuracy_score(y_test, predictions)

print(f"Predictions: {predictions}")
print(f"Accuracy: {accuracy:.2f}")

Important Notes

Data preparation often takes 70-80% of the total project time.

Good data preparation improves model accuracy and reliability.

Skipping data cleaning can cause wrong or poor model results.

Summary

Data preparation cleans and organizes raw data for machine learning.

This step takes most time because real data is often messy and incomplete.

Proper preparation leads to better and more trustworthy models.

Practice

1. Why does data preparation usually take the most time in a machine learning project?

easy

A. Because testing the model takes a long time

B. Because training the model requires many calculations

C. Because raw data is often messy and needs cleaning before use

D. Because deploying the model to production is complex

Solution

1. Step 1: Understand the nature of raw data

   Raw data collected from real sources often contains errors, missing values, and inconsistencies.

2. Step 2: Recognize the need for cleaning and organizing

   Before training, data must be cleaned and formatted properly to ensure the model learns correctly.

3. Final Answer:

   Because raw data is often messy and needs cleaning before use -> Option C

4. Quick Check:

   Messy data = More preparation time [OK]

Hint: Messy raw data means more cleaning time [OK]

Common Mistakes:

• Confusing model training time with data cleaning time
• Thinking deployment takes the most time
• Assuming testing is the longest step

2. Which of the following is the correct Python code snippet to check for missing values in a pandas DataFrame named df?

easy

A. df.isnull().sum()

B. df.missing_values()

C. df.check_na()

D. df.nulls_count()

Solution

1. Step 1: Recall pandas method for missing values

   The method isnull() returns a DataFrame of booleans indicating missing values.

2. Step 2: Summing missing values per column

   Using sum() on the boolean DataFrame counts missing values per column.

3. Final Answer:

   df.isnull().sum() -> Option A

4. Quick Check:

   pandas isnull + sum = count missing [OK]

Hint: Use df.isnull().sum() to count missing data [OK]

Common Mistakes:

• Using non-existent pandas methods
• Confusing isnull() with isna() (both work but only isnull() shown)
• Trying to call methods that don't exist on DataFrame

3. Consider this Python code for data preparation:

import pandas as pd

data = {'age': [25, None, 30], 'salary': [50000, 60000, None]}
df = pd.DataFrame(data)
df_filled = df.fillna(df.mean())
print(df_filled)

What will be the output?

medium

A. DataFrame with missing values replaced by zeros

B. DataFrame unchanged with null values still present

C. Error because df.mean() cannot be used with fillna

D. DataFrame with missing values replaced by column means

Solution

1. Step 1: Understand df.mean()

   Calculates mean of each numeric column ignoring null (NaN) values.

2. Step 2: fillna with means

   fillna replaces missing values with the corresponding column mean.

3. Final Answer:

   DataFrame with missing values replaced by column means -> Option D

4. Quick Check:

   fillna(df.mean()) fills missing with mean [OK]

Hint: fillna(df.mean()) replaces missing with average [OK]

Common Mistakes:

• Thinking fillna(df.mean()) causes error
• Assuming missing values remain unchanged
• Believing missing values become zero

4. You have this code snippet to drop rows with missing values:

import pandas as pd
data = {'height': [170, None, 160], 'weight': [70, 65, None]}
df = pd.DataFrame(data)
df_clean = df.dropna(axis=1)
print(df_clean)

What is wrong with this code if the goal is to remove rows with missing data?

medium

A. dropna() requires a threshold parameter to work

B. dropna(axis=1) removes columns, not rows

C. dropna() cannot be used on DataFrames

D. dropna(axis=1) removes rows, not columns

Solution

1. Step 1: Understand dropna axis parameter

   axis=0 drops rows; axis=1 drops columns with missing values.

2. Step 2: Identify the mistake

   Using axis=1 removes columns, but the goal is to remove rows.

3. Final Answer:

   dropna(axis=1) removes columns, not rows -> Option B

4. Quick Check:

   axis=1 means columns, not rows [OK]

Hint: Use dropna(axis=0) to remove rows with missing data [OK]

Common Mistakes:

• Confusing axis=1 with rows
• Thinking dropna always removes rows by default
• Believing dropna needs threshold parameter

5. You have a dataset with many missing values and inconsistent formats. Which two data preparation steps combined will most improve model quality and reduce errors?

hard

A. Filling missing values with mean and converting all text to lowercase

B. Randomly shuffling data and dropping duplicate rows

C. Ignoring missing values and training the model directly

D. Removing all rows with any missing data and normalizing numeric columns

Solution

1. Step 1: Consider filling missing values

   Replacing missing data with mean keeps data size and reduces bias from dropping rows.

2. Step 2: Standardize text format

   Converting text to lowercase fixes inconsistencies and helps model learn better.

3. Final Answer:

   Filling missing values with mean and converting all text to lowercase -> Option A

4. Quick Check:

   Fill missing + text clean = better data [OK]

Hint: Fill missing + clean text improves data quality [OK]

Common Mistakes:

• Dropping all rows loses too much data
• Ignoring missing values causes errors
• Shuffling or dropping duplicates alone won't fix missing data