Exploratory data analysis in ML Python - Model Pipeline Trace

Model Pipeline - Exploratory data analysis

Exploratory Data Analysis (EDA) helps us understand the data before building a model. It shows data shapes, missing values, and basic statistics.

Data Flow - 4 Stages

1Load Data

CSV file with 1000 rows x 5 columns→Read CSV file into a DataFrame→1000 rows x 5 columns

First 3 rows: {'Age': 25, 'Salary': 50000, 'Gender': 'M', 'Purchased': 'No', 'Score': 70}

↓

2Check Missing Values

1000 rows x 5 columns→Count missing values per column→5 columns with missing counts

{'Age': 0, 'Salary': 10, 'Gender': 0, 'Purchased': 0, 'Score': 5}

↓

3Summary Statistics

1000 rows x 5 columns→Calculate mean, median, min, max for numeric columns→Summary table with 5 columns

{'Age': {'mean': 35, 'min': 18, 'max': 65}, 'Salary': {'mean': 55000, 'min': 30000, 'max': 120000}}

↓

4Visualize Data

Summary statistics and raw data→Create histograms and boxplots→Plots showing data distribution

Histogram of Age shows most people between 25 and 45

Training Trace - Epoch by Epoch

EDA step - no training loss to show

Epoch	Loss ↓	Accuracy ↑	Observation
1	N/A	N/A	EDA does not involve training, so no loss or accuracy.

Prediction Trace - 2 Layers

Layer 1: Data Inspection

Layer 2: Distribution Visualization

Model Quiz - 3 Questions

What is the main goal of Exploratory Data Analysis?

A Understand data patterns and quality

B Train a machine learning model

C Deploy the model to production

D Optimize model hyperparameters

Key Insight

Exploratory Data Analysis is a crucial first step to understand data quality and patterns before building any machine learning model. It does not involve training but prepares us to make better modeling decisions.

Practice

1. What is the main goal of exploratory data analysis (EDA) in machine learning?

easy

A. To train the machine learning model

B. To understand the main features and quality of the data

C. To deploy the model to production

D. To write the final report without looking at data

Solution

1. Step 1: Understand the purpose of EDA

   EDA is used to explore and understand the data before modeling.

2. Step 2: Differentiate EDA from other steps

   Training and deployment happen after EDA, so they are not the main goal of EDA.

3. Final Answer:

   To understand the main features and quality of the data -> Option B

4. Quick Check:

   EDA = Understand data [OK]

Hint: EDA means exploring data, not modeling or deploying [OK]

Common Mistakes:

• Confusing EDA with model training
• Thinking EDA is for deployment
• Skipping data understanding step

2. Which Python library is commonly used for basic exploratory data analysis tasks like viewing data samples and summary statistics?

easy

A. matplotlib

B. tensorflow

C. pandas

D. scikit-learn

Solution

1. Step 1: Identify libraries for EDA

   Pandas provides functions like head() and describe() for data samples and summary stats.

2. Step 2: Exclude other libraries

   Matplotlib is for plotting, scikit-learn for modeling, tensorflow for deep learning, not basic EDA.

3. Final Answer:

   pandas -> Option C

4. Quick Check:

   pandas = data samples and stats [OK]

Hint: Use pandas for quick data overview and stats [OK]

Common Mistakes:

• Using matplotlib for data samples
• Confusing modeling libraries with EDA
• Trying tensorflow for simple stats

3. Given the following Python code using pandas, what will print(df.describe()) output?

import pandas as pd
data = {'age': [25, 30, 22, 40, 28], 'score': [88, 92, 85, 95, 90]}
df = pd.DataFrame(data)
print(df.describe())

medium

A. Summary statistics including count, mean, std, min, max for 'age' and 'score'

B. The first 5 rows of the dataframe

C. A plot showing distribution of 'age' and 'score'

D. An error because describe() needs parameters

Solution

1. Step 1: Understand what describe() does

   It returns summary statistics like count, mean, std, min, max for numeric columns.

2. Step 2: Check the code output

   Since 'age' and 'score' are numeric, describe() will show stats for both columns.

3. Final Answer:

   Summary statistics including count, mean, std, min, max for 'age' and 'score' -> Option A

4. Quick Check:

   describe() = summary stats [OK]

Hint: describe() shows stats, head() shows rows [OK]

Common Mistakes:

• Thinking describe() shows data rows
• Expecting a plot from describe()
• Assuming describe() needs arguments

4. You run this code but get an error:

import pandas as pd
data = {'age': [25, 30, None, 40], 'score': [88, 92, 85, 95]}
df = pd.DataFrame(data)
print(df.mean())

What is the likely cause and fix?

medium

A. Error due to None value; fix by using df.mean(skipna=True)

B. Error due to missing import; fix by importing numpy

C. Error due to wrong data type; fix by converting all to strings

D. No error; code runs fine as is

Solution

1. Step 1: Check how pandas handles None in numeric columns

   Pandas treats None as NaN and by default mean() skips NaN values.

2. Step 2: Verify if code runs without error

   The code runs fine and calculates mean ignoring None.

3. Final Answer:

   No error; code runs fine as is -> Option D

4. Quick Check:

   pandas mean() skips None by default [OK]

Hint: pandas mean() ignores None/NaN by default [OK]

Common Mistakes:

• Assuming None causes error in mean()
• Thinking numpy import is needed here
• Trying to convert numbers to strings

5. You have a dataset with columns age, income, and purchased (Yes/No). Which EDA step helps you find if income affects purchased?

hard

A. Plotting boxplots of income grouped by purchased categories

B. Calculating mean age of all customers

C. Checking for missing values in age column

D. Sorting data by income in ascending order

Solution

1. Step 1: Identify relationship to explore

   We want to see if income differs between those who purchased and those who did not.

2. Step 2: Choose appropriate EDA method

   Boxplots grouped by purchased categories show income distribution differences clearly.

3. Final Answer:

   Plotting boxplots of income grouped by purchased categories -> Option A

4. Quick Check:

   Boxplots show group differences [OK]

Hint: Use grouped boxplots to compare numeric by category [OK]

Common Mistakes:

• Ignoring grouping by purchased
• Checking unrelated columns
• Just sorting without visualization