Exploratory data analysis in ML Python

Introduction

Exploratory data analysis helps us understand data by looking at it carefully. It shows patterns, problems, and important details before building models.

When you get a new dataset and want to know what it contains.

Before cleaning data to find missing or strange values.

To find relationships between different data points.

To decide which features are important for a model.

To check if data matches your expectations or needs fixing.

Syntax

ML Python

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Load data
data = pd.read_csv('data.csv')

# Show first rows
print(data.head())

# Summary statistics
print(data.describe())

# Check missing values
print(data.isnull().sum())

# Visualize data
sns.histplot(data['column'])
plt.show()

Use head() to see first few rows quickly.

describe() gives basic stats like mean and quartiles.

Examples

Shows the first 10 rows of the dataset to get a quick look.

ML Python

data.head(10)

Gives summary statistics for numeric columns like mean, min, max.

ML Python

data.describe()

Counts how many missing values are in each column.

ML Python

data.isnull().sum()

Draws a box plot to see data spread and outliers.

ML Python

sns.boxplot(x=data['column'])
plt.show()

Sample Model

This code creates a small dataset, shows its first rows, summary stats, missing values, and draws a histogram of ages.

ML Python

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Create a small sample dataset
data = pd.DataFrame({
    'Age': [25, 30, 22, 40, 28, None, 35],
    'Salary': [50000, 60000, 45000, 80000, 52000, 58000, None],
    'Department': ['Sales', 'IT', 'Sales', 'HR', 'IT', 'HR', 'Sales']
})

# Show first rows
print('First rows:')
print(data.head())

# Summary statistics
print('\nSummary statistics:')
print(data.describe())

# Missing values
print('\nMissing values:')
print(data.isnull().sum())

# Visualize Age distribution
sns.histplot(data['Age'], kde=True)
plt.title('Age Distribution')
plt.show()

Important Notes

Always check for missing or strange values early.

Visualizations help see patterns that numbers alone may hide.

Use simple stats first before complex analysis.

Summary

Exploratory data analysis helps you understand your data clearly.

Look at data samples, summary stats, and missing values first.

Use charts to find patterns and problems easily.

Practice

1. What is the main goal of exploratory data analysis (EDA) in machine learning?

easy

A. To train the machine learning model

B. To understand the main features and quality of the data

C. To deploy the model to production

D. To write the final report without looking at data

Solution

1. Step 1: Understand the purpose of EDA

   EDA is used to explore and understand the data before modeling.

2. Step 2: Differentiate EDA from other steps

   Training and deployment happen after EDA, so they are not the main goal of EDA.

3. Final Answer:

   To understand the main features and quality of the data -> Option B

4. Quick Check:

   EDA = Understand data [OK]

Hint: EDA means exploring data, not modeling or deploying [OK]

Common Mistakes:

• Confusing EDA with model training
• Thinking EDA is for deployment
• Skipping data understanding step

2. Which Python library is commonly used for basic exploratory data analysis tasks like viewing data samples and summary statistics?

easy

A. matplotlib

B. tensorflow

C. pandas

D. scikit-learn

Solution

1. Step 1: Identify libraries for EDA

   Pandas provides functions like head() and describe() for data samples and summary stats.

2. Step 2: Exclude other libraries

   Matplotlib is for plotting, scikit-learn for modeling, tensorflow for deep learning, not basic EDA.

3. Final Answer:

   pandas -> Option C

4. Quick Check:

   pandas = data samples and stats [OK]

Hint: Use pandas for quick data overview and stats [OK]

Common Mistakes:

• Using matplotlib for data samples
• Confusing modeling libraries with EDA
• Trying tensorflow for simple stats

3. Given the following Python code using pandas, what will print(df.describe()) output?

import pandas as pd
data = {'age': [25, 30, 22, 40, 28], 'score': [88, 92, 85, 95, 90]}
df = pd.DataFrame(data)
print(df.describe())

medium

A. Summary statistics including count, mean, std, min, max for 'age' and 'score'

B. The first 5 rows of the dataframe

C. A plot showing distribution of 'age' and 'score'

D. An error because describe() needs parameters

Solution

1. Step 1: Understand what describe() does

   It returns summary statistics like count, mean, std, min, max for numeric columns.

2. Step 2: Check the code output

   Since 'age' and 'score' are numeric, describe() will show stats for both columns.

3. Final Answer:

   Summary statistics including count, mean, std, min, max for 'age' and 'score' -> Option A

4. Quick Check:

   describe() = summary stats [OK]

Hint: describe() shows stats, head() shows rows [OK]

Common Mistakes:

• Thinking describe() shows data rows
• Expecting a plot from describe()
• Assuming describe() needs arguments

4. You run this code but get an error:

import pandas as pd
data = {'age': [25, 30, None, 40], 'score': [88, 92, 85, 95]}
df = pd.DataFrame(data)
print(df.mean())

What is the likely cause and fix?

medium

A. Error due to None value; fix by using df.mean(skipna=True)

B. Error due to missing import; fix by importing numpy

C. Error due to wrong data type; fix by converting all to strings

D. No error; code runs fine as is

Solution

1. Step 1: Check how pandas handles None in numeric columns

   Pandas treats None as NaN and by default mean() skips NaN values.

2. Step 2: Verify if code runs without error

   The code runs fine and calculates mean ignoring None.

3. Final Answer:

   No error; code runs fine as is -> Option D

4. Quick Check:

   pandas mean() skips None by default [OK]

Hint: pandas mean() ignores None/NaN by default [OK]

Common Mistakes:

• Assuming None causes error in mean()
• Thinking numpy import is needed here
• Trying to convert numbers to strings

5. You have a dataset with columns age, income, and purchased (Yes/No). Which EDA step helps you find if income affects purchased?

hard

A. Plotting boxplots of income grouped by purchased categories

B. Calculating mean age of all customers

C. Checking for missing values in age column

D. Sorting data by income in ascending order

Solution

1. Step 1: Identify relationship to explore

   We want to see if income differs between those who purchased and those who did not.

2. Step 2: Choose appropriate EDA method

   Boxplots grouped by purchased categories show income distribution differences clearly.

3. Final Answer:

   Plotting boxplots of income grouped by purchased categories -> Option A

4. Quick Check:

   Boxplots show group differences [OK]

Hint: Use grouped boxplots to compare numeric by category [OK]

Common Mistakes:

• Ignoring grouping by purchased
• Checking unrelated columns
• Just sorting without visualization