Exploratory data analysis in ML Python - ML Experiment: Train & Evaluate

Experiment - Exploratory data analysis

Problem:You have a dataset with information about houses, including features like size, number of rooms, and price. You want to understand the data better before building any model.

Current Metrics:No model yet; just raw data with unknown patterns and possible missing values.

Issue:You do not know the distribution of features, presence of missing values, or relationships between variables, which can lead to poor model performance later.

Your Task

Perform exploratory data analysis (EDA) to summarize the dataset, identify missing values, visualize feature distributions, and find correlations between features.

Use only Python with pandas, matplotlib, and seaborn libraries.

Do not build or train any machine learning model yet.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Load dataset
url = 'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/diamonds.csv'
df = pd.read_csv(url)

# 1. Check basic info and missing values
print(df.info())
print('Missing values per column:')
print(df.isnull().sum())

# 2. Summary statistics
print(df.describe())

# 3. Visualize distributions of numeric features
numeric_features = ['carat', 'depth', 'table', 'price', 'x', 'y', 'z']
for feature in numeric_features:
    plt.figure(figsize=(6, 4))
    sns.histplot(df[feature], kde=True)
    plt.title(f'Distribution of {feature}')
    plt.show()

# 4. Boxplots to detect outliers
for feature in numeric_features:
    plt.figure(figsize=(6, 4))
    sns.boxplot(x=df[feature])
    plt.title(f'Boxplot of {feature}')
    plt.show()

# 5. Correlation heatmap
plt.figure(figsize=(8, 6))
corr = df[numeric_features].corr()
sns.heatmap(corr, annot=True, cmap='coolwarm')
plt.title('Correlation between numeric features')
plt.show()

# 6. Scatter plot example: carat vs price
plt.figure(figsize=(6, 4))
sns.scatterplot(x='carat', y='price', data=df)
plt.title('Scatter plot of Carat vs Price')
plt.show()

Loaded dataset and checked for missing values and data types.

Generated summary statistics to understand feature ranges and central tendencies.

Plotted histograms and boxplots to visualize distributions and detect outliers.

Created a correlation heatmap to identify relationships between numeric features.

Visualized scatter plot to see direct relationship between carat and price.

Results Interpretation

Before EDA: Dataset was raw with unknown quality and patterns.

After EDA: Data quality confirmed good (no missing values), distributions understood, outliers detected, and strong feature relationships identified.

Exploratory data analysis helps you understand your data's quality and patterns. This knowledge guides better decisions when building machine learning models.

Bonus Experiment

Try performing EDA on a different dataset, such as the Titanic dataset, to explore missing values and feature relationships.

💡 Hint

Use pandas to load the Titanic dataset from seaborn (sns.load_dataset('titanic')) and repeat similar EDA steps.

Practice

1. What is the main goal of exploratory data analysis (EDA) in machine learning?

easy

A. To train the machine learning model

B. To understand the main features and quality of the data

C. To deploy the model to production

D. To write the final report without looking at data

Solution

1. Step 1: Understand the purpose of EDA

   EDA is used to explore and understand the data before modeling.

2. Step 2: Differentiate EDA from other steps

   Training and deployment happen after EDA, so they are not the main goal of EDA.

3. Final Answer:

   To understand the main features and quality of the data -> Option B

4. Quick Check:

   EDA = Understand data [OK]

Hint: EDA means exploring data, not modeling or deploying [OK]

Common Mistakes:

• Confusing EDA with model training
• Thinking EDA is for deployment
• Skipping data understanding step

2. Which Python library is commonly used for basic exploratory data analysis tasks like viewing data samples and summary statistics?

easy

A. matplotlib

B. tensorflow

C. pandas

D. scikit-learn

Solution

1. Step 1: Identify libraries for EDA

   Pandas provides functions like head() and describe() for data samples and summary stats.

2. Step 2: Exclude other libraries

   Matplotlib is for plotting, scikit-learn for modeling, tensorflow for deep learning, not basic EDA.

3. Final Answer:

   pandas -> Option C

4. Quick Check:

   pandas = data samples and stats [OK]

Hint: Use pandas for quick data overview and stats [OK]

Common Mistakes:

• Using matplotlib for data samples
• Confusing modeling libraries with EDA
• Trying tensorflow for simple stats

3. Given the following Python code using pandas, what will print(df.describe()) output?

import pandas as pd
data = {'age': [25, 30, 22, 40, 28], 'score': [88, 92, 85, 95, 90]}
df = pd.DataFrame(data)
print(df.describe())

medium

A. Summary statistics including count, mean, std, min, max for 'age' and 'score'

B. The first 5 rows of the dataframe

C. A plot showing distribution of 'age' and 'score'

D. An error because describe() needs parameters

Solution

1. Step 1: Understand what describe() does

   It returns summary statistics like count, mean, std, min, max for numeric columns.

2. Step 2: Check the code output

   Since 'age' and 'score' are numeric, describe() will show stats for both columns.

3. Final Answer:

   Summary statistics including count, mean, std, min, max for 'age' and 'score' -> Option A

4. Quick Check:

   describe() = summary stats [OK]

Hint: describe() shows stats, head() shows rows [OK]

Common Mistakes:

• Thinking describe() shows data rows
• Expecting a plot from describe()
• Assuming describe() needs arguments

4. You run this code but get an error:

import pandas as pd
data = {'age': [25, 30, None, 40], 'score': [88, 92, 85, 95]}
df = pd.DataFrame(data)
print(df.mean())

What is the likely cause and fix?

medium

A. Error due to None value; fix by using df.mean(skipna=True)

B. Error due to missing import; fix by importing numpy

C. Error due to wrong data type; fix by converting all to strings

D. No error; code runs fine as is

Solution

1. Step 1: Check how pandas handles None in numeric columns

   Pandas treats None as NaN and by default mean() skips NaN values.

2. Step 2: Verify if code runs without error

   The code runs fine and calculates mean ignoring None.

3. Final Answer:

   No error; code runs fine as is -> Option D

4. Quick Check:

   pandas mean() skips None by default [OK]

Hint: pandas mean() ignores None/NaN by default [OK]

Common Mistakes:

• Assuming None causes error in mean()
• Thinking numpy import is needed here
• Trying to convert numbers to strings

5. You have a dataset with columns age, income, and purchased (Yes/No). Which EDA step helps you find if income affects purchased?

hard

A. Plotting boxplots of income grouped by purchased categories

B. Calculating mean age of all customers

C. Checking for missing values in age column

D. Sorting data by income in ascending order

Solution

1. Step 1: Identify relationship to explore

   We want to see if income differs between those who purchased and those who did not.

2. Step 2: Choose appropriate EDA method

   Boxplots grouped by purchased categories show income distribution differences clearly.

3. Final Answer:

   Plotting boxplots of income grouped by purchased categories -> Option A

4. Quick Check:

   Boxplots show group differences [OK]

Hint: Use grouped boxplots to compare numeric by category [OK]

Common Mistakes:

• Ignoring grouping by purchased
• Checking unrelated columns
• Just sorting without visualization