Exploratory data analysis workflow in Pandas - Step-by-Step Execution

Concept Flow - Exploratory data analysis workflow

Load Data

↓

Inspect Data

↓

Clean Data

↓

Summarize Data

↓

Visualize Data

↓

Draw Insights

This flow shows the main steps in exploring data: loading, inspecting, cleaning, summarizing, visualizing, and drawing insights.

Execution Sample

Pandas

import pandas as pd

df = pd.read_csv('data.csv')
print(df.head())
print(df.describe())

This code loads data from a CSV file, shows the first rows, and prints summary statistics.

Execution Table

Step	Action	Code/Method	Output Description
1	Load Data	pd.read_csv('data.csv')	DataFrame with all rows and columns from CSV
2	Inspect Data	df.head()	First 5 rows of the DataFrame shown
3	Inspect Data	df.info()	Summary of columns, data types, and non-null counts
4	Clean Data	df = df.dropna()	DataFrame with rows containing missing values removed
5	Summarize Data	df.describe()	Statistics like mean, std, min, max for numeric columns
6	Visualize Data	df['column'].hist()	Histogram plot showing distribution of a column
7	Draw Insights	Look at summaries and plots	Understand patterns, outliers, and trends
Exit	End of workflow		All main EDA steps completed

💡 All main exploratory data analysis steps have been executed.

Variable Tracker

Variable	Start	After Load	After Clean	After Summarize
df	None	DataFrame with raw data	DataFrame with missing rows removed	Summary statistics DataFrame (describe output)

Key Moments - 3 Insights

Why do we use df.head() instead of printing the whole DataFrame?

df.head() shows only the first few rows, making it easier to quickly see the data structure without overwhelming output, as shown in execution_table step 2.

What does df.describe() tell us about the data?

df.describe() gives key statistics like mean and min for numeric columns, helping us understand data distribution, as seen in execution_table step 5.

Why is cleaning data important before analysis?

Cleaning removes missing or incorrect data that can mislead results, shown in execution_table step 4 where rows with missing values are dropped.

Visual Quiz - 3 Questions

Look at the execution table, what does df.head() show at step 2?

A Data types of each column

B First 5 rows of the DataFrame

C Summary statistics of the DataFrame

D All rows of the DataFrame

💡 Hint

Refer to execution_table row with Step 2 under 'Output Description'

At which step do we remove rows with missing values?

A Step 5

B Step 3

C Step 4

D Step 6

💡 Hint

Check execution_table row where Action is 'Clean Data'

If we skip cleaning data, what might happen to the summary statistics?

A They might be misleading due to missing values

B They will be more accurate

C They will not change

D They will show only categorical data

💡 Hint

Consider the purpose of cleaning in execution_table step 4 before summarizing in step 5

Concept Snapshot

Exploratory Data Analysis (EDA) Workflow:
1. Load data into a DataFrame
2. Inspect data with head() and info()
3. Clean data by handling missing values
4. Summarize data with describe()
5. Visualize data distributions
6. Draw insights from patterns and outliers

Full Transcript

Exploratory data analysis is a step-by-step process to understand data. First, we load data into a DataFrame using pandas. Then, we inspect the data by looking at the first few rows and checking data types and missing values. Next, we clean the data by removing or fixing missing values. After cleaning, we summarize the data using statistics like mean and standard deviation. We also visualize data to see distributions and patterns. Finally, we draw insights to guide further analysis or decisions. Each step builds on the previous to help us understand the data clearly and avoid mistakes.

Practice

1. Which step is usually done first in an exploratory data analysis workflow?

easy

A. Viewing the first few rows of the dataset

B. Creating complex machine learning models

C. Removing all missing values immediately

D. Generating final reports without checking data

Solution

1. Step 1: Understand the initial data

   Exploratory data analysis starts by looking at the first few rows to get a feel for the data structure and values.

2. Step 2: Avoid premature actions

   Jumping to modeling or removing missing data without understanding can cause mistakes.

3. Final Answer:

   Viewing the first few rows of the dataset -> Option A

4. Quick Check:

   First step = View data sample [OK]

Hint: Always start by looking at data samples [OK]

Common Mistakes:

• Starting with modeling before understanding data
• Removing missing values without checking their impact
• Ignoring data overview and jumping to conclusions

2. Which pandas command correctly shows summary statistics of numeric columns in a DataFrame df?

easy

A. df.head()

B. df.info()

C. df.describe()

D. df.sample()

Solution

1. Step 1: Identify summary statistics function

   df.describe() returns count, mean, std, min, max, and quartiles for numeric columns.

2. Step 2: Differentiate other commands

   df.head() shows first rows, df.info() shows data types, and df.sample() shows random rows.

3. Final Answer:

   df.describe() -> Option C

4. Quick Check:

   Summary stats = df.describe() [OK]

Hint: Use df.describe() for quick numeric summaries [OK]

Common Mistakes:

• Using df.head() to get statistics
• Confusing df.info() with statistical summary
• Expecting df.sample() to summarize data

3. Given the DataFrame df below, what is the output of df.isnull().sum()?

import pandas as pd
data = {'A': [1, 2, None], 'B': [None, 2, 3], 'C': [1, 2, 3]}
df = pd.DataFrame(data)

medium

A. {'A': 1, 'B': 1, 'C': 0}

B. {'A': 0, 'B': 0, 'C': 0}

C. {'A': 1, 'B': 2, 'C': 1}

D. {'A': 2, 'B': 1, 'C': 0}

Solution

1. Step 1: Check missing values per column

   Column 'A' has one None, 'B' has one None, 'C' has none.

2. Step 2: Use isnull().sum() to count missing

   This method counts missing values per column, so output is {'A': 1, 'B': 1, 'C': 0}.

3. Final Answer:

   {'A': 1, 'B': 1, 'C': 0} -> Option A

4. Quick Check:

   Missing counts = {'A':1,'B':1,'C':0} [OK]

Hint: Count missing with df.isnull().sum() [OK]

Common Mistakes:

• Counting total missing instead of per column
• Confusing None with zero or empty string
• Misreading which columns have missing values

4. You run df.info() but get an error: AttributeError: 'DataFrame' object has no attribute 'infoo'. What is the likely cause?

medium

A. The DataFrame is empty

B. You typed the method name incorrectly

C. The pandas library is not imported

D. The DataFrame has no columns

Solution

1. Step 1: Check the error message

   The error says 'infoo' attribute not found, which suggests a typo in method name.

2. Step 2: Confirm correct method name

   The correct method is df.info() with one 'o'.

3. Final Answer:

   You typed the method name incorrectly -> Option B

4. Quick Check:

   Typo in method name = error [OK]

Hint: Check spelling of method names carefully [OK]

Common Mistakes:

• Assuming empty DataFrame causes this error
• Forgetting to import pandas
• Thinking DataFrame must have columns to use info()

5. You have a DataFrame df with columns ['Age', 'Salary', 'Department']. You want to find the average salary per department but only for employees older than 30. Which pandas workflow is correct?

hard

A. df.groupby('Department')['Salary'].mean()

B. df['Salary'].mean().groupby('Department')[df['Age'] > 30]

C. df.groupby('Department').mean()['Salary'][df['Age'] > 30]

D. df[df['Age'] > 30].groupby('Department')['Salary'].mean()

Solution

1. Step 1: Filter rows where Age > 30

   Use df[df['Age'] > 30] to keep only employees older than 30.

2. Step 2: Group filtered data by Department and calculate mean Salary

   Apply .groupby('Department')['Salary'].mean() on filtered data to get average salary per department.

3. Final Answer:

   df[df['Age'] > 30].groupby('Department')['Salary'].mean() -> Option D

4. Quick Check:

   Filter then group and mean = df[df['Age'] > 30].groupby('Department')['Salary'].mean() [OK]

Hint: Filter first, then group and aggregate [OK]

Common Mistakes:

• Grouping before filtering data
• Trying to filter after aggregation
• Using mean() without grouping