Working with large datasets strategies in Pandas - Mini Project: Build & Apply

Working with Large Datasets Strategies

📖 Scenario: You work as a data analyst for a retail company. You have a large dataset of sales transactions. The dataset is too big to load all at once, so you need to work with smaller parts and apply filters to manage memory and speed.

🎯 Goal: Learn how to load a large dataset in chunks, filter data based on a condition, and combine the filtered results into a smaller dataset for analysis.

📋 What You'll Learn

Use pandas to read CSV data in chunks

Filter data based on a sales amount threshold

Combine filtered chunks into a single DataFrame

Print the final filtered DataFrame

💡 Why This Matters

🌍 Real World

Working with large datasets is common in data science. Loading data in chunks helps manage memory and speeds up processing.

💼 Career

Data analysts and scientists often need to handle big data files efficiently. Knowing how to filter and process data in parts is a valuable skill.

1

Create a sample large dataset CSV file

Create a CSV file named sales_data.csv with these exact columns and rows:
TransactionID,Product,Quantity,Price
Rows:
1,Apple,10,0.5
2,Banana,5,0.3
3,Orange,8,0.7
4,Apple,3,0.5
5,Banana,7,0.3
6,Orange,2,0.7
7,Apple,12,0.5
8,Banana,1,0.3
9,Orange,4,0.7
10,Apple,6,0.5

Pandas

# Create the CSV file sales_data.csv with the exact data
# Your code here

Need a hint?

Use pandas.DataFrame.to_csv to save the data to a CSV file named sales_data.csv.

2

Set the sales amount threshold

Create a variable called sales_threshold and set it to 5. This will be used to filter transactions with total sales above this value.

Pandas

import pandas as pd

# Set the sales amount threshold
# Your code here

Need a hint?

Just create a variable named sales_threshold and assign the value 5.

3

Read the CSV in chunks and filter by sales amount

Use pandas.read_csv with chunksize=3 to read sales_data.csv in chunks. For each chunk, calculate a new column SalesAmount as Quantity * Price. Filter rows where SalesAmount is greater than sales_threshold. Append these filtered rows to a list called filtered_chunks.

Pandas

import pandas as pd

filtered_chunks = []
# Read sales_data.csv in chunks of size 3
# For each chunk, calculate SalesAmount and filter rows where SalesAmount > sales_threshold
# Append filtered rows to filtered_chunks
# Your code here

Need a hint?

Use a for loop to read the CSV in chunks. Calculate SalesAmount and filter rows where it is greater than sales_threshold. Append filtered rows to filtered_chunks.

4

Combine filtered chunks and print the result

Combine all DataFrames in the list filtered_chunks into a single DataFrame called filtered_data using pd.concat. Then print filtered_data.

Pandas

import pandas as pd

# Combine filtered_chunks into filtered_data
# Print filtered_data
# Your code here

Need a hint?

Use pd.concat(filtered_chunks) to combine the filtered DataFrames. Then print the combined DataFrame.