Working with large datasets strategies in Pandas - Mini Project: Build & Apply

Working with Large Datasets Strategies

📖 Scenario: You work as a data analyst for a retail company. You have a large dataset of sales transactions. The dataset is too big to load all at once, so you need to work with smaller parts and apply filters to manage memory and speed.

🎯 Goal: Learn how to load a large dataset in chunks, filter data based on a condition, and combine the filtered results into a smaller dataset for analysis.

📋 What You'll Learn

Use pandas to read CSV data in chunks

Filter data based on a sales amount threshold

Combine filtered chunks into a single DataFrame

Print the final filtered DataFrame

💡 Why This Matters

🌍 Real World

Working with large datasets is common in data science. Loading data in chunks helps manage memory and speeds up processing.

💼 Career

Data analysts and scientists often need to handle big data files efficiently. Knowing how to filter and process data in parts is a valuable skill.

1

Create a sample large dataset CSV file

Create a CSV file named sales_data.csv with these exact columns and rows:
TransactionID,Product,Quantity,Price
Rows:
1,Apple,10,0.5
2,Banana,5,0.3
3,Orange,8,0.7
4,Apple,3,0.5
5,Banana,7,0.3
6,Orange,2,0.7
7,Apple,12,0.5
8,Banana,1,0.3
9,Orange,4,0.7
10,Apple,6,0.5

Pandas

# Create the CSV file sales_data.csv with the exact data
# Your code here

Hint

Use pandas.DataFrame.to_csv to save the data to a CSV file named sales_data.csv.

2

Set the sales amount threshold

Create a variable called sales_threshold and set it to 5. This will be used to filter transactions with total sales above this value.

Pandas

import pandas as pd

# Set the sales amount threshold
# Your code here

Hint

Just create a variable named sales_threshold and assign the value 5.

3

Read the CSV in chunks and filter by sales amount

Use pandas.read_csv with chunksize=3 to read sales_data.csv in chunks. For each chunk, calculate a new column SalesAmount as Quantity * Price. Filter rows where SalesAmount is greater than sales_threshold. Append these filtered rows to a list called filtered_chunks.

Pandas

import pandas as pd

filtered_chunks = []
# Read sales_data.csv in chunks of size 3
# For each chunk, calculate SalesAmount and filter rows where SalesAmount > sales_threshold
# Append filtered rows to filtered_chunks
# Your code here

Hint

Use a for loop to read the CSV in chunks. Calculate SalesAmount and filter rows where it is greater than sales_threshold. Append filtered rows to filtered_chunks.

4

Combine filtered chunks and print the result

Combine all DataFrames in the list filtered_chunks into a single DataFrame called filtered_data using pd.concat. Then print filtered_data.

Pandas

import pandas as pd

# Combine filtered_chunks into filtered_data
# Print filtered_data
# Your code here

Hint

Use pd.concat(filtered_chunks) to combine the filtered DataFrames. Then print the combined DataFrame.

Practice

1. Which of the following is a common strategy to handle large datasets efficiently in pandas?

easy

A. Loading the entire dataset into memory at once

B. Avoiding any filtering or selection of columns

C. Using only string data types for all columns

D. Reading the data in smaller chunks

Solution

1. Step 1: Understand memory constraints with large data

   Loading a large dataset fully can use too much memory and slow down processing.

2. Step 2: Identify efficient data loading methods

   Reading data in smaller chunks allows processing parts of the data without loading all at once.

3. Final Answer:

   Reading the data in smaller chunks -> Option D

4. Quick Check:

   Chunk reading = efficient large data handling [OK]

Hint: Use chunk reading to save memory with big data [OK]

Common Mistakes:

• Trying to load entire dataset at once
• Not filtering unnecessary columns
• Using inefficient data types

2. Which pandas function parameter helps to read only specific columns from a large CSV file?

easy

A. skiprows

B. usecols

C. nrows

D. chunksize

Solution

1. Step 1: Recall pandas read_csv parameters

   Parameters like usecols, chunksize, nrows, and skiprows control how data is read.

2. Step 2: Identify parameter for selecting columns

   usecols lets you specify which columns to load, reducing memory usage.

3. Final Answer:

   usecols -> Option B

4. Quick Check:

   usecols = select columns [OK]

Hint: Select columns with usecols to reduce memory [OK]

Common Mistakes:

• Confusing chunksize with column selection
• Using nrows to select columns
• Using skiprows to filter columns

3. What will be the output of the following code snippet?

import pandas as pd

df = pd.read_csv('data.csv', usecols=['A', 'B'], nrows=3)
print(df.shape)

medium

A. (3, 2)

B. (2, 3)

C. (3, 3)

D. (2, 2)

Solution

1. Step 1: Understand parameters usecols and nrows

   usecols=['A', 'B'] selects 2 columns; nrows=3 reads first 3 rows.

2. Step 2: Determine shape of resulting DataFrame

   Shape is (number of rows, number of columns) = (3, 2).

3. Final Answer:

   (3, 2) -> Option A

4. Quick Check:

   Rows=3, Columns=2 = (3, 2) [OK]

Hint: Shape = (rows, columns) after filtering [OK]

Common Mistakes:

• Mixing rows and columns in shape output
• Assuming all columns are loaded
• Ignoring nrows parameter effect

4. Identify the error in this code for reading a large CSV file in chunks:

import pandas as pd

chunks = pd.read_csv('large.csv', chunksize=1000)
for chunk in chunks:
    print(chunk.head())
    break

medium

A. No error; code runs correctly

B. Missing parentheses in print statement

C. chunksize parameter should be a string

D. Cannot iterate over chunks

Solution

1. Step 1: Check syntax and usage of read_csv with chunksize

   chunksize=1000 returns an iterable TextFileReader object.

2. Step 2: Verify iteration and print usage

   Iterating over chunks and printing first rows with chunk.head() is valid.

3. Final Answer:

   No error; code runs correctly -> Option A

4. Quick Check:

   Chunksize iteration = valid [OK]

Hint: Chunksize returns iterable; loop over chunks works [OK]

Common Mistakes:

• Thinking chunksize must be string
• Assuming chunks are not iterable
• Forgetting parentheses in print (Python 3)

5. You have a large CSV file with many columns, but you only need columns 'X', 'Y', and 'Z'. You want to read the file efficiently and convert column 'Y' to a smaller numeric type to save memory. Which code snippet correctly applies these strategies?

hard

A. df = pd.read_csv('file.csv', nrows=1000)
df['Y'] = df['Y'].astype('int64')

B. df = pd.read_csv('file.csv')
df = df[['X', 'Y', 'Z']]
df['Y'] = df['Y'].astype('float64')

C. df = pd.read_csv('file.csv', usecols=['X', 'Y', 'Z'])
df['Y'] = df['Y'].astype('float32')

D. df = pd.read_csv('file.csv', usecols=['X', 'Y', 'Z'])
df['Y'] = df['Y'].astype('object')

Solution

1. Step 1: Select only needed columns while reading

   Using usecols=['X', 'Y', 'Z'] reads only required columns, saving memory.

2. Step 2: Convert column 'Y' to smaller numeric type

   astype('float32') reduces memory compared to default float64.

3. Final Answer:

   Use usecols and convert 'Y' to float32 -> Option C

4. Quick Check:

   usecols + astype('float32') = efficient [OK]

Hint: Select columns with usecols and downcast numeric types [OK]

Common Mistakes:

• Loading all columns then filtering
• Using larger numeric types unnecessarily
• Converting numeric to object type