Working with large datasets strategies in Pandas

Introduction

Large datasets can be slow and hard to handle. Using smart strategies helps you work faster and avoid errors.

When your data file is too big to open in memory all at once.

When you want to speed up data loading and processing.

When you need to save memory while analyzing data.

When you want to process data in smaller parts instead of all at once.

When you want to avoid crashing your computer due to too much data.

Syntax

Pandas

import pandas as pd

# Read data in chunks
for chunk in pd.read_csv('data.csv', chunksize=10000):
    # process each chunk
    pass

# Use specific columns to reduce memory
df = pd.read_csv('data.csv', usecols=['col1', 'col2'])

# Convert data types to save memory
df['col1'] = df['col1'].astype('category')

Reading data in chunks lets you handle big files bit by bit.

Choosing only needed columns and changing data types saves memory.

Examples

This reads the big file in small parts and prints the size of each part.

Pandas

import pandas as pd

# Read CSV file in chunks of 5000 rows
for chunk in pd.read_csv('big_data.csv', chunksize=5000):
    print(chunk.shape)

This loads only the 'Name' and 'Age' columns from the file.

Pandas

import pandas as pd

# Load only two columns to save memory
df = pd.read_csv('big_data.csv', usecols=['Name', 'Age'])
print(df.head())

Changing a column to 'category' type reduces memory if it has repeated values.

Pandas

import pandas as pd

df = pd.read_csv('big_data.csv')
# Change 'Category' column to category type
if 'Category' in df.columns:
    df['Category'] = df['Category'].astype('category')
print(df.info())

Sample Program

This program shows three strategies: reading data in chunks, loading only needed columns, and converting a column to a category type to save memory.

Pandas

import pandas as pd

# Simulate reading a large CSV in chunks
# Here we create a sample CSV first
sample_data = pd.DataFrame({
    'ID': range(1, 20001),
    'Value': ['A']*10000 + ['B']*10000,
    'Number': range(20000, 0, -1)
})
sample_data.to_csv('sample_large.csv', index=False)

# Process the CSV in chunks and count values
value_counts = {}
for chunk in pd.read_csv('sample_large.csv', chunksize=5000):
    counts = chunk['Value'].value_counts()
    for val, count in counts.items():
        value_counts[val] = value_counts.get(val, 0) + count

print('Value counts from chunks:')
print(value_counts)

# Load only 'ID' and 'Number' columns to save memory
df_small = pd.read_csv('sample_large.csv', usecols=['ID', 'Number'])
print('\nDataFrame with selected columns:')
print(df_small.head())

# Convert 'Value' column to category type
df_full = pd.read_csv('sample_large.csv')
df_full['Value'] = df_full['Value'].astype('category')
print('\nMemory usage after converting to category:')
print(df_full.memory_usage(deep=True))

Important Notes

Always test chunk size to find the best balance between speed and memory.

Using 'category' type works best for columns with few unique values.

Loading only needed columns reduces memory but you lose other data.

Summary

Large datasets can be handled by reading in chunks, selecting columns, and changing data types.

These strategies help save memory and speed up processing.

Try these methods to work comfortably with big data on your computer.

Practice

1. Which of the following is a common strategy to handle large datasets efficiently in pandas?

easy

A. Loading the entire dataset into memory at once

B. Avoiding any filtering or selection of columns

C. Using only string data types for all columns

D. Reading the data in smaller chunks

Solution

1. Step 1: Understand memory constraints with large data

   Loading a large dataset fully can use too much memory and slow down processing.

2. Step 2: Identify efficient data loading methods

   Reading data in smaller chunks allows processing parts of the data without loading all at once.

3. Final Answer:

   Reading the data in smaller chunks -> Option D

4. Quick Check:

   Chunk reading = efficient large data handling [OK]

Hint: Use chunk reading to save memory with big data [OK]

Common Mistakes:

• Trying to load entire dataset at once
• Not filtering unnecessary columns
• Using inefficient data types

2. Which pandas function parameter helps to read only specific columns from a large CSV file?

easy

A. skiprows

B. usecols

C. nrows

D. chunksize

Solution

1. Step 1: Recall pandas read_csv parameters

   Parameters like usecols, chunksize, nrows, and skiprows control how data is read.

2. Step 2: Identify parameter for selecting columns

   usecols lets you specify which columns to load, reducing memory usage.

3. Final Answer:

   usecols -> Option B

4. Quick Check:

   usecols = select columns [OK]

Hint: Select columns with usecols to reduce memory [OK]

Common Mistakes:

• Confusing chunksize with column selection
• Using nrows to select columns
• Using skiprows to filter columns

3. What will be the output of the following code snippet?

import pandas as pd

df = pd.read_csv('data.csv', usecols=['A', 'B'], nrows=3)
print(df.shape)

medium

A. (3, 2)

B. (2, 3)

C. (3, 3)

D. (2, 2)

Solution

1. Step 1: Understand parameters usecols and nrows

   usecols=['A', 'B'] selects 2 columns; nrows=3 reads first 3 rows.

2. Step 2: Determine shape of resulting DataFrame

   Shape is (number of rows, number of columns) = (3, 2).

3. Final Answer:

   (3, 2) -> Option A

4. Quick Check:

   Rows=3, Columns=2 = (3, 2) [OK]

Hint: Shape = (rows, columns) after filtering [OK]

Common Mistakes:

• Mixing rows and columns in shape output
• Assuming all columns are loaded
• Ignoring nrows parameter effect

4. Identify the error in this code for reading a large CSV file in chunks:

import pandas as pd

chunks = pd.read_csv('large.csv', chunksize=1000)
for chunk in chunks:
    print(chunk.head())
    break

medium

A. No error; code runs correctly

B. Missing parentheses in print statement

C. chunksize parameter should be a string

D. Cannot iterate over chunks

Solution

1. Step 1: Check syntax and usage of read_csv with chunksize

   chunksize=1000 returns an iterable TextFileReader object.

2. Step 2: Verify iteration and print usage

   Iterating over chunks and printing first rows with chunk.head() is valid.

3. Final Answer:

   No error; code runs correctly -> Option A

4. Quick Check:

   Chunksize iteration = valid [OK]

Hint: Chunksize returns iterable; loop over chunks works [OK]

Common Mistakes:

• Thinking chunksize must be string
• Assuming chunks are not iterable
• Forgetting parentheses in print (Python 3)

5. You have a large CSV file with many columns, but you only need columns 'X', 'Y', and 'Z'. You want to read the file efficiently and convert column 'Y' to a smaller numeric type to save memory. Which code snippet correctly applies these strategies?

hard

A. df = pd.read_csv('file.csv', nrows=1000)
df['Y'] = df['Y'].astype('int64')

B. df = pd.read_csv('file.csv')
df = df[['X', 'Y', 'Z']]
df['Y'] = df['Y'].astype('float64')

C. df = pd.read_csv('file.csv', usecols=['X', 'Y', 'Z'])
df['Y'] = df['Y'].astype('float32')

D. df = pd.read_csv('file.csv', usecols=['X', 'Y', 'Z'])
df['Y'] = df['Y'].astype('object')

Solution

1. Step 1: Select only needed columns while reading

   Using usecols=['X', 'Y', 'Z'] reads only required columns, saving memory.

2. Step 2: Convert column 'Y' to smaller numeric type

   astype('float32') reduces memory compared to default float64.

3. Final Answer:

   Use usecols and convert 'Y' to float32 -> Option C

4. Quick Check:

   usecols + astype('float32') = efficient [OK]

Hint: Select columns with usecols and downcast numeric types [OK]

Common Mistakes:

• Loading all columns then filtering
• Using larger numeric types unnecessarily
• Converting numeric to object type