Working with large datasets strategies in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this code when reading a large CSV in chunks?

Consider this code that reads a large CSV file in chunks and counts total rows:

import pandas as pd
chunk_iter = pd.read_csv('large_data.csv', chunksize=1000)
total_rows = 0
for chunk in chunk_iter:
    total_rows += len(chunk)
print(total_rows)

If the CSV has 4523 rows, what will be printed?

Pandas

import pandas as pd
chunk_iter = pd.read_csv('large_data.csv', chunksize=1000)
total_rows = 0
for chunk in chunk_iter:
    total_rows += len(chunk)
print(total_rows)

A 4000

B 4523

C 5000

D 1000

💡 Hint

Think about how chunksize controls the number of rows per chunk and how the loop sums all rows.

The code reads the CSV in chunks of 1000 rows. It sums the length of each chunk. Since the file has 4523 rows, the total sum is 4523.

❓ data_output

intermediate

What is the memory usage after downcasting numeric columns?

Given this DataFrame with integer columns:

import pandas as pd
import numpy as np
df = pd.DataFrame({
    'A': np.random.randint(0, 1000, size=10000),
    'B': np.random.randint(0, 100000, size=10000)
})
df_optimized = df.copy()
df_optimized['A'] = pd.to_numeric(df_optimized['A'], downcast='unsigned')
df_optimized['B'] = pd.to_numeric(df_optimized['B'], downcast='unsigned')
print(df_optimized.memory_usage(deep=True).sum())

Which option best describes the output?

Pandas

import pandas as pd
import numpy as np
df = pd.DataFrame({
    'A': np.random.randint(0, 1000, size=10000),
    'B': np.random.randint(0, 100000, size=10000)
})
df_optimized = df.copy()
df_optimized['A'] = pd.to_numeric(df_optimized['A'], downcast='unsigned')
df_optimized['B'] = pd.to_numeric(df_optimized['B'], downcast='unsigned')
print(df_optimized.memory_usage(deep=True).sum())

A Exactly the same as original memory usage

B More than original memory usage due to copying

C Less than original memory usage because of downcasting

D Raises a TypeError because downcast='unsigned' is invalid

💡 Hint

Downcasting reduces the size of numeric columns by using smaller data types.

Downcasting converts columns to smaller unsigned integer types, reducing memory usage compared to original int64 columns.

🔧 Debug

advanced

What error does this code raise when filtering a large DataFrame?

Look at this code snippet:

import pandas as pd
df = pd.DataFrame({'value': range(1000000)})
filtered = df[df['value'] > 500000]
filtered.reset_index(inplace=True)
print(filtered.head())

What error will this code produce?

Pandas

import pandas as pd
df = pd.DataFrame({'value': range(1000000)})
filtered = df[df['value'] > 500000]
filtered.reset_index(inplace=True)
print(filtered.head())

A No error, prints first 5 rows with old index as a column

B KeyError because 'index' column does not exist

C TypeError due to inplace=True with reset_index

D ValueError because filtered DataFrame is empty

💡 Hint

reset_index with inplace=True adds the old index as a column named 'index'.

The code filters rows and resets the index inplace. This adds the old index as a new column named 'index'. No error occurs.

❓ visualization

advanced

Which plot best visualizes memory usage reduction after optimization?

You have original and optimized DataFrames with memory usage values:

memory_original = 8000000
memory_optimized = 2000000

Which plot code will clearly show the reduction?

Pandas

import matplotlib.pyplot as plt
memory_original = 8000000
memory_optimized = 2000000
plt.bar(['Original', 'Optimized'], [memory_original, memory_optimized])
plt.ylabel('Memory Usage (bytes)')
plt.title('Memory Usage Before and After Optimization')
plt.show()

A Scatter plot of memory usage vs. row count

B Line plot with memory usage over time

C Pie chart showing percentage of memory used by each column

D Bar chart comparing original and optimized memory usage

💡 Hint

Bar charts are good for comparing two values side by side.

The bar chart clearly shows the difference between original and optimized memory usage as two bars.

🧠 Conceptual

expert

Which strategy is best for handling datasets larger than RAM?

You have a dataset too large to fit into your computer's RAM. Which approach is best to analyze it efficiently?

A Use chunked reading with pandas.read_csv and process each chunk separately

B Load entire dataset into memory using pandas.read_csv without chunks

C Convert dataset to Excel and open in spreadsheet software

D Increase RAM physically and then load dataset fully

💡 Hint

Think about how to work with data that doesn't fit in memory.

Reading data in chunks allows processing parts of the dataset sequentially without loading all data at once, which is efficient for large datasets.

Practice

1. Which of the following is a common strategy to handle large datasets efficiently in pandas?

easy

A. Loading the entire dataset into memory at once

B. Avoiding any filtering or selection of columns

C. Using only string data types for all columns

D. Reading the data in smaller chunks

Solution

1. Step 1: Understand memory constraints with large data

   Loading a large dataset fully can use too much memory and slow down processing.

2. Step 2: Identify efficient data loading methods

   Reading data in smaller chunks allows processing parts of the data without loading all at once.

3. Final Answer:

   Reading the data in smaller chunks -> Option D

4. Quick Check:

   Chunk reading = efficient large data handling [OK]

Hint: Use chunk reading to save memory with big data [OK]

Common Mistakes:

• Trying to load entire dataset at once
• Not filtering unnecessary columns
• Using inefficient data types

2. Which pandas function parameter helps to read only specific columns from a large CSV file?

easy

A. skiprows

B. usecols

C. nrows

D. chunksize

Solution

1. Step 1: Recall pandas read_csv parameters

   Parameters like usecols, chunksize, nrows, and skiprows control how data is read.

2. Step 2: Identify parameter for selecting columns

   usecols lets you specify which columns to load, reducing memory usage.

3. Final Answer:

   usecols -> Option B

4. Quick Check:

   usecols = select columns [OK]

Hint: Select columns with usecols to reduce memory [OK]

Common Mistakes:

• Confusing chunksize with column selection
• Using nrows to select columns
• Using skiprows to filter columns

3. What will be the output of the following code snippet?

import pandas as pd

df = pd.read_csv('data.csv', usecols=['A', 'B'], nrows=3)
print(df.shape)

medium

A. (3, 2)

B. (2, 3)

C. (3, 3)

D. (2, 2)

Solution

1. Step 1: Understand parameters usecols and nrows

   usecols=['A', 'B'] selects 2 columns; nrows=3 reads first 3 rows.

2. Step 2: Determine shape of resulting DataFrame

   Shape is (number of rows, number of columns) = (3, 2).

3. Final Answer:

   (3, 2) -> Option A

4. Quick Check:

   Rows=3, Columns=2 = (3, 2) [OK]

Hint: Shape = (rows, columns) after filtering [OK]

Common Mistakes:

• Mixing rows and columns in shape output
• Assuming all columns are loaded
• Ignoring nrows parameter effect

4. Identify the error in this code for reading a large CSV file in chunks:

import pandas as pd

chunks = pd.read_csv('large.csv', chunksize=1000)
for chunk in chunks:
    print(chunk.head())
    break

medium

A. No error; code runs correctly

B. Missing parentheses in print statement

C. chunksize parameter should be a string

D. Cannot iterate over chunks

Solution

1. Step 1: Check syntax and usage of read_csv with chunksize

   chunksize=1000 returns an iterable TextFileReader object.

2. Step 2: Verify iteration and print usage

   Iterating over chunks and printing first rows with chunk.head() is valid.

3. Final Answer:

   No error; code runs correctly -> Option A

4. Quick Check:

   Chunksize iteration = valid [OK]

Hint: Chunksize returns iterable; loop over chunks works [OK]

Common Mistakes:

• Thinking chunksize must be string
• Assuming chunks are not iterable
• Forgetting parentheses in print (Python 3)

5. You have a large CSV file with many columns, but you only need columns 'X', 'Y', and 'Z'. You want to read the file efficiently and convert column 'Y' to a smaller numeric type to save memory. Which code snippet correctly applies these strategies?

hard

A. df = pd.read_csv('file.csv', nrows=1000)
df['Y'] = df['Y'].astype('int64')

B. df = pd.read_csv('file.csv')
df = df[['X', 'Y', 'Z']]
df['Y'] = df['Y'].astype('float64')

C. df = pd.read_csv('file.csv', usecols=['X', 'Y', 'Z'])
df['Y'] = df['Y'].astype('float32')

D. df = pd.read_csv('file.csv', usecols=['X', 'Y', 'Z'])
df['Y'] = df['Y'].astype('object')

Solution

1. Step 1: Select only needed columns while reading

   Using usecols=['X', 'Y', 'Z'] reads only required columns, saving memory.

2. Step 2: Convert column 'Y' to smaller numeric type

   astype('float32') reduces memory compared to default float64.

3. Final Answer:

   Use usecols and convert 'Y' to float32 -> Option C

4. Quick Check:

   usecols + astype('float32') = efficient [OK]

Hint: Select columns with usecols and downcast numeric types [OK]

Common Mistakes:

• Loading all columns then filtering
• Using larger numeric types unnecessarily
• Converting numeric to object type