Chunked reading for large files in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of chunked sum aggregation

What is the output of this code that reads a CSV file in chunks and sums a column?

Pandas

import pandas as pd
from io import StringIO

data = '''id,value
1,10
2,20
3,30
4,40
5,50
'''

chunk_iter = pd.read_csv(StringIO(data), chunksize=2)
total = 0
for chunk in chunk_iter:
    total += chunk['value'].sum()
print(total)

A SyntaxError

B 100

C 50

D 150

💡 Hint

Sum all values in the 'value' column across all chunks.

The code reads the CSV in chunks of 2 rows each. The values are 10,20,30,40,50. Summing all gives 150.

❓ data_output

intermediate

Number of chunks read

How many chunks will be read from this CSV file with 7 rows and chunksize=3?

Pandas

import pandas as pd
from io import StringIO

data = '''id,value
1,5
2,10
3,15
4,20
5,25
6,30
7,35
'''

chunk_iter = pd.read_csv(StringIO(data), chunksize=3)
count = 0
for chunk in chunk_iter:
    count += 1
print(count)

A 4

B 3

C 2

D 7

💡 Hint

Divide total rows by chunk size and round up.

7 rows with chunksize 3 means chunks: 3,3,1 rows. So total 3 chunks.

🔧 Debug

advanced

Identify error in chunk processing code

What error does this code raise when reading a CSV in chunks and trying to access a non-existent column?

Pandas

import pandas as pd
from io import StringIO

data = '''id,value
1,100
2,200
'''

chunk_iter = pd.read_csv(StringIO(data), chunksize=1)
for chunk in chunk_iter:
    print(chunk['amount'].sum())

A KeyError

B TypeError

C ValueError

D No error, prints 0

💡 Hint

Check if the column 'amount' exists in the data.

The column 'amount' does not exist in the DataFrame, so accessing it raises a KeyError.

🚀 Application

advanced

Efficiently count rows with condition using chunks

You want to count how many rows in a large CSV have 'value' greater than 50 using chunked reading. Which code snippet correctly does this?

A

count = 0
for chunk in pd.read_csv('file.csv', chunksize=1000):
    count += chunk['value'].count() > 50
print(count)

B

count = 0
for chunk in pd.read_csv('file.csv', chunksize=1000):
    count += chunk['value'].sum() > 50
print(count)

C

count = 0
for chunk in pd.read_csv('file.csv', chunksize=1000):
    count += (chunk['value'] > 50).sum()
print(count)

D

count = 0
for chunk in pd.read_csv('file.csv', chunksize=1000):
    count = (chunk['value'] > 50).sum()
print(count)

💡 Hint

Sum the boolean condition counts for each chunk.

Option C correctly sums the number of rows where 'value' > 50 across all chunks. Others either overwrite count or use wrong logic.

🧠 Conceptual

expert

Why use chunked reading for large files?

Which is the main reason to use chunked reading when processing large CSV files?

A To reduce memory usage by loading only parts of the file at a time

B To speed up reading by loading the entire file at once

C To convert CSV files into JSON format

D To automatically clean data during reading

💡 Hint

Think about memory limits when files are very large.

Chunked reading loads small parts of the file sequentially, avoiding high memory use from loading the whole file.