Memory-efficient operations in Data Analysis Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of memory-efficient data filtering

What is the output of this code that filters a large list using a generator expression?

Data Analysis Python

data = range(1_000_000)
filtered = (x for x in data if x % 100_000 == 0)
result = list(filtered)
print(result)

A [0, 100000, 200000, 300000, 400000, 500000, 600000, 700000, 800000, 900000, 1000000]

B [0, 100000, 200000, 300000, 400000, 500000, 600000, 700000, 800000, 900000]

C [100000, 200000, 300000, 400000, 500000, 600000, 700000, 800000, 900000]

D SyntaxError

💡 Hint

Remember that range stops before the stop value and 0 is included in range(1_000_000).

The range goes from 0 to 999,999. Numbers divisible by 100,000 in this range include 0 up to 900,000. 1,000,000 is not included.

❓ data_output

intermediate

Memory usage difference between list and generator

Which option correctly describes the memory usage difference when creating a list vs a generator for numbers 0 to 999,999?

Data Analysis Python

import sys
list_data = list(range(1_000_000))
gen_data = (x for x in range(1_000_000))
print(sys.getsizeof(list_data))
print(sys.getsizeof(gen_data))

A Both use the same amount of memory

B Generator uses more memory than list

C List uses more memory than generator

D Both cause MemoryError

💡 Hint

Think about when values are stored in memory for lists vs generators.

Lists store all elements in memory, while generators produce items one by one, using much less memory.

🔧 Debug

advanced

Identify the error in memory-efficient data processing

What error will this code raise when trying to sum a large dataset using a generator?

Data Analysis Python

data = (int(x) for x in ['1', '2', 'three', '4'])
total = sum(data)
print(total)

A ValueError

B No error, output is 10

C SyntaxError

D TypeError

💡 Hint

Check the conversion of strings to integers.

The string 'three' cannot be converted to int, causing a ValueError during iteration.

🚀 Application

advanced

Choosing memory-efficient data aggregation method

You have a huge CSV file with millions of rows. Which method is most memory-efficient to calculate the average of a numeric column without loading all data at once?

A Convert CSV to JSON and then parse all data into memory

B Load entire CSV into a pandas DataFrame and use df['col'].mean()

C Use list comprehension to create a list of all values, then average

D Read the file line by line, sum values and count rows, then compute average

💡 Hint

Think about how to avoid loading all data into memory.

Reading line by line and aggregating avoids loading the entire dataset, saving memory.

🧠 Conceptual

expert

Understanding memory-efficient streaming with pandas

Which pandas function allows processing large CSV files in chunks to reduce memory usage?

A pd.read_csv with chunksize parameter

B pd.read_csv with header=None

C pd.read_csv with skiprows parameter

D pd.read_csv with nrows parameter

💡 Hint

Look for a parameter that reads the file in parts.

The chunksize parameter reads the file in smaller parts (chunks), allowing memory-efficient processing.