Memory-efficient operations in Data Analysis Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of memory-efficient data filtering

What is the output of this code that filters a large list using a generator expression?

Data Analysis Python

data = range(1_000_000)
filtered = (x for x in data if x % 100_000 == 0)
result = list(filtered)
print(result)

A [0, 100000, 200000, 300000, 400000, 500000, 600000, 700000, 800000, 900000, 1000000]

B [0, 100000, 200000, 300000, 400000, 500000, 600000, 700000, 800000, 900000]

C [100000, 200000, 300000, 400000, 500000, 600000, 700000, 800000, 900000]

D SyntaxError

💡 Hint

Remember that range stops before the stop value and 0 is included in range(1_000_000).

The range goes from 0 to 999,999. Numbers divisible by 100,000 in this range include 0 up to 900,000. 1,000,000 is not included.

❓ data_output

intermediate

Memory usage difference between list and generator

Which option correctly describes the memory usage difference when creating a list vs a generator for numbers 0 to 999,999?

Data Analysis Python

import sys
list_data = list(range(1_000_000))
gen_data = (x for x in range(1_000_000))
print(sys.getsizeof(list_data))
print(sys.getsizeof(gen_data))

A Both use the same amount of memory

B Generator uses more memory than list

C List uses more memory than generator

D Both cause MemoryError

💡 Hint

Think about when values are stored in memory for lists vs generators.

Lists store all elements in memory, while generators produce items one by one, using much less memory.

🔧 Debug

advanced

Identify the error in memory-efficient data processing

What error will this code raise when trying to sum a large dataset using a generator?

Data Analysis Python

data = (int(x) for x in ['1', '2', 'three', '4'])
total = sum(data)
print(total)

A ValueError

B No error, output is 10

C SyntaxError

D TypeError

💡 Hint

Check the conversion of strings to integers.

The string 'three' cannot be converted to int, causing a ValueError during iteration.

🚀 Application

advanced

Choosing memory-efficient data aggregation method

You have a huge CSV file with millions of rows. Which method is most memory-efficient to calculate the average of a numeric column without loading all data at once?

A Convert CSV to JSON and then parse all data into memory

B Load entire CSV into a pandas DataFrame and use df['col'].mean()

C Use list comprehension to create a list of all values, then average

D Read the file line by line, sum values and count rows, then compute average

💡 Hint

Think about how to avoid loading all data into memory.

Reading line by line and aggregating avoids loading the entire dataset, saving memory.

🧠 Conceptual

expert

Understanding memory-efficient streaming with pandas

Which pandas function allows processing large CSV files in chunks to reduce memory usage?

A pd.read_csv with chunksize parameter

B pd.read_csv with header=None

C pd.read_csv with skiprows parameter

D pd.read_csv with nrows parameter

💡 Hint

Look for a parameter that reads the file in parts.

The chunksize parameter reads the file in smaller parts (chunks), allowing memory-efficient processing.

Practice

1. Which of the following is a common method to reduce memory usage when working with large datasets in Python?

easy

A. Increase the size of integer columns to 64-bit

B. Use print statements to debug data

C. Convert text columns to 'category' data type

D. Load the entire dataset into memory at once

Solution

1. Step 1: Understand memory usage of data types

   Text columns with repeated values use more memory as strings than as categories.

2. Step 2: Apply category conversion

   Converting text columns to 'category' type stores unique values once, saving memory.

3. Final Answer:

   Convert text columns to 'category' data type -> Option C

4. Quick Check:

   Category type reduces memory for repeated text [OK]

Hint: Convert repeated text columns to 'category' type to save memory [OK]

Common Mistakes:

• Thinking increasing integer size saves memory
• Loading full dataset without chunking
• Using print for memory optimization

2. Which of the following is the correct syntax to read a large CSV file in chunks of 1000 rows using pandas?

easy

A. pd.read_csv('file.csv', chunksize=1000)

B. pd.read_csv('file.csv', chunk=1000)

C. pd.read_csv('file.csv', size=1000)

D. pd.read_csv('file.csv', batch=1000)

Solution

1. Step 1: Recall pandas chunk reading parameter

   The correct parameter to read in chunks is 'chunksize'.

2. Step 2: Match syntax

   Only pd.read_csv('file.csv', chunksize=1000) uses 'chunksize=1000', which is correct syntax.

3. Final Answer:

   pd.read_csv('file.csv', chunksize=1000) -> Option A

4. Quick Check:

   chunksize=1000 reads file in 1000-row chunks [OK]

Hint: Use 'chunksize' parameter to read CSV in parts [OK]

Common Mistakes:

• Using 'chunk' instead of 'chunksize'
• Using 'size' or 'batch' which are invalid
• Forgetting to specify chunk size

3. What will be the output of the following code snippet?

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'A': np.random.randint(0, 100, 5),
    'B': ['cat', 'dog', 'cat', 'dog', 'cat']
})
df['B'] = df['B'].astype('category')
print(df.memory_usage(deep=True))

medium

A. Raises a TypeError due to invalid astype conversion

B. Prints the DataFrame instead of memory usage

C. Shows memory usage with 'B' column using more memory than object type

D. Shows memory usage with 'B' column using less memory than object type

Solution

1. Step 1: Understand astype('category') effect

   Converting column 'B' to category reduces memory by storing unique values once.

2. Step 2: Interpret memory_usage(deep=True)

   This method shows memory used by each column including actual string data size.

3. Final Answer:

   Shows memory usage with 'B' column using less memory than object type -> Option D

4. Quick Check:

   Category type reduces memory usage [OK]

Hint: astype('category') lowers memory for repeated text columns [OK]

Common Mistakes:

• Expecting error from astype conversion
• Thinking category uses more memory
• Confusing print output with DataFrame display

4. Identify the error in the following code that attempts to reduce memory usage:

import pandas as pd

df = pd.read_csv('data.csv')
df['col'] = df['col'].astype('category')
df = df.astype({'num_col': 'int8'})
print(df.memory_usage())

medium

A. Using 'int8' may cause overflow if values exceed range

B. memory_usage() does not show memory savings

C. astype cannot convert columns after reading CSV

D. No error; code correctly reduces memory usage

Solution

1. Step 1: Check int8 range suitability

   int8 supports values from -128 to 127; larger values cause overflow.

2. Step 2: Identify potential overflow risk

   If 'num_col' has values outside int8 range, conversion causes incorrect data.

3. Final Answer:

   Using 'int8' may cause overflow if values exceed range -> Option A

4. Quick Check:

   int8 range limits can cause data errors [OK]

Hint: Check numeric ranges before downcasting to smaller types [OK]

Common Mistakes:

• Assuming all int conversions are safe
• Believing memory_usage() ignores changes
• Thinking astype fails after CSV read

5. You have a large dataset with a column 'status' containing only three unique string values repeated many times. Which memory-efficient approach is best to reduce memory usage and why? Options: A) Drop the 'status' column to save memory. B) Convert 'status' to string type and keep numeric columns as float64. C) Convert 'status' to category type and downcast numeric columns to smallest possible integer type. D) Convert all columns to object type for flexibility.

hard

A. Drop the 'status' column to save memory.

B. Convert 'status' to category type and downcast numeric columns to smallest possible integer type.

C. Convert 'status' to string type and keep numeric columns as float64.

D. Convert all columns to object type for flexibility.

Solution

1. Step 1: Use category for repeated strings

   Converting 'status' to category stores unique values once, saving memory.

2. Step 2: Downcast numeric columns

   Downcasting numeric columns to smallest integer type reduces memory without losing data.

3. Step 3: Evaluate other options

   Keeping strings as object or dropping columns wastes memory or data; object type uses more memory.

4. Final Answer:

   Convert 'status' to category type and downcast numeric columns to smallest possible integer type. -> Option B

5. Quick Check:

   Category + downcast = best memory saving [OK]

Hint: Use category for repeated strings and downcast numbers [OK]

Common Mistakes:

• Dropping useful columns unnecessarily
• Keeping strings as object wastes memory
• Converting all to object increases memory