Working with large datasets strategies in Pandas - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to read a large CSV file in chunks using pandas.

Pandas

chunk_iter = pd.read_csv('large_data.csv', chunksize=[1])

Drag options to blanks, or click blank then click option'

A1000

B10000

C100

D10

Attempts:

3 left

💡 Hint

Common Mistakes

Using too small chunksize slows down processing.

Using no chunksize reads entire file at once, causing memory issues.

Using a chunksize of 10000 allows pandas to read the file in manageable pieces, which helps with memory efficiency.

2fill in blank

medium

Complete the code to select only specific columns while reading a large CSV file.

Pandas

df = pd.read_csv('large_data.csv', usecols=[1])

Drag options to blanks, or click blank then click option'

A['col1', 'col2']

B['all']

C['col1', 'col3', 'col5']

D['1', '2']

Attempts:

3 left

💡 Hint

Common Mistakes

Passing column indexes as strings instead of names.

Using 'all' which is not a valid option.

Selecting only needed columns reduces memory use and speeds up processing.

3fill in blank

hard

Fix the error in the code to convert a large DataFrame column to category type for memory saving.

Pandas

df['category_col'] = df['category_col'].[1]('category')

Drag options to blanks, or click blank then click option'

Achange_type

Bconvert

Cto_type

Dastype

Attempts:

3 left

💡 Hint

Common Mistakes

Using non-existent methods like convert or to_type.

Trying to assign type directly without a method.

The astype method converts the column to the specified type, here 'category' to save memory.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that maps words to their lengths only if length is greater than 3.

Pandas

{word: [1] for word in words if [2] > 3}

Drag options to blanks, or click blank then click option'

Alen(word)

Bword

Attempts:

3 left

💡 Hint

Common Mistakes

Using the word itself as the value instead of its length.

Checking the word instead of its length in the condition.

The dictionary comprehension maps each word to its length using len(word) and filters words with length greater than 3.

5fill in blank

hard

Fill all three blanks to create a filtered dictionary from data where keys are uppercase and values are positive.

Pandas

result = [1]: [2] for [3], v in data.items() if v > 0}

Drag options to blanks, or click blank then click option'

Ak.upper()

Bv

Ck

Dv.upper()

Attempts:

3 left

💡 Hint

Common Mistakes

Using value instead of key for uppercase conversion.

Including negative or zero values.

This comprehension creates a dictionary with keys converted to uppercase and values kept as is, only including positive values.

Practice

1. Which of the following is a common strategy to handle large datasets efficiently in pandas?

easy

A. Loading the entire dataset into memory at once

B. Avoiding any filtering or selection of columns

C. Using only string data types for all columns

D. Reading the data in smaller chunks

Solution

1. Step 1: Understand memory constraints with large data

   Loading a large dataset fully can use too much memory and slow down processing.

2. Step 2: Identify efficient data loading methods

   Reading data in smaller chunks allows processing parts of the data without loading all at once.

3. Final Answer:

   Reading the data in smaller chunks -> Option D

4. Quick Check:

   Chunk reading = efficient large data handling [OK]

Hint: Use chunk reading to save memory with big data [OK]

Common Mistakes:

• Trying to load entire dataset at once
• Not filtering unnecessary columns
• Using inefficient data types

2. Which pandas function parameter helps to read only specific columns from a large CSV file?

easy

A. skiprows

B. usecols

C. nrows

D. chunksize

Solution

1. Step 1: Recall pandas read_csv parameters

   Parameters like usecols, chunksize, nrows, and skiprows control how data is read.

2. Step 2: Identify parameter for selecting columns

   usecols lets you specify which columns to load, reducing memory usage.

3. Final Answer:

   usecols -> Option B

4. Quick Check:

   usecols = select columns [OK]

Hint: Select columns with usecols to reduce memory [OK]

Common Mistakes:

• Confusing chunksize with column selection
• Using nrows to select columns
• Using skiprows to filter columns

3. What will be the output of the following code snippet?

import pandas as pd

df = pd.read_csv('data.csv', usecols=['A', 'B'], nrows=3)
print(df.shape)

medium

A. (3, 2)

B. (2, 3)

C. (3, 3)

D. (2, 2)

Solution

1. Step 1: Understand parameters usecols and nrows

   usecols=['A', 'B'] selects 2 columns; nrows=3 reads first 3 rows.

2. Step 2: Determine shape of resulting DataFrame

   Shape is (number of rows, number of columns) = (3, 2).

3. Final Answer:

   (3, 2) -> Option A

4. Quick Check:

   Rows=3, Columns=2 = (3, 2) [OK]

Hint: Shape = (rows, columns) after filtering [OK]

Common Mistakes:

• Mixing rows and columns in shape output
• Assuming all columns are loaded
• Ignoring nrows parameter effect

4. Identify the error in this code for reading a large CSV file in chunks:

import pandas as pd

chunks = pd.read_csv('large.csv', chunksize=1000)
for chunk in chunks:
    print(chunk.head())
    break

medium

A. No error; code runs correctly

B. Missing parentheses in print statement

C. chunksize parameter should be a string

D. Cannot iterate over chunks

Solution

1. Step 1: Check syntax and usage of read_csv with chunksize

   chunksize=1000 returns an iterable TextFileReader object.

2. Step 2: Verify iteration and print usage

   Iterating over chunks and printing first rows with chunk.head() is valid.

3. Final Answer:

   No error; code runs correctly -> Option A

4. Quick Check:

   Chunksize iteration = valid [OK]

Hint: Chunksize returns iterable; loop over chunks works [OK]

Common Mistakes:

• Thinking chunksize must be string
• Assuming chunks are not iterable
• Forgetting parentheses in print (Python 3)

5. You have a large CSV file with many columns, but you only need columns 'X', 'Y', and 'Z'. You want to read the file efficiently and convert column 'Y' to a smaller numeric type to save memory. Which code snippet correctly applies these strategies?

hard

A. df = pd.read_csv('file.csv', nrows=1000)
df['Y'] = df['Y'].astype('int64')

B. df = pd.read_csv('file.csv')
df = df[['X', 'Y', 'Z']]
df['Y'] = df['Y'].astype('float64')

C. df = pd.read_csv('file.csv', usecols=['X', 'Y', 'Z'])
df['Y'] = df['Y'].astype('float32')

D. df = pd.read_csv('file.csv', usecols=['X', 'Y', 'Z'])
df['Y'] = df['Y'].astype('object')

Solution

1. Step 1: Select only needed columns while reading

   Using usecols=['X', 'Y', 'Z'] reads only required columns, saving memory.

2. Step 2: Convert column 'Y' to smaller numeric type

   astype('float32') reduces memory compared to default float64.

3. Final Answer:

   Use usecols and convert 'Y' to float32 -> Option C

4. Quick Check:

   usecols + astype('float32') = efficient [OK]

Hint: Select columns with usecols and downcast numeric types [OK]

Common Mistakes:

• Loading all columns then filtering
• Using larger numeric types unnecessarily
• Converting numeric to object type