Why data I/O matters in Pandas - Performance Analysis

Time Complexity: Why data I/O matters

O(n)

Understanding Time Complexity

Reading and writing data can take a lot of time when working with pandas.

We want to know how the time to load or save data grows as the data gets bigger.

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

df = pd.read_csv('large_file.csv')
df.to_csv('output_file.csv', index=False)

This code reads a CSV file into a DataFrame and then writes it back to a new CSV file.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Reading and writing each row of the file.
• How many times: Once for each row in the file (n times).

How Execution Grows With Input

As the number of rows grows, the time to read and write grows roughly the same way.

Input Size (n)	Approx. Operations
10	10 reads + 10 writes
100	100 reads + 100 writes
1000	1000 reads + 1000 writes

Pattern observation: The time grows directly with the number of rows; doubling rows doubles the work.

Final Time Complexity

Time Complexity: O(n)

This means the time to read or write data grows in a straight line with the number of rows.

Common Mistake

[X] Wrong: "Reading a file is instant no matter how big it is."

[OK] Correct: The computer must process each row, so bigger files take more time.

Interview Connect

Understanding how data input and output time grows helps you write better code and explain performance clearly.

Self-Check

"What if we read the file in chunks instead of all at once? How would the time complexity change?"

Practice

1. What does pandas.read_csv() do in data science?

easy

A. It reads data from a CSV file into a DataFrame.

B. It saves a DataFrame to a CSV file.

C. It creates a new empty DataFrame.

D. It deletes a CSV file from disk.

Solution

1. Step 1: Understand the function name

   The function name read_csv suggests it reads CSV files.

2. Step 2: Recall pandas I/O functions

   read_csv loads data from a CSV file into a pandas DataFrame for analysis.

3. Final Answer:

   It reads data from a CSV file into a DataFrame. -> Option A

4. Quick Check:

   read_csv = read data [OK]

Hint: Remember: read_csv reads data in, to_csv writes data out [OK]

Common Mistakes:

• Confusing read_csv with to_csv
• Thinking it creates empty DataFrame
• Assuming it deletes files

2. Which of the following is the correct syntax to save a DataFrame df to a CSV file named data.csv?

easy

A. df.to_csv('data.csv')

B. df.read_csv('data.csv')

C. pandas.save_csv(df, 'data.csv')

D. df.save('data.csv')

Solution

1. Step 1: Identify the correct pandas method for saving

   The method to save a DataFrame to CSV is to_csv.

2. Step 2: Check the syntax

   The syntax is df.to_csv('filename.csv'), which matches df.to_csv('data.csv').

3. Final Answer:

   df.to_csv('data.csv') -> Option A

4. Quick Check:

   to_csv saves DataFrame to CSV [OK]

Hint: Use df.to_csv('filename.csv') to save data [OK]

Common Mistakes:

• Using read_csv instead of to_csv
• Using non-existent methods like save_csv or save
• Wrong method call order

3. What will be the output of this code?

import pandas as pd
from io import StringIO
csv_data = 'name,age\nAlice,30\nBob,25'
df = pd.read_csv(StringIO(csv_data))
print(df.shape)

medium

A. (3, 3)

B. (3, 2)

C. (2, 3)

D. (2, 2)

Solution

1. Step 1: Understand the CSV data

   The CSV string has 2 rows of data (Alice and Bob) and 2 columns (name, age).

2. Step 2: Check DataFrame shape

   The shape is (rows, columns) = (2, 2).

3. Final Answer:

   (2, 2) -> Option D

4. Quick Check:

   2 rows and 2 columns = (2, 2) [OK]

Hint: Count rows and columns in CSV to find shape [OK]

Common Mistakes:

• Counting header as a data row
• Mixing rows and columns in shape
• Confusing number of columns

4. Identify the error in this code snippet that tries to read a CSV file:

import pandas as pd
df = pd.read_csv('data.txt')

medium

A. read_csv cannot read text files

B. No error, code is correct

C. File extension should be .csv, not .txt

D. Missing import statement

Solution

1. Step 1: Examine the code for syntax issues

   The code has proper import and correct read_csv call. No syntax errors.

2. Step 2: Understand file handling

   pandas read_csv reads files based on content, not extension. 'data.txt' works if it contains CSV data.

3. Final Answer:

   No error, code is correct -> Option B

4. Quick Check:

   read_csv ignores file extension [OK]

Hint: File extension (.csv or .txt) doesn't matter for read_csv if format is correct [OK]

Common Mistakes:

• Insisting on .csv extension only
• Believing read_csv can't handle .txt files
• Overlooking the present import statement

5. You have a DataFrame df with columns ['A', 'B', 'C']. You want to save only columns A and C to a CSV file named subset.csv. Which code correctly does this?

hard

A. df.to_csv('subset.csv', columns=['A', 'C'])

B. df.to_csv('subset.csv', cols=['A', 'C'])

C. df[['A', 'C']].to_csv('subset.csv', index=False)

D. df.to_csv('subset.csv').select(['A', 'C'])

Solution

1. Step 1: Recall to_csv column selection

   Slicing df[['A', 'C']] creates a subset DataFrame with only desired columns.

2. Step 2: Proper saving syntax

   df[['A', 'C']].to_csv('subset.csv', index=False) saves exactly A and C without index column.

3. Step 3: Evaluate alternatives

   df.to_csv('subset.csv', columns=['A', 'C']) saves index + A + C (extra column). df.to_csv('subset.csv', cols=['A', 'C']) invalid cols param. df.to_csv('subset.csv').select(['A', 'C']) invalid method chain.

4. Final Answer:

   df[['A', 'C']].to_csv('subset.csv', index=False) -> Option C

5. Quick Check:

   df[cols].to_csv(index=False) excludes index [OK]

Hint: df[['A','C']].to_csv('subset.csv', index=False) for only those columns [OK]

Common Mistakes:

• Using columns= without index=False (adds index column)
• Using invalid cols parameter
• Chaining non-existent .select() method