Why indexing matters in Pandas - Performance Analysis

Time Complexity: Why indexing matters

O(log n)

Understanding Time Complexity

We want to see how using an index in pandas affects how fast operations run.

Does having an index make searching or selecting data faster as the data grows?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

# Create a DataFrame with 1 million rows
n = 1_000_000
df = pd.DataFrame({
    'id': range(n),
    'value': range(n)
})

# Set 'id' as index
indexed_df = df.set_index('id')

# Select a row by index label
result = indexed_df.loc[500_000]

This code creates a large DataFrame, sets an index on the 'id' column, and selects a row by that index.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Searching for a row by index label using the index structure.
• How many times: The search happens once, but the cost depends on how the index is built and how many rows there are.

How Execution Grows With Input

When using an index, searching for a row is much faster than scanning all rows.

Input Size (n)	Approx. Operations
10	About 3-4 steps
100	About 7 steps
1,000,000	About 20 steps

Pattern observation: The number of steps grows slowly as data grows, not one-by-one.

Final Time Complexity

Time Complexity: O(log n)

This means finding a row by index label takes only a few steps even if the data is very large.

Common Mistake

[X] Wrong: "Searching by index is as slow as scanning all rows one by one."

[OK] Correct: Because pandas uses a special structure for the index, it can jump quickly to the right row without checking every row.

Interview Connect

Knowing how indexing speeds up data access shows you understand how to handle big data efficiently, a key skill in data science.

Self-Check

"What if we select a row without setting an index first? How would the time complexity change?"

Practice

1. Why is indexing important in pandas DataFrames?

easy

A. It deletes duplicate rows.

B. It changes the data types of columns.

C. It sorts the DataFrame automatically.

D. It labels rows to find data quickly.

Solution

1. Step 1: Understand the role of indexing

   Indexing assigns labels to rows, making it easier to locate and access data quickly.

2. Step 2: Compare with other options

   Changing data types, sorting, or deleting duplicates are different operations unrelated to indexing.

3. Final Answer:

   It labels rows to find data quickly. -> Option D

4. Quick Check:

   Indexing = labeling rows [OK]

Hint: Indexing helps find rows fast by labeling them [OK]

Common Mistakes:

• Confusing indexing with sorting
• Thinking indexing changes data types
• Assuming indexing deletes duplicates

2. Which of the following is the correct way to set the 'Name' column as the index of a DataFrame df?

easy

A. df.setIndex('Name')

B. df.index('Name')

C. df.set_index('Name')

D. df.index = 'Name'

Solution

1. Step 1: Recall the correct pandas method

   The method to set a column as index is set_index, called as df.set_index('column_name').

2. Step 2: Check other options for syntax errors

   df.index('Name') and df.setIndex('Name') are invalid methods. Assigning df.index = 'Name' sets index to a string, not a column.

3. Final Answer:

   df.set_index('Name') -> Option C

4. Quick Check:

   Use set_index('column') to set index [OK]

Hint: Use df.set_index('column') to set index [OK]

Common Mistakes:

• Using wrong method names like setIndex
• Trying to call df.index as a function
• Assigning a string directly to df.index

3. Given the DataFrame df below:

   Name  Age  Score
0  Alice   25     88
1    Bob   30     92
2  Carol   22     85

What is the output of df.loc[1, 'Score']?

medium

A. 30

B. 92

C. Bob

D. 85

Solution

1. Step 1: Understand loc usage

   loc selects data by label. Here, row label 1 and column 'Score' is requested.

2. Step 2: Find value at row 1, column 'Score'

   Row 1 corresponds to Bob, and his Score is 92.

3. Final Answer:

   92 -> Option B

4. Quick Check:

   df.loc[1, 'Score'] = 92 [OK]

Hint: loc uses row labels, so df.loc[1, 'Score'] = 92 [OK]

Common Mistakes:

• Confusing loc with iloc (position vs label)
• Selecting wrong column value
• Assuming row 1 is index 0

4. What is wrong with this code snippet?

df = pd.DataFrame({'A': [1,2], 'B': [3,4]})
df.set_index('C')

medium

A. Column 'C' does not exist, causing an error.

B. set_index cannot be used on DataFrames.

C. The code will run but not change the index.

D. The DataFrame is empty, so set_index fails.

Solution

1. Step 1: Check if column 'C' exists

   The DataFrame has columns 'A' and 'B' only; 'C' is missing.

2. Step 2: Understand set_index behavior

   Trying to set index to a non-existent column raises a KeyError.

3. Final Answer:

   Column 'C' does not exist, causing an error. -> Option A

4. Quick Check:

   Missing column in set_index causes error [OK]

Hint: Check if column exists before set_index [OK]

Common Mistakes:

• Assuming set_index works without error on missing columns
• Thinking set_index can't be used on DataFrames
• Ignoring error messages

5. You have a DataFrame df with columns 'City', 'Year', and 'Population'. You want to quickly select all rows for the city 'Paris' using indexing. Which steps correctly prepare and select the data?

hard

A. df.set_index('City', inplace=True)\ndf.loc['Paris']

B. df.set_index('Year', inplace=True)\ndf.loc['Paris']

C. df.set_index('Population', inplace=True)\ndf.loc['Paris']

D. df.loc[df['City'] == 'Paris']

Solution

1. Step 1: Set 'City' as the index

   Setting 'City' as index allows direct row selection by city name.

2. Step 2: Use loc to select rows labeled 'Paris'

   With 'City' as index, df.loc['Paris'] returns all rows for Paris efficiently.

3. Final Answer:

   df.set_index('City', inplace=True)\ndf.loc['Paris'] -> Option A

4. Quick Check:

   Index by City, then loc['Paris'] selects rows [OK]

Hint: Set index to City, then use loc['Paris'] to select [OK]

Common Mistakes:

• Setting wrong column as index
• Using loc with non-index column
• Not setting index before loc selection