Creating new columns in Pandas - Performance & Efficiency

Time Complexity: Creating new columns

O(n)

Understanding Time Complexity

When we add new columns to a pandas DataFrame, it takes some time to process. We want to understand how this time changes as the DataFrame gets bigger.

The question is: How does the time to create new columns grow when the number of rows increases?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

n = 10  # Example value for n

df = pd.DataFrame({
    'A': range(n),
    'B': range(n, 2*n)
})

df['C'] = df['A'] + df['B']

This code creates a DataFrame with two columns and then adds a new column 'C' by adding columns 'A' and 'B' element-wise.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Adding values from columns 'A' and 'B' for each row.
• How many times: Once for each row in the DataFrame (n times).

How Execution Grows With Input

As the number of rows grows, the time to add the new column grows roughly in direct proportion.

Input Size (n)	Approx. Operations
10	10 additions
100	100 additions
1000	1000 additions

Pattern observation: Doubling the rows doubles the work needed to create the new column.

Final Time Complexity

Time Complexity: O(n)

This means the time to create a new column grows linearly with the number of rows in the DataFrame.

Common Mistake

[X] Wrong: "Creating a new column is instant and does not depend on DataFrame size."

[OK] Correct: Each row must be processed to compute the new column values, so time grows with the number of rows.

Interview Connect

Understanding how data size affects operations like adding columns helps you write efficient code and explain your choices clearly in real projects.

Self-Check

"What if we create the new column using a constant value instead of adding two columns? How would the time complexity change?"

Practice

1. What is the correct way to add a new column named age + 10 to a DataFrame df that adds 10 to the existing age column?

easy

A. df['age + 10'] == df['age'] + 10

B. df['age + 10'] = df['age'] + 10

C. df.add_column('age + 10', df['age'] + 10)

D. df.age + 10 = df.age + 10

Solution

1. Step 1: Understand how to create a new column in pandas

   In pandas, new columns are created by assigning a value to df['new_column_name'].

2. Step 2: Apply the addition operation to the existing column

   Adding 10 to the age column is done by df['age'] + 10. Assigning this to df['age + 10'] creates the new column.

3. Final Answer:

   df['age + 10'] = df['age'] + 10 -> Option B

4. Quick Check:

   New column assignment = df['age + 10'] = df['age'] + 10 [OK]

Hint: Use df['new_col'] = expression to add columns [OK]

Common Mistakes:

• Using double equals (==) instead of single equals (=) for assignment
• Trying to use a method like add_column which doesn't exist
• Using dot notation for new columns with spaces or special characters

2. Which of the following is the correct syntax to create a new column full_name by combining first_name and last_name columns with a space in between in DataFrame df?

easy

A. df['full_name'] = df['first_name'] + ' ' + df['last_name']

B. df['full_name'] = df['first_name'].append(' ').append(df['last_name'])

C. df['full_name'] = df['first_name'].join(' ', df['last_name'])

D. df['full_name'] = df['first_name'] & ' ' & df['last_name']

Solution

1. Step 1: Combine string columns with + operator

   In pandas, string columns can be combined using the + operator. To add a space, include ' ' between them.

2. Step 2: Assign the combined string to a new column

   Assign the result to df['full_name'] to create the new column.

3. Final Answer:

   df['full_name'] = df['first_name'] + ' ' + df['last_name'] -> Option A

4. Quick Check:

   String concat with + = df['full_name'] = df['first_name'] + ' ' + df['last_name'] [OK]

Hint: Use + to join string columns with spaces [OK]

Common Mistakes:

• Using append or join methods which don't work on pandas Series
• Using & operator which is for bitwise operations
• Forgetting to add space between names

3. Given the DataFrame df below:

   price  quantity
0    100         2
1    150         3
2    200         1

What will be the output of this code?

df['total'] = df['price'] * df['quantity']
df['total']

medium

A. [200, 450, 200]

B. [100, 150, 200]

C. [2, 3, 1]

D. [300, 450, 600]

Solution

1. Step 1: Multiply price and quantity element-wise

   Multiply each row's price by quantity: 100*2=200, 150*3=450, 200*1=200.

2. Step 2: Assign result to new column and output it

   The new column 'total' will have values [200, 450, 200]. Printing df['total'] shows this list.

3. Final Answer:

   [200, 450, 200] -> Option A

4. Quick Check:

   Element-wise multiply = [200, 450, 200] [OK]

Hint: Multiply columns element-wise for new column [OK]

Common Mistakes:

• Adding instead of multiplying columns
• Confusing row-wise with column-wise operations
• Expecting a scalar instead of a Series output

4. What is wrong with this code that tries to create a new column discounted_price by subtracting 10 from price in DataFrame df?

df.discounted_price = df['price'] - 10

medium

A. It will fail if the column name has spaces or special characters.

B. It will cause a syntax error because of dot notation.

C. It will create a new variable, not a DataFrame column.

D. It will create the column correctly.

Solution

1. Step 1: Understand dot notation for DataFrame columns

   Dot notation df.col = value works to create new columns only if 'col' is a valid Python identifier (no spaces, no special characters).

2. Step 2: Check the column name 'discounted_price'

   'discounted_price' consists of letters and underscores, which is valid. Therefore, the code creates the column correctly.

3. Final Answer:

   It will create the column correctly. -> Option D

4. Quick Check:

   Dot notation works for valid identifiers [OK]

Hint: Use df['col'] if column name has spaces or special chars [OK]

Common Mistakes:

• Thinking dot notation fails for valid column names
• Confusing DataFrame column assignment with local variable creation
• Believing underscores invalidate column names for dot notation

5. You have a DataFrame df with columns score and max_score. You want to create a new column percentage that shows the score as a percentage of max_score, but if max_score is zero, set percentage to None. Which code correctly does this?

hard

A. df['percentage'] = df['score'] / df['max_score'].replace(0, None) * 100

B. df.loc[df['max_score'] == 0, 'percentage'] = None
df['percentage'] = df['score'] / df['max_score'] * 100

C. df['percentage'] = df.apply(lambda row: row['score'] / row['max_score'] * 100 if row['max_score'] != 0 else None, axis=1)

D. df['percentage'] = df['score'] / df['max_score'] * 100
df['percentage'][df['max_score'] == 0] = None

Solution

1. Step 1: Use row-wise function to handle zero max_score

   Using df.apply with a lambda lets us check each row and apply the condition: if max_score is zero, return None; else calculate percentage.

2. Step 2: Assign the result to new column 'percentage'

   The lambda returns the correct value per row, and axis=1 ensures row-wise operation.

3. Final Answer:

   df['percentage'] = df.apply(lambda row: row['score'] / row['max_score'] * 100 if row['max_score'] != 0 else None, axis=1) -> Option C

4. Quick Check:

   Use apply with condition for safe division [OK]

Hint: Use apply with lambda for row-wise conditional columns [OK]

Common Mistakes:

• Replacing zero with None in Series causes errors
• Using chained indexing which raises warnings
• Not handling division by zero properly