Creating DataFrames (dict, list, CSV) in Data Analysis Python - Performance & Efficiency

Time Complexity: Creating DataFrames (dict, list, CSV)

O(n)

Understanding Time Complexity

When we create DataFrames from different data sources, the time it takes depends on the input size and format.

We want to know how the time grows as the data gets bigger.

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

n = 10  # example value for n
data_dict = {"A": list(range(n)), "B": list(range(n))}
df_from_dict = pd.DataFrame(data_dict)

list_of_lists = [[i, i*2] for i in range(n)]
df_from_list = pd.DataFrame(list_of_lists, columns=["A", "B"])

# Assume CSV file with n rows
# df_from_csv = pd.read_csv('data.csv')  # reading CSV into DataFrame

This code creates DataFrames from a dictionary, a list of lists, and mentions reading from a CSV file.

Identify Repeating Operations

• Primary operation: Iterating over n rows to build DataFrame structure.
• How many times: Each element in the input data is processed once.

How Execution Grows With Input

As the number of rows n increases, the time to create the DataFrame grows roughly in direct proportion.

Input Size (n)	Approx. Operations
10	About 10 operations
100	About 100 operations
1000	About 1000 operations

Pattern observation: Doubling the input roughly doubles the work done.

Final Time Complexity

Time Complexity: O(n)

This means the time to create a DataFrame grows linearly with the number of rows.

Common Mistake

[X] Wrong: "Creating a DataFrame from a dictionary or list is instant and does not depend on data size."

[OK] Correct: Even though it feels fast for small data, the process reads each row once, so time grows with data size.

Interview Connect

Understanding how data loading time grows helps you explain performance in real projects and shows you think about efficiency.

Self-Check

"What if we read a CSV file with many columns instead of rows? How would the time complexity change?"

Practice

1. What is the main purpose of creating a DataFrame in data analysis?

easy

A. To organize data in a table format for easy analysis

B. To write data to a text file

C. To create a list of numbers

D. To perform calculations without storing data

Solution

1. Step 1: Understand what a DataFrame is

   A DataFrame is like a table with rows and columns, used to organize data clearly.

2. Step 2: Identify the purpose of DataFrames

   They help analyze data easily by structuring it in rows and columns.

3. Final Answer:

   To organize data in a table format for easy analysis -> Option A

4. Quick Check:

   DataFrame = Table for data analysis [OK]

Hint: DataFrames = tables for data, not just lists or files [OK]

Common Mistakes:

• Confusing DataFrame with a simple list
• Thinking DataFrame is for saving files only
• Believing DataFrame is just for calculations

2. Which of the following is the correct way to create a DataFrame from a dictionary in Python using pandas?

easy

A. pd.DataFrame(name=['Alice', 'Bob'], age=[25, 30])

B. pd.DataFrame({'name': ['Alice', 'Bob'], 'age': [25, 30]})

C. pd.DataFrame({'name': 'Alice', 'age': 25})

D. pd.DataFrame['name', 'age'] = ['Alice', 'Bob'], [25, 30]

Solution

1. Step 1: Recall pandas DataFrame creation syntax

   Use pd.DataFrame() with a dictionary where keys are column names and values are lists of column data.

2. Step 2: Check each option

   pd.DataFrame({'name': ['Alice', 'Bob'], 'age': [25, 30]}) correctly uses a dictionary with lists for columns. Others have syntax errors or wrong formats.

3. Final Answer:

   pd.DataFrame({'name': ['Alice', 'Bob'], 'age': [25, 30]}) -> Option B

4. Quick Check:

   Dictionary with lists = pd.DataFrame() [OK]

Hint: Use dictionary with lists inside pd.DataFrame() [OK]

Common Mistakes:

• Using pd.DataFrame with wrong brackets
• Passing single values instead of lists
• Trying to assign columns after creating DataFrame

3. What will be the output of this code?

import pandas as pd
data = [[1, 'Alice'], [2, 'Bob']]
df = pd.DataFrame(data, columns=['ID', 'Name'])
print(df)

medium

A. A DataFrame with columns 'ID' and 'Name' showing the two rows of data

B. A list of lists printed as is

C. An error because columns parameter is missing

D. A DataFrame with default numeric column names

Solution

1. Step 1: Understand DataFrame creation from list of lists

   Passing a list of lists with columns names creates a DataFrame with those columns and rows.

2. Step 2: Check the print output

   The DataFrame will show two rows with columns 'ID' and 'Name' filled with given data.

3. Final Answer:

   A DataFrame with columns 'ID' and 'Name' showing the two rows of data -> Option A

4. Quick Check:

   List + columns = DataFrame table [OK]

Hint: List + columns param creates labeled DataFrame [OK]

Common Mistakes:

• Expecting a list output instead of DataFrame
• Forgetting columns parameter causes default columns
• Confusing list of lists with dictionary input

4. Identify the error in this code that tries to create a DataFrame from a CSV file:

import pandas as pd
df = pd.read_csv('data.csv', header=2)
print(df.head())

medium

A. read_csv cannot read CSV files

B. The file name must be 'data.txt' not 'data.csv'

C. The header parameter should be 0 to read the first row as column names

D. print(df.head()) is incorrect syntax

Solution

1. Step 1: Understand header parameter in read_csv

   header=2 means pandas treats the third row as column names, which may be wrong if first row has headers.

2. Step 2: Identify correct header usage

   Usually header=0 is used to read the first row as column names. Using 2 can cause wrong column names or data misalignment.

3. Final Answer:

   The header parameter should be 0 to read the first row as column names -> Option C

4. Quick Check:

   header=0 reads first row as columns [OK]

Hint: Use header=0 to read first row as column names [OK]

Common Mistakes:

• Setting header to wrong row number
• Confusing file extensions
• Misunderstanding print(df.head()) usage

5. You have a CSV file with columns 'Name', 'Age', and 'Score'. You want to create a DataFrame but only keep rows where 'Score' is above 80. Which code correctly creates the DataFrame and filters it?

hard

A. import pandas as pd\ndf = pd.read_csv('file.csv', filter='Score > 80')\nprint(df)

B. import pandas as pd\ndf = pd.read_csv('file.csv')\ndf_filtered = df.where('Score' > 80)\nprint(df_filtered)

C. import pandas as pd\ndf = pd.read_csv('file.csv')\ndf_filtered = df.filter('Score > 80')\nprint(df_filtered)

D. import pandas as pd\ndf = pd.read_csv('file.csv')\ndf_filtered = df[df['Score'] > 80]\nprint(df_filtered)

Solution

1. Step 1: Read CSV file into DataFrame

   Use pd.read_csv('file.csv') to load all data.

2. Step 2: Filter rows where 'Score' > 80

   Use boolean indexing: df[df['Score'] > 80] to keep only rows with Score above 80.

3. Final Answer:

   import pandas as pd\ndf = pd.read_csv('file.csv')\ndf_filtered = df[df['Score'] > 80]\nprint(df_filtered) -> Option D

4. Quick Check:

   Boolean indexing filters rows [OK]

Hint: Use df[df['Score'] > 80] to filter rows [OK]

Common Mistakes:

• Trying to filter inside read_csv
• Using df.filter() incorrectly
• Using df.where() without proper condition