Descriptive statistics review in Data Analysis Python - Time & Space Complexity

Time Complexity: Descriptive statistics review

O(n)

Understanding Time Complexity

We want to understand how the time to calculate descriptive statistics changes as the data size grows.

How does the work increase when we have more data points?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

def describe_data(df):
    mean_val = df['value'].mean()
    median_val = df['value'].median()
    std_val = df['value'].std()
    count_val = df['value'].count()
    return mean_val, median_val, std_val, count_val

This code calculates basic descriptive statistics (mean, median, standard deviation, count) on one column of a data table.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Scanning through the column values to compute each statistic.
• How many times: Each statistic requires going through the data once or a few times.

How Execution Grows With Input

As the number of data points increases, the time to compute each statistic grows roughly in direct proportion.

Input Size (n)	Approx. Operations
10	About 10 operations per statistic
100	About 100 operations per statistic
1000	About 1000 operations per statistic

Pattern observation: The work grows linearly with the number of data points.

Final Time Complexity

Time Complexity: O(n)

This means the time to calculate descriptive statistics grows directly with the size of the data.

Common Mistake

[X] Wrong: "Calculating mean and median takes the same time regardless of data size."

[OK] Correct: Both mean and median require looking at the data values, so more data means more work.

Interview Connect

Understanding how descriptive statistics scale helps you explain data processing steps clearly and shows you can think about efficiency in real tasks.

Self-Check

"What if we added sorting to find the median? How would the time complexity change?"

Practice

1. Which of the following best describes the median in a dataset?

easy

A. The middle value when data is sorted from smallest to largest

B. The most frequently occurring value in the data

C. The average of all values in the data

D. The difference between the largest and smallest values

Solution

1. Step 1: Understand what median means

   The median is the value that lies in the middle of a sorted list of numbers.

2. Step 2: Compare with other options

   Mode is the most frequent value, mean is the average, and range is max-min.

3. Final Answer:

   The middle value when data is sorted from smallest to largest -> Option A

4. Quick Check:

   Median = middle value [OK]

Hint: Median splits sorted data in half [OK]

Common Mistakes:

• Confusing median with mean
• Thinking median is the most frequent value
• Mixing median with range

2. Which of the following is the correct way to get descriptive statistics of a pandas DataFrame df?

easy

A. describe(df)

B. df.summary()

C. df.describe()

D. df.stats()

Solution

1. Step 1: Recall pandas method for summary

   In pandas, describe() is the method to get descriptive stats.

2. Step 2: Check other options

   Methods like summary(), stats() do not exist in pandas, and describe(df) is not valid syntax.

3. Final Answer:

   df.describe() -> Option C

4. Quick Check:

   Use df.describe() for stats [OK]

Hint: Use df.describe() for quick stats [OK]

Common Mistakes:

• Using non-existent methods like summary()
• Calling describe as a function with df argument
• Confusing with other libraries

3. Given the following code, what will be the output of print(df.mean())?

import pandas as pd
data = {'A': [1, 2, 3], 'B': [4, 5, 6]}
df = pd.DataFrame(data)

medium

A. A 2.0
B 5.0
dtype: float64

B. A 3.0
B 6.0
dtype: int64

C. A 1.5
B 4.5
dtype: float64

D. Error: mean() not defined

Solution

1. Step 1: Calculate mean for each column

   Column A values: 1, 2, 3; mean = (1+2+3)/3 = 2.0. Column B values: 4, 5, 6; mean = (4+5+6)/3 = 5.0.

2. Step 2: Understand pandas output format

   Printing df.mean() shows means per column with dtype float64.

3. Final Answer:

   A 2.0 B 5.0 dtype: float64 -> Option A

4. Quick Check:

   Mean = sum/number of values [OK]

Hint: Mean = sum of values divided by count [OK]

Common Mistakes:

• Calculating median instead of mean
• Mixing up column values
• Expecting integer dtype instead of float

4. The following code is intended to calculate the standard deviation of column 'X' in DataFrame df. What is the error?

import pandas as pd
data = {'X': [10, 20, 30, 40]}
df = pd.DataFrame(data)
std_dev = df.std('X')
print(std_dev)

medium

A. DataFrame std() method requires axis=1 argument

B. The std() method does not take any arguments; column must be selected first

C. The code should use df['X'].variance() instead

D. No error; code runs correctly

Solution

1. Step 1: Check std() method usage

   In pandas, std() calculates standard deviation for all numeric columns or Series. It does not accept a column name as argument.

2. Step 2: Correct way to get std of column 'X'

   We must select the column first: df['X'].std().

3. Final Answer:

   The std() method does not take any arguments; column must be selected first -> Option B

4. Quick Check:

   Use df['X'].std() to get column std [OK]

Hint: Select column before std() call [OK]

Common Mistakes:

• Passing column name as argument to std()
• Using variance() instead of std()
• Assuming std() needs axis=1

5. You have a DataFrame df with columns 'Age' and 'Salary'. You want to find the average salary for people older than 30. Which code correctly does this?

hard

A. df['Salary'].mean(df['Age'] > 30)

B. df['Salary'].mean().filter(df['Age'] > 30)

C. df.mean()['Salary'][df['Age'] > 30]

D. df[df['Age'] > 30]['Salary'].mean()

Solution

1. Step 1: Filter rows where Age > 30

   Use boolean indexing: df[df['Age'] > 30] selects rows with Age greater than 30.

2. Step 2: Select 'Salary' column and calculate mean

   From filtered rows, select 'Salary' column and call mean() to get average salary.

3. Final Answer:

   df[df['Age'] > 30]['Salary'].mean() -> Option D

4. Quick Check:

   Filter then mean for conditional average [OK]

Hint: Filter rows first, then calculate mean [OK]

Common Mistakes:

• Passing condition inside mean()
• Trying to filter after mean()
• Using mean() on entire DataFrame