Descriptive statistics (describe) in SciPy - Time & Space Complexity

Time Complexity: Descriptive statistics (describe)

O(n)

Understanding Time Complexity

We want to understand how the time needed to get descriptive statistics changes as the data size grows.

How does the work increase when we have more numbers to summarize?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

from scipy import stats
import numpy as np

data = np.random.rand(1000)
result = stats.describe(data)
print(result)
    

This code calculates summary statistics like mean, variance, min, max, and percentiles for a list of numbers.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Scanning through the data array to compute statistics.
• How many times: Each element is visited a few times to calculate different statistics.

How Execution Grows With Input

As the number of data points increases, the time to compute statistics grows roughly in direct proportion.

Input Size (n)	Approx. Operations
10	About 10 to 50 operations
100	About 100 to 500 operations
1000	About 1000 to 5000 operations

Pattern observation: The work grows roughly linearly with the number of data points.

Final Time Complexity

Time Complexity: O(n)

This means the time to get descriptive statistics grows directly with the size of the data.

Common Mistake

[X] Wrong: "Calculating descriptive statistics takes the same time no matter how much data there is."

[OK] Correct: The function must look at each number to compute summaries, so more data means more work.

Interview Connect

Understanding how summary calculations scale helps you explain efficiency when working with large datasets.

Self-Check

"What if we used a streaming method that updates statistics without storing all data? How would the time complexity change?"

Practice

1. What does the scipy.stats.describe function provide when applied to a dataset?

easy

A. Summary statistics like count, mean, variance, min, max, skewness, and kurtosis

B. Only the mean and median of the dataset

C. A plot showing data distribution

D. A list of unique values in the dataset

Solution

1. Step 1: Understand the purpose of scipy.stats.describe

   This function is designed to give a quick summary of key statistics about the data.

2. Step 2: Identify the statistics returned

   It returns count, mean, variance, minimum, maximum, skewness, and kurtosis, which describe data shape and spread.

3. Final Answer:

   Summary statistics like count, mean, variance, min, max, skewness, and kurtosis -> Option A

4. Quick Check:

   Descriptive stats = count, mean, variance, min, max, skewness, kurtosis [OK]

Hint: Remember: describe summarizes data with key stats [OK]

Common Mistakes:

• Thinking it returns only mean and median
• Confusing it with plotting functions
• Expecting a list of unique values

2. Which of the following is the correct way to import and use describe from scipy.stats?

easy

A. from scipy.stats import describe; result = describe(data)

B. import scipy.stats; result = describe(data)

C. from scipy import describe; result = describe(data)

D. import describe from scipy.stats; result = describe(data)

Solution

1. Step 1: Check correct import syntax for a function

   Functions are imported using from module import function syntax in Python.

2. Step 2: Verify usage of describe after import

   After importing, call describe(data) directly.

3. Final Answer:

   from scipy.stats import describe; result = describe(data) -> Option A

4. Quick Check:

   Correct import and call = from scipy.stats import describe; result = describe(data) [OK]

Hint: Use 'from scipy.stats import describe' to call directly [OK]

Common Mistakes:

• Using wrong import syntax like 'import describe from ...'
• Importing scipy.stats but not calling with module prefix
• Importing from wrong module

3. Given the code below, what will be the output for the mean and min values?

import numpy as np
from scipy.stats import describe

data = np.array([2, 4, 6, 8, 10])
result = describe(data)
print(result.mean, result.minmax[0])

medium

A. 6.0 0

B. 5.0 2

C. 6.0 2

D. 5.0 0

Solution

1. Step 1: Calculate the mean of the data array

   The data is [2, 4, 6, 8, 10]. Mean = (2+4+6+8+10)/5 = 30/5 = 6.0.

2. Step 2: Identify the minimum value from minmax

   minmax returns (min, max). The minimum is 2.

3. Final Answer:

   6.0 2 -> Option C

4. Quick Check:

   Mean = 6.0, Min = 2 [OK]

Hint: Mean is average; min is smallest value [OK]

Common Mistakes:

• Calculating mean incorrectly as 5.0
• Confusing minmax tuple order
• Assuming min is 0 without checking data

4. What is wrong with the following code snippet?

from scipy.stats import describe
data = [1, 2, 3, 4, 5]
result = describe(data, axis=1)
print(result)

medium

A. The print statement is missing parentheses

B. The data list should be converted to a numpy array first

C. The describe function requires a second argument for axis

D. Axis=1 is invalid for 1D data, causing an error

Solution

1. Step 1: Understand the data shape and axis parameter

   The data is a 1D list. Axis=1 means operate along columns of 2D or higher arrays.

2. Step 2: Identify the error caused by axis=1 on 1D data

   Using axis=1 on 1D data causes an AxisError because axis 1 does not exist.

3. Final Answer:

   Axis=1 is invalid for 1D data, causing an error -> Option D

4. Quick Check:

   Axis must exist in data shape [OK]

Hint: Check data dimensions before setting axis [OK]

Common Mistakes:

• Assuming axis=1 works for 1D data
• Thinking data must be numpy array for describe
• Ignoring Python 3 print syntax

5. You have a dataset with some zero and negative values: data = [0, -1, 2, 3, 4]. Using scipy.stats.describe, which statistic helps you understand if the data distribution is skewed?

hard

A. Variance

B. Skewness

C. Count

D. Minimum

Solution

1. Step 1: Recall what skewness measures

   Skewness shows if data is symmetric or leans left/right (skewed).

2. Step 2: Identify which statistic indicates skewness

   Among the options, only skewness directly measures data asymmetry.

3. Final Answer:

   Skewness -> Option B

4. Quick Check:

   Skewness = data asymmetry [OK]

Hint: Skewness shows data leaning direction [OK]

Common Mistakes:

• Confusing variance with skewness
• Thinking count or minimum show skewness
• Ignoring negative and zero values impact