P-values and significance in Data Analysis Python - Step-by-Step Execution

Concept Flow - P-values and significance

Start with null hypothesis

↓

Collect sample data

↓

Calculate test statistic

↓

Calculate P-value

↓

Compare P-value to significance level (alpha)

↓

Reject H0

↓

Conclude effect

This flow shows how we start with a hypothesis, collect data, calculate a P-value, and decide if the result is significant by comparing to a threshold.

Execution Sample

Data Analysis Python

import scipy.stats as stats

# Sample data
sample = [5, 7, 8, 6, 9]

# Test if mean equals 6
stat, p = stats.ttest_1samp(sample, 6)
print(f"P-value: {p:.4f}")

This code calculates the P-value for a t-test checking if the sample mean differs from 6.

Execution Table

Step	Action	Calculation	Result
1	Calculate sample mean	mean = (5+7+8+6+9)/5	mean = 7.0
2	Calculate sample std deviation	std ≈ 1.58	std ≈ 1.58
3	Calculate t-statistic	t = (7.0 - 6) / (1.58 / sqrt(5))	t ≈ 1.41
4	Calculate degrees of freedom	df = 5 - 1	df = 4
5	Calculate P-value (two-tailed)	p = 2 * (1 - CDF_t(1.41, 4))	p ≈ 0.23
6	Compare P-value to alpha=0.05	0.23 > 0.05	Fail to reject null hypothesis

💡 P-value 0.23 is greater than significance level 0.05, so we fail to reject the null hypothesis.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	Final
mean	undefined	7.0	7.0	7.0	7.0	7.0	7.0
std	undefined	undefined	1.58	1.58	1.58	1.58	1.58
t	undefined	undefined	undefined	1.41	1.41	1.41	1.41
df	undefined	undefined	undefined	undefined	4	4	4
p	undefined	undefined	undefined	undefined	undefined	0.23	0.23

Key Moments - 3 Insights

Why do we compare the P-value to the significance level (alpha)?

Because the P-value tells us how likely the observed data is if the null hypothesis is true. If it's less than alpha (like 0.05), it means the data is unlikely under the null, so we reject it. See execution_table step 6.

Does a high P-value prove the null hypothesis is true?

No, a high P-value means we do not have strong evidence against the null hypothesis, but it does not prove it is true. See execution_table step 6 where we 'fail to reject' but do not accept the null.

Why do we use a two-tailed test here?

Because we are testing if the mean is different from 6 in either direction (higher or lower). This doubles the tail probability in the P-value calculation. See execution_table step 5.

Visual Quiz - 3 Questions

Look at the execution table, what is the calculated t-statistic at step 3?

A 1.41

B 0.05

C 4

D 7.0

💡 Hint

Check the 'Result' column at step 3 in the execution_table.

At which step does the P-value get compared to the significance level?

A Step 4

B Step 5

C Step 6

D Step 2

💡 Hint

Look for the step mentioning comparison to alpha=0.05 in the execution_table.

If the sample mean was 8 instead of 7, how would the P-value change?

A It would increase

B It would decrease

C It would stay the same

D It would become zero

💡 Hint

Higher difference from null mean increases t-statistic, lowering P-value (see variable_tracker for mean and p).

Concept Snapshot

P-values measure how likely data is if the null hypothesis is true.
Calculate test statistic (like t), then P-value.
Compare P-value to significance level (alpha, e.g., 0.05).
If P-value < alpha, reject null hypothesis (significant).
If P-value >= alpha, fail to reject null (not significant).
Two-tailed tests check for difference in both directions.

Full Transcript

We start with a null hypothesis that the mean equals a value (6 here). We collect sample data and calculate the sample mean and standard deviation. Using these, we compute the t-statistic, which measures how far the sample mean is from the null mean in units of standard error. We find the degrees of freedom (sample size minus one). Then, we calculate the P-value, which tells us the probability of seeing a t-statistic as extreme as ours if the null hypothesis is true. We compare this P-value to a chosen significance level (alpha), usually 0.05. If the P-value is less than alpha, we reject the null hypothesis, concluding the sample mean is significantly different. If not, we fail to reject the null, meaning we do not have strong evidence against it. This process helps us decide if our data shows a meaningful effect or difference.

Practice

1. What does a p-value less than 0.05 usually indicate in a hypothesis test?

easy

A. The results are statistically significant and unlikely due to chance.

B. The results are definitely caused by random chance.

C. The sample size is too small to draw conclusions.

D. The test was performed incorrectly.

Solution

1. Step 1: Understand the meaning of p-value

   A p-value measures how likely the observed data would occur if the null hypothesis were true (random chance).

2. Step 2: Compare p-value to significance level

   If the p-value is less than 0.05 (common threshold), it means the observed result is unlikely due to chance, so we reject the null hypothesis.

3. Final Answer:

   The results are statistically significant and unlikely due to chance. -> Option A

4. Quick Check:

   p-value < 0.05 means significant [OK]

Hint: p-value < 0.05 means strong evidence against chance [OK]

Common Mistakes:

• Thinking p-value shows probability null is true
• Confusing significance level with p-value
• Assuming p-value proves practical importance

2. Which of the following is the correct way to check if a p-value stored in variable p_val is significant at 0.05 level in Python?

easy

A. if p_val == 0.05: print('Significant')

B. if p_val > 0.05: print('Significant')

C. if p_val < 0.05: print('Significant')

D. if p_val <= 0.5: print('Significant')

Solution

1. Step 1: Recall significance condition

   A p-value less than 0.05 means significance, so we check if p_val < 0.05.

2. Step 2: Match condition to code

   if p_val < 0.05: print('Significant') uses < 0.05 correctly. Others check wrong conditions or wrong thresholds.

3. Final Answer:

   if p_val < 0.05: print('Significant') -> Option C

4. Quick Check:

   Use < 0.05 for significance [OK]

Hint: Check if p_val < 0.05 for significance [OK]

Common Mistakes:

• Using > instead of < for significance
• Checking equality to 0.05 exactly
• Using wrong threshold like 0.5

3. Given this Python code snippet, what will be printed?

p_val = 0.07
if p_val < 0.05:
    print('Reject null hypothesis')
else:
    print('Fail to reject null hypothesis')

medium

A. Reject null hypothesis

B. SyntaxError

C. No output

D. Fail to reject null hypothesis

Solution

1. Step 1: Check the p-value against 0.05

   The p_val is 0.07, which is greater than 0.05, so the condition p_val < 0.05 is False.

2. Step 2: Determine which branch runs

   Since the condition is False, the else block runs, printing 'Fail to reject null hypothesis'.

3. Final Answer:

   Fail to reject null hypothesis -> Option D

4. Quick Check:

   p_val = 0.07 > 0.05 means fail to reject [OK]

Hint: p_val > 0.05 means fail to reject null [OK]

Common Mistakes:

• Confusing greater than with less than
• Assuming equality means reject
• Ignoring else branch

4. This code is intended to print 'Significant' if p-value is less than 0.05, but it always prints 'Not Significant'. What is the error?

p_val = 0.03
if p_val > 0.05:
    print('Significant')
else:
    print('Not Significant')

medium

A. The comparison operator should be '<' instead of '>'

B. The variable p_val is not defined

C. The print statements are reversed

D. The p_val value is incorrect

Solution

1. Step 1: Analyze the condition logic

   The code checks if p_val > 0.05 to print 'Significant', but significance means p_val < 0.05.

2. Step 2: Identify the fix

   Changing the operator to < will correctly print 'Significant' when p_val is 0.03.

3. Final Answer:

   The comparison operator should be '<' instead of '>' -> Option A

4. Quick Check:

   Use < 0.05 for significance check [OK]

Hint: Use '<' to check p-value significance [OK]

Common Mistakes:

• Using '>' instead of '<' for significance
• Swapping print statements without fixing condition
• Assuming variable is undefined

5. You run a test and get a p-value of 0.049. You want to be very strict and use a significance level of 0.01. What should you conclude?

hard

A. Reject the null hypothesis because 0.049 < 0.05

B. Fail to reject the null hypothesis because 0.049 > 0.01

C. Reject the null hypothesis because 0.049 < 0.01

D. Fail to reject the null hypothesis because 0.049 > 0.05

Solution

1. Step 1: Understand significance level

   The significance level is set to 0.01, meaning p-values must be less than 0.01 to reject the null hypothesis.

2. Step 2: Compare p-value to significance level

   The p-value 0.049 is greater than 0.01, so we fail to reject the null hypothesis at this stricter level.

3. Final Answer:

   Fail to reject the null hypothesis because 0.049 > 0.01 -> Option B

4. Quick Check:

   p-value > significance level means fail to reject [OK]

Hint: Compare p-value to chosen significance level carefully [OK]

Common Mistakes:

• Using default 0.05 without checking chosen level
• Confusing less than and greater than
• Ignoring stricter significance level