Hypothesis testing (ttest, chi2test) in MATLAB - Deep Dive

Overview - Hypothesis testing (ttest, chi2test)

What is it?

Hypothesis testing is a way to check if a guess about data is likely true or not. It uses math to decide if differences or patterns in data happened by chance or because of something real. Two common tests are the t-test, which compares averages, and the chi-square test, which checks relationships between categories. These tests help us make decisions based on data, not just guesses.

Why it matters

Without hypothesis testing, we would rely on guesses or gut feelings when looking at data. This could lead to wrong conclusions, like thinking a medicine works when it doesn't or missing important patterns in data. Hypothesis testing gives a clear, fair way to decide if results are meaningful, helping in science, business, and everyday decisions.

Where it fits

Before learning hypothesis testing, you should understand basic statistics like mean, variance, and probability. After this, you can learn about confidence intervals, regression analysis, and advanced statistical modeling. Hypothesis testing is a key step in the journey from raw data to making informed decisions.

Mental Model

Core Idea

Hypothesis testing is a formal way to decide if data supports a specific claim or if observed results could just be random chance.

Think of it like...

It's like a courtroom trial where the hypothesis is the accused, and the data is the evidence; the test decides if there is enough proof to reject the claim or not.

┌─────────────────────────────┐
│      Start with a claim     │
│   (Null Hypothesis H0)      │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Collect data and calculate   │
│   a test statistic (t or χ²) │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Compare statistic to         │
│ critical value or p-value   │
└─────────────┬───────────────┘
              │
      ┌───────┴────────┐
      │                │
      ▼                ▼
Reject H0          Fail to reject H0
(Effect likely)    (No strong evidence)

Build-Up - 7 Steps

1

Foundation

Understanding Null and Alternative Hypotheses

Concept: Learn what the null hypothesis (H0) and alternative hypothesis (H1) mean in testing.

The null hypothesis (H0) is a statement that there is no effect or difference. For example, 'the average height of group A equals group B.' The alternative hypothesis (H1) says there is an effect or difference, like 'the average height of group A is different from group B.' Hypothesis testing checks if data supports rejecting H0 in favor of H1.

Result

You can clearly state what you are testing and what it means to reject or not reject the null hypothesis.

Understanding the roles of H0 and H1 is essential because all hypothesis tests revolve around deciding whether to reject H0 based on data.

2

Foundation

Collecting Data and Calculating Test Statistics

Concept: Learn how to summarize data into a single number that measures evidence against H0.

After collecting data, we calculate a test statistic. For a t-test, this measures how far the sample mean is from the hypothesized mean, scaled by variability and sample size. For a chi-square test, it measures how observed counts differ from expected counts under H0. These statistics help us decide if differences are big enough to reject H0.

Result

You get a number (test statistic) that summarizes how unusual your data is if H0 were true.

Knowing how to calculate and interpret test statistics is key to connecting raw data to the decision-making process.

3

Intermediate

Using the t-Test for Comparing Means

🤔Before reading on: do you think the t-test can be used for comparing more than two groups? Commit to your answer.

Concept: The t-test compares the average values of two groups to see if they differ significantly.

In MATLAB, you can use the function ttest to check if the mean of one sample differs from a known value, or ttest2 to compare means of two samples. The test calculates a t-statistic and a p-value. If the p-value is small (usually less than 0.05), we reject H0 and say the means are different.

Result

You can decide if two groups have statistically different averages using MATLAB commands like [h,p] = ttest2(sample1, sample2).

Understanding the t-test lets you analyze differences in averages, a common question in many fields like medicine and business.

4

Intermediate

Applying the Chi-Square Test for Categorical Data

🤔Before reading on: do you think the chi-square test works with numerical averages or category counts? Commit to your answer.

Concept: The chi-square test checks if observed counts in categories differ from expected counts under the null hypothesis.

In MATLAB, the function chi2gof tests if observed frequencies fit a distribution, and the function crosstab can be used to create contingency tables for chi-square tests of independence. The test calculates a chi-square statistic and p-value. A small p-value means the observed counts are unlikely under H0, so we reject it.

Result

You can test if categories are related or if data fits a pattern using MATLAB's chi-square functions.

Knowing how to use chi-square tests helps analyze relationships in categorical data, like survey responses or genetics.

5

Intermediate

Interpreting p-Values and Significance Levels

🤔Before reading on: does a p-value of 0.03 mean the null hypothesis is definitely false? Commit to your answer.

Concept: Learn what p-values mean and how to use significance levels to make decisions.

A p-value is the chance of seeing data as extreme as yours if H0 is true. A small p-value (like <0.05) suggests data is unlikely under H0, so we reject it. But p-values do not prove H0 false; they measure evidence strength. The significance level (alpha) is a threshold you set before testing.

Result

You can correctly interpret test results and avoid common mistakes about what p-values mean.

Understanding p-values prevents misinterpretation and helps make sound conclusions from tests.

6

Advanced

Handling Assumptions and Limitations of Tests

🤔Before reading on: do you think t-tests require data to be perfectly normal? Commit to your answer.

Concept: Learn the assumptions behind t-tests and chi-square tests and what happens if they are violated.

The t-test assumes data is roughly normal and samples are independent. The chi-square test assumes expected counts are large enough. If assumptions fail, results may be invalid. MATLAB offers alternative tests like nonparametric tests (ranksum) or Fisher's exact test for small counts.

Result

You know when to trust test results and when to use alternatives.

Recognizing assumptions helps avoid wrong conclusions and choose the right test for your data.

7

Expert

Advanced Use: Multiple Testing and Effect Sizes

🤔Before reading on: does running many hypothesis tests increase the chance of false positives? Commit to your answer.

Concept: Understand the challenges of multiple tests and the importance of measuring effect size beyond p-values.

When testing many hypotheses, the chance of false positives grows. Techniques like Bonferroni correction adjust significance levels to control errors. Also, p-values don't show how big or important an effect is. Reporting effect sizes (like Cohen's d for t-tests) gives more insight. MATLAB users can calculate effect sizes manually or with toolboxes.

Result

You can correctly handle multiple comparisons and report meaningful results in complex studies.

Knowing these advanced topics prevents common pitfalls in research and improves the quality of conclusions.

Under the Hood

Hypothesis tests calculate a test statistic from data, which measures how far the data is from what the null hypothesis predicts. This statistic is compared to a theoretical distribution (t-distribution for t-tests, chi-square distribution for chi-square tests) to find the p-value. The p-value quantifies the probability of observing such data if the null hypothesis is true. Internally, MATLAB uses numerical algorithms to compute these statistics and p-values efficiently.

Why designed this way?

These tests were designed to provide a standardized, objective way to evaluate evidence in data. The t-test was developed by William Gosset to handle small sample sizes with unknown variance. The chi-square test was created by Karl Pearson to analyze categorical data. Both rely on probability distributions to model random variation, allowing consistent decision rules. Alternatives like Bayesian methods exist but classical tests remain popular for their simplicity and interpretability.

Data ──▶ Calculate Test Statistic ──▶ Compare to Distribution ──▶ Compute p-value ──▶ Decision

┌───────────────┐    ┌─────────────────────┐    ┌─────────────────────┐    ┌───────────────┐
│   Raw Data    │──▶ │ Test Statistic (t,χ²)│──▶│ Theoretical Dist.    │──▶│ p-value       │
└───────────────┘    └─────────────────────┘    └─────────────────────┘    └───────────────┘
                                                                                 │
                                                                                 ▼
                                                                       Reject or Fail to Reject H0

Myth Busters - 4 Common Misconceptions

Quick: Does a p-value below 0.05 prove the alternative hypothesis is true? Commit to yes or no.

Common Belief: A p-value less than 0.05 means the alternative hypothesis is definitely true.

Reality: A small p-value only shows data is unlikely under the null hypothesis; it does not prove the alternative is true with certainty.

Why it matters: Believing this leads to overconfidence and ignoring uncertainty, causing poor decisions or false claims.

Quick: Can the t-test be used safely on any data regardless of distribution? Commit to yes or no.

Common Belief: The t-test works well no matter the data distribution.

Reality: The t-test assumes data is approximately normal; if this is violated, results may be invalid or misleading.

Why it matters: Ignoring assumptions can cause wrong conclusions, especially with small samples or skewed data.

Quick: Does a non-significant test result mean the null hypothesis is true? Commit to yes or no.

Common Belief: Failing to reject the null hypothesis means it is true.

Reality: Failing to reject H0 means there is not enough evidence against it, but it does not prove H0 is true.

Why it matters: Misinterpreting this can cause missed discoveries or false confidence in no effect.

Quick: Does running many hypothesis tests increase the chance of false positives? Commit to yes or no.

Common Belief: Running multiple tests does not affect the chance of false positives.

Reality: Multiple tests increase the chance of false positives unless corrections are applied.

Why it matters: Ignoring this leads to many false discoveries, wasting time and resources.

Expert Zone

1

Effect sizes are crucial to report alongside p-values to understand practical significance, not just statistical significance.

2

The choice of one-tailed vs two-tailed tests affects interpretation and must be decided before data collection to avoid bias.

3

In small samples, exact tests or bootstrap methods can provide more reliable results than classical asymptotic tests.

When NOT to use

Hypothesis testing is not ideal when data is heavily non-normal with small samples, or when prior knowledge is strong; Bayesian methods or resampling techniques may be better. Also, for very large datasets, tiny differences can be statistically significant but meaningless, so focus on effect sizes and practical relevance.

Production Patterns

In real-world data science, hypothesis tests are often part of automated pipelines for A/B testing, quality control, or clinical trials. Professionals combine tests with visualization and effect size reporting. Multiple testing corrections are standard in genomics and marketing analytics. MATLAB scripts integrate tests with data preprocessing and reporting for reproducible workflows.

Connections

Confidence Intervals

Builds-on

Confidence intervals provide a range of plausible values for parameters, complementing hypothesis tests by showing effect size and uncertainty.

Bayesian Inference

Alternative approach

Bayesian methods update beliefs with data and provide probability statements about hypotheses, offering a different perspective than classical hypothesis testing.

Legal Trials

Analogous decision process

Both hypothesis testing and legal trials weigh evidence to decide on a claim, balancing risks of false positives and negatives.

Common Pitfalls

#1 Misinterpreting p-values as the probability that the null hypothesis is true.

Wrong approach:

[h,p] = ttest(sample); if p < 0.05, disp('Null hypothesis is false');

Correct approach:

[h,p] = ttest(sample); if p < 0.05, disp('Reject null hypothesis due to low p-value'); else disp('Insufficient evidence to reject null');

Root cause:

Confusing p-value definition with direct probability of hypotheses.

#2 Using t-test on heavily skewed data without checking assumptions.

Wrong approach:

[h,p] = ttest(sample); % without checking normality

Correct approach:

if kstest(sample) == 0, [h,p] = ttest(sample); else disp('Use nonparametric test'); end

Root cause:

Ignoring test assumptions leads to invalid results.

#3 Running multiple t-tests without adjusting significance level.

Wrong approach:

for i=1:10, [h,p] = ttest(data(:,i)); disp(p); end

Correct approach:

alpha = 0.05/10; for i=1:10, [h,p] = ttest(data(:,i)); if p < alpha, disp('Significant'); else disp('Not significant'); end; end

Root cause:

Not controlling for multiple comparisons inflates false positive rate.

Key Takeaways

Hypothesis testing helps decide if data supports a claim or if results could be random chance.

The t-test compares means of groups, while the chi-square test analyzes categorical data relationships.

P-values measure evidence against the null hypothesis but do not prove it false.

Test assumptions must be checked to avoid wrong conclusions; alternatives exist for non-ideal data.

Advanced topics like multiple testing corrections and effect sizes improve the reliability and meaning of results.

Practice

1. What does the MATLAB function ttest primarily test?

easy

A. Whether the data follows a normal distribution

B. Whether two samples have the same variance

C. Whether the mean of a sample differs from a specified value

D. Whether the counts in categories match expected counts

Solution

1. Step 1: Understand the purpose of ttest

   The ttest function in MATLAB tests if the mean of a sample is equal to a given value.

2. Step 2: Compare with other tests

   chi2gof tests counts, not means; variance tests are different; normality tests are separate.

3. Final Answer:

   Whether the mean of a sample differs from a specified value -> Option C

4. Quick Check:

   ttest tests mean difference [OK]

Hint: Remember: ttest checks means, chi2 tests counts [OK]

Common Mistakes:

• Confusing ttest with chi-square test
• Thinking ttest checks variance
• Assuming ttest checks distribution shape

2. Which of the following is the correct syntax to perform a one-sample t-test in MATLAB on data vector X against mean 0?

easy

A. chi2gof(X, 0)

B. ttest(X, 0)

C. ttest(0, X)

D. ttest(X)

Solution

1. Step 1: Recall ttest syntax

   The syntax ttest(X, mu) tests if mean of X equals mu.

2. Step 2: Check options

   chi2gof(X, 0) uses wrong function; ttest(0, X) reverses arguments; ttest(X) misses mean value.

3. Final Answer:

   ttest(X, 0) -> Option B

4. Quick Check:

   Correct syntax is ttest(data, mean) [OK]

Hint: Syntax: ttest(data, mean) for one-sample test [OK]

Common Mistakes:

• Swapping arguments order
• Using chi2gof for mean testing
• Omitting mean parameter

3. What is the output of the following MATLAB code?

X = [5.1, 5.3, 5.0, 5.2, 5.4];
[h, p] = ttest(X, 5.0);
disp(h); disp(p);

medium

A. h = 1, p < 0.05 (reject mean = 5.0)

B. h = 1, p > 0.05 (fail to reject mean = 5.0)

C. h = 0, p < 0.05 (reject mean = 5.0)

D. h = 0, p > 0.05 (fail to reject mean = 5.0)

Solution

1. Step 1: Compute sample statistics

   X mean = (5.1 + 5.3 + 5.0 + 5.2 + 5.4)/5 = 5.2; std ≈ 0.158.

2. Step 2: Interpret ttest output

   t ≈ (5.2 - 5.0) / (0.158 / √5) ≈ 2.83, df=4, p ≈ 0.047 < 0.05, so h=1 (reject null).

3. Final Answer:

   h = 1, p < 0.05 (reject mean = 5.0) -> Option A

4. Quick Check:

   mean=5.2, t≈2.83 > 2.776, p<0.05 [OK]

Hint: Sample mean 5.2 sufficiently differs from 5.0 (t>crit) -> h=1, p<0.05 [OK]

Common Mistakes:

• Assuming data too close to 5.0 for significance (h=0)
• Miscalculating p-value as >0.05
• Confusing h=1 (reject) meaning

4. Identify the error in this MATLAB code snippet for a chi-square goodness-of-fit test:

observed = [20, 30, 50];
expected = [25, 25, 50];
h = chi2gof(observed, 'ExpectedCounts', expected);

medium

A. The 'Expected' parameter is misspelled

B. The vectors observed and expected must be the same length

C. chi2gof cannot be used with expected counts

D. The observed data should be a vector of counts, but chi2gof expects raw data samples

Solution

1. Step 1: Understand chi2gof input requirements

   chi2gof expects raw sample data, not observed counts directly.

2. Step 2: Check usage of 'ExpectedCounts'

   The 'ExpectedCounts' parameter is valid but only used with raw data; providing counts directly leads to incorrect results.

3. Final Answer:

   The observed data should be a vector of counts, but chi2gof expects raw data samples -> Option D

4. Quick Check:

   chi2gof needs raw data, not counts [OK]

Hint: chi2gof needs raw data, not counts vector [OK]

Common Mistakes:

• Passing counts instead of raw data
• Misspelling parameter names
• Assuming chi2gof accepts counts directly

5. You have observed counts obs = [40, 35, 25] and expected proportions expProp = [0.3, 0.4, 0.3]. How do you correctly perform a chi-square goodness-of-fit test in MATLAB to check if observed counts fit expected proportions?

hard

A. Convert expected proportions to expected counts, then use chi2gof with raw data generated from counts

B. Directly use chi2gof(obs, 'Expected', expProp)

C. Use ttest(obs, expProp) to compare counts and proportions

D. Use chi2gof with observed counts and expected counts vectors

Solution

1. Step 1: Calculate expected counts from proportions

   Multiply total observed count by expected proportions: total = 40+35+25=100; expected counts = 100 * expProp = [30, 40, 30].

2. Step 2: Generate raw data samples for chi2gof

   Since chi2gof needs raw data, create a vector with repeated category labels matching observed counts.

3. Step 3: Perform chi-square test

   Use chi2gof with raw data and specify expected counts via 'ExpectedCounts' parameter.

4. Final Answer:

   Convert expected proportions to expected counts, then use chi2gof with raw data generated from counts -> Option A

5. Quick Check:

   chi2gof needs raw data and expected counts [OK]

Hint: Convert proportions to counts, create raw data for chi2gof [OK]

Common Mistakes:

• Passing proportions directly as expected counts
• Using ttest for count data
• Not converting counts to raw data samples