Goodness of fit evaluation in SciPy - Deep Dive

Overview - Goodness of fit evaluation

What is it?

Goodness of fit evaluation is a way to check how well a statistical model matches observed data. It helps us see if the model's predictions are close to what actually happened. This is done by comparing the data to what the model expects, using numbers or charts. It is important for making sure our models are useful and reliable.

Why it matters

Without goodness of fit evaluation, we might trust models that do not represent reality well. This can lead to wrong decisions in fields like medicine, business, or science. By measuring fit, we can improve models, choose better ones, and avoid costly mistakes. It makes data science results more trustworthy and actionable.

Where it fits

Before learning goodness of fit, you should understand basic statistics like distributions and hypothesis testing. After this, you can explore model selection, regression diagnostics, and advanced statistical modeling. It fits in the journey after learning how to build models and before refining or comparing them.

Mental Model

Core Idea

Goodness of fit evaluation measures how closely a model's predictions match the actual observed data.

Think of it like...

It's like trying on a pair of shoes to see if they fit your feet comfortably; if they don't fit well, you know you need a different pair.

Observed Data ──▶ Compare ──▶ Model Predictions
       │                          │
       └───────── Goodness of Fit ─────────┘
                 (Measure of closeness)

Build-Up - 6 Steps

1

Foundation

Understanding observed and expected data

Concept: Learn the difference between observed data and what a model expects.

Observed data are the actual values collected from experiments or surveys. Expected data are the values predicted by a model based on assumptions or parameters. Goodness of fit compares these two sets to see how close they are.

Result

You can clearly identify what data you have and what your model predicts.

Understanding the two data types is essential because goodness of fit is about measuring their difference.

2

Foundation

Introduction to chi-square goodness of fit test

Concept: Learn a basic statistical test to measure goodness of fit for categorical data.

The chi-square test compares observed counts in categories to expected counts. It calculates a statistic that shows how much the observed data deviate from the expected. A small value means a good fit; a large value means a poor fit.

Result

You get a chi-square statistic and a p-value to decide if the model fits well.

Knowing this test gives a simple, widely used tool to check model fit for categorical data.

3

Intermediate

Using scipy for chi-square test

🤔Before reading on: do you think scipy can calculate both the chi-square statistic and p-value automatically? Commit to your answer.

Concept: Learn how to use scipy's stats module to perform the chi-square goodness of fit test easily.

You can use scipy.stats.chisquare(observed, expected) to get the chi-square statistic and p-value. This function handles the calculation and returns results you can interpret.

Result

You get numerical output showing how well your model fits the data.

Knowing how to use scipy saves time and reduces errors compared to manual calculations.

4

Intermediate

Interpreting p-values in goodness of fit

🤔Before reading on: does a high p-value mean the model fits well or poorly? Commit to your answer.

Concept: Understand what the p-value tells you about the model fit quality.

The p-value shows the probability of seeing data as extreme as observed if the model is correct. A high p-value (usually above 0.05) means the model fits well enough. A low p-value means the model likely does not fit the data.

Result

You can decide whether to accept or reject the model based on p-value.

Understanding p-values prevents misinterpretation of test results and wrong conclusions.

5

Advanced

Goodness of fit for continuous data with Kolmogorov-Smirnov test

🤔Before reading on: do you think the Kolmogorov-Smirnov test compares distributions or just counts? Commit to your answer.

Concept: Learn a test for checking fit when data are continuous, not categorical.

The Kolmogorov-Smirnov (KS) test compares the observed data distribution to a reference distribution. It measures the largest difference between their cumulative distributions. scipy.stats.kstest can perform this test.

Result

You get a KS statistic and p-value indicating fit quality for continuous data.

Knowing this test expands your toolkit to handle different data types beyond categories.

6

Expert

Limitations and assumptions of goodness of fit tests

🤔Before reading on: do you think goodness of fit tests always give reliable results regardless of sample size? Commit to your answer.

Concept: Understand when goodness of fit tests can mislead and what assumptions they rely on.

Goodness of fit tests assume independent observations and sufficient sample size. Small samples can give unreliable p-values. Also, tests may not detect subtle model mismatches. Knowing these limits helps avoid overconfidence in results.

Result

You become cautious and critical when interpreting goodness of fit outcomes.

Understanding test assumptions prevents misuse and guides better model evaluation strategies.

Under the Hood

Goodness of fit tests calculate a statistic that measures the difference between observed and expected data. For example, the chi-square test sums squared differences divided by expected counts. The test then uses probability theory to find how likely such a difference is if the model is true. This involves distributions like chi-square or Kolmogorov distribution to get p-values.

Why designed this way?

These tests were designed to provide objective, quantifiable measures of model fit using probability theory. Early statisticians needed simple formulas to compare data and models without complex computations. The chi-square test was chosen for categorical data because it is easy to calculate and interpret. The KS test was developed to handle continuous data where counts are not meaningful.

Observed Data ──▶ Calculate Differences ──▶ Compute Statistic
       │                                    │
       └───────────── Expected Data ───────┘
                     │
                     ▼
               Use Distribution
                     │
                     ▼
                 Get p-value
                     │
                     ▼
             Decide Model Fit

Myth Busters - 4 Common Misconceptions

Quick: Does a high p-value prove the model is correct? Commit to yes or no before reading on.

Common Belief: A high p-value means the model is definitely correct.

Reality: A high p-value only means there is not enough evidence to reject the model; it does not prove correctness.

Why it matters: Believing this can lead to overconfidence and ignoring model flaws that the test cannot detect.

Quick: Can goodness of fit tests be used with very small sample sizes reliably? Commit to yes or no before reading on.

Common Belief: Goodness of fit tests work well even with very small samples.

Reality: Small samples often produce unreliable test results and misleading p-values.

Why it matters: Using tests on small data can cause wrong conclusions about model fit.

Quick: Does a low chi-square statistic always mean a perfect model fit? Commit to yes or no before reading on.

Common Belief: A low chi-square statistic means the model fits perfectly.

Reality: A low statistic means observed and expected are close, but it does not guarantee the model is perfect or the best choice.

Why it matters: Misinterpreting this can prevent exploring better models or understanding data nuances.

Quick: Is the chi-square test suitable for continuous data without grouping? Commit to yes or no before reading on.

Common Belief: Chi-square test works directly on continuous data without any changes.

Reality: Chi-square requires categorical data; continuous data must be grouped or use other tests like KS.

Why it matters: Applying chi-square incorrectly leads to invalid results and wrong model assessments.

Expert Zone

1

Goodness of fit tests are sensitive to sample size; large samples can detect trivial differences, while small samples may miss important ones.

2

The choice of bins or categories in chi-square tests affects results; poor binning can hide or exaggerate misfit.

3

Multiple goodness of fit tests can be combined to get a fuller picture, as each test has different strengths and weaknesses.

When NOT to use

Avoid goodness of fit tests when sample sizes are too small or data violate independence assumptions. Instead, use graphical methods like Q-Q plots or bootstrap methods for model assessment.

Production Patterns

In real-world systems, goodness of fit evaluation is automated in model pipelines to flag poor models early. It is combined with cross-validation and residual analysis to ensure robust model performance before deployment.

Connections

Hypothesis Testing

Goodness of fit tests are a type of hypothesis test checking if data fit a model.

Understanding hypothesis testing principles helps grasp why goodness of fit tests use p-values and significance levels.

Machine Learning Model Evaluation

Goodness of fit relates to evaluating how well models predict data, similar to metrics like accuracy or RMSE.

Knowing goodness of fit deepens understanding of model evaluation beyond just prediction errors.

Quality Control in Manufacturing

Both use statistical tests to check if observed outcomes match expected standards.

Seeing goodness of fit as a quality check helps appreciate its role in ensuring reliable models and processes.

Common Pitfalls

#1 Using chi-square test on data with very small expected counts.

Wrong approach:

from scipy.stats import chisquare
observed = [5, 1, 0]
expected = [3, 3, 3]
chisquare(observed, expected)

Correct approach:

from scipy.stats import chisquare
observed = [5, 1, 0]
expected = [3, 3, 3]
# Combine categories or use Fisher's exact test if counts are too small

Root cause:

Chi-square test assumptions require expected counts to be sufficiently large; ignoring this leads to invalid results.

#2 Interpreting a p-value of 0.06 as strong evidence against the model.

Wrong approach:

if p_value < 0.05:
print('Reject model')
else:
print('Reject model') # Wrong: rejects even if p=0.06

Correct approach:

if p_value < 0.05:
print('Reject model')
else:
print('Fail to reject model') # Correct interpretation

Root cause:

Misunderstanding that p-value > 0.05 means insufficient evidence to reject, not proof of rejection.

#3 Applying chi-square test directly on continuous data without binning.

Wrong approach:

from scipy.stats import chisquare
observed = [1.2, 2.5, 3.7, 4.1]
expected = [1.0, 2.0, 4.0, 4.0]
chisquare(observed, expected)

Correct approach:

from scipy.stats import kstest
import numpy as np
observed = np.array([1.2, 2.5, 3.7, 4.1])
kstest(observed, 'norm') # Use KS test for continuous data

Root cause:

Chi-square test requires categorical data; continuous data must be handled with appropriate tests.

Key Takeaways

Goodness of fit evaluation checks how well a model's predictions match actual data to ensure reliability.

The chi-square test is a common method for categorical data, while the Kolmogorov-Smirnov test works for continuous data.

Interpreting p-values correctly is crucial: a high p-value means the model is plausible, not proven correct.

Goodness of fit tests have assumptions and limits; knowing these prevents misuse and wrong conclusions.

Using scipy makes performing these tests easy and reduces calculation errors, helping focus on interpretation.