A/B testing models in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - A/B testing models

Which metric matters for A/B testing models and WHY

A/B testing compares two models to see which one works better in real life. The key metrics depend on the goal. For example, if the goal is to increase clicks, use click-through rate (CTR). If it is to improve sales, use conversion rate. These metrics show the real impact of each model on user behavior.

Besides these, statistical significance (like p-value) is important to know if the difference is real or just by chance.

Confusion matrix or equivalent visualization

In A/B testing, we often track outcomes like success or failure for each model. Here is an example confusion matrix for Model A and Model B showing user actions:

      Model A Confusion Matrix:
      -------------------------
      |           | Clicked | Not Clicked |
      |-----------|---------|-------------|
      | Positive  |   80    |     20      |
      | Negative  |   30    |    870      |
      -------------------------

      Model B Confusion Matrix:
      -------------------------
      |           | Clicked | Not Clicked |
      |-----------|---------|-------------|
      | Positive  |   90    |     10      |
      | Negative  |   40    |    860      |
      -------------------------
    

These matrices help calculate metrics like CTR and conversion rate for each model.

Precision vs Recall tradeoff with concrete examples

A/B testing focuses less on precision and recall and more on overall impact metrics like conversion rate or revenue. However, if the models are classifiers, precision and recall matter.

For example, if Model A catches more true positives (high recall) but also more false positives (low precision), it might annoy users. Model B might be more precise but miss some opportunities. A/B testing helps find the balance that leads to better user experience and business results.

What "good" vs "bad" metric values look like for A/B testing

Good: Model B increases conversion rate from 5% to 7% with a p-value < 0.05, meaning the improvement is real and not by chance.

Bad: Model B shows a 1% increase but with a p-value > 0.1, so the change might be random. Or Model B increases clicks but decreases sales, which is bad for business.

Common pitfalls in A/B testing metrics

• Small sample size: Leads to unreliable results that can mislead decisions.
• Multiple testing: Running many tests increases chance of false positives.
• Ignoring user segments: Different groups may react differently, hiding true effects.
• Data leakage: Mixing data from before and after the test can bias results.
• Stopping test too early: Can cause wrong conclusions due to random fluctuations.

Self-check question

Your A/B test shows Model B has 98% accuracy but only 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. High accuracy can be misleading if fraud cases are rare. Low recall means the model misses most frauds, which is risky. You want high recall to catch as many frauds as possible.

Key Result

A/B testing metrics focus on real user impact like conversion rate and statistical significance to decide which model performs better.