Why machine learning matters in ML Python - Why Metrics Matter

Metrics & Evaluation - Why machine learning matters

Which metric matters for this concept and WHY

Machine learning is about making predictions or decisions from data. The key metrics depend on the task. For classification, accuracy, precision, and recall matter because they show how well the model identifies correct and incorrect cases. For regression, mean squared error or mean absolute error show how close predictions are to real values. These metrics matter because they tell us if the machine learning model is useful and trustworthy in real life.

Confusion matrix or equivalent visualization (ASCII)

      Confusion Matrix Example for a Spam Email Classifier:

          Predicted
          Spam   Not Spam
    Actual
    Spam     90       10
    Not Spam  5       95

    Here:
    - True Positives (TP) = 90 (Spam correctly found)
    - False Negatives (FN) = 10 (Spam missed)
    - False Positives (FP) = 5 (Good email marked spam)
    - True Negatives (TN) = 95 (Good email correctly found)
    

Precision vs Recall tradeoff with concrete examples

Precision and recall balance different needs:

• Precision is about how many predicted positives are actually correct. For example, in spam detection, high precision means few good emails are wrongly marked as spam.
• Recall is about how many actual positives are found. For example, in cancer detection, high recall means catching most cancer cases, even if some false alarms happen.

Choosing which metric to focus on depends on what mistakes cost more in real life.

What "good" vs "bad" metric values look like for this use case

For a spam filter:

• Good: Precision around 0.95 or higher (few good emails marked spam), recall around 0.85 or higher (most spam caught).
• Bad: Precision below 0.7 (many good emails lost), recall below 0.5 (many spam emails missed).

For a cancer detector:

• Good: Recall above 0.9 (most cancer cases found), precision can be lower because false alarms can be checked again.
• Bad: Recall below 0.6 (many cancer cases missed), even if precision is high.

Metrics pitfalls

• Accuracy paradox: High accuracy can be misleading if data is unbalanced. For example, if 99% emails are good, a model always predicting "not spam" gets 99% accuracy but is useless.
• Data leakage: When test data leaks into training, metrics look perfect but model fails in real life.
• Overfitting indicators: Very high training accuracy but low test accuracy means the model memorizes data and won't generalize.

Self-check question

Your model has 98% accuracy but 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses 88% of fraud cases, which is very risky. Even with high accuracy, missing most fraud is unacceptable.

Key Result

Choosing the right metric like precision or recall depends on the real-world cost of mistakes to know if a machine learning model truly works.

Practice

1. Why is machine learning important in everyday technology?

easy

A. It helps computers learn from data to make better decisions.

B. It replaces all human jobs immediately.

C. It makes computers run faster without data.

D. It only works with fixed rules programmed by humans.

Solution

1. Step 1: Understand machine learning's role

   Machine learning allows computers to learn from data instead of relying on fixed rules.

2. Step 2: Connect to everyday use

   This learning helps computers make better decisions, like recommending movies or predicting weather.

3. Final Answer:

   It helps computers learn from data to make better decisions. -> Option A

4. Quick Check:

   Machine learning = learning from data [OK]

Hint: Machine learning means learning from data, not fixed rules [OK]

Common Mistakes:

• Thinking machine learning replaces all human jobs immediately
• Believing machine learning works without data
• Confusing fixed rules with learning from data

2. Which of the following is the correct way to describe machine learning?

easy

A. A method where computers follow fixed rules only.

B. A technique where computers learn patterns from data.

C. A process that requires no data to work.

D. A system that ignores data and guesses randomly.

Solution

1. Step 1: Define machine learning

   Machine learning is about computers learning patterns from data, not following fixed rules.

2. Step 2: Eliminate incorrect options

   Options A, B, and C describe wrong ideas: fixed rules only, ignores data and guesses randomly, or requires no data.

3. Final Answer:

   A technique where computers learn patterns from data. -> Option B

4. Quick Check:

   Machine learning = learning patterns from data [OK]

Hint: Machine learning means learning patterns from data [OK]

Common Mistakes:

• Confusing fixed rules with learning
• Thinking machine learning works without data
• Assuming machine learning guesses randomly

3. Consider this simple example: A machine learning model is trained to predict if a fruit is an apple or orange based on color. If the model sees a red fruit, what is the most likely prediction?

medium

A. Orange

B. Banana

C. Apple

D. Unknown fruit

Solution

1. Step 1: Understand the training data context

   The model learned to predict apple or orange based on color. Red fruits are usually apples.

2. Step 2: Predict based on color input

   Given a red fruit, the model will most likely predict apple, as oranges are usually orange-colored.

3. Final Answer:

   Apple -> Option C

4. Quick Check:

   Red fruit = Apple prediction [OK]

Hint: Match color to fruit learned during training [OK]

Common Mistakes:

• Choosing orange for red fruit
• Picking banana which model was not trained on
• Selecting unknown when model predicts known classes

4. A machine learning model is trained to predict house prices but always predicts zero. What is the most likely problem?

medium

A. The model learned to predict zero due to incorrect training.

B. The model was not trained with any data.

C. The model is perfect and zero is the correct price.

D. The model is using random guesses.

Solution

1. Step 1: Analyze the model's output

   The model always predicts zero, which is unlikely for house prices.

2. Step 2: Identify training issue

   This usually means the model learned a wrong pattern, possibly due to bad or missing training data.

3. Final Answer:

   The model learned to predict zero due to incorrect training. -> Option A

4. Quick Check:

   Always zero output = training problem [OK]

Hint: Always same output means training issue [OK]

Common Mistakes:

• Assuming no data means random guesses
• Thinking zero is a perfect prediction
• Ignoring training data quality

5. You want to build a machine learning system to recommend movies based on user ratings. Which step is most important to make the system useful?

hard

A. Ignoring user preferences and recommending random movies.

B. Writing fixed rules for every movie genre manually.

C. Using only one movie rating to decide recommendations.

D. Collecting lots of user rating data to learn patterns.

Solution

1. Step 1: Understand recommendation system needs

   To recommend movies well, the system needs to learn from many user ratings to find patterns.

2. Step 2: Compare options

   Fixed rules or ignoring preferences won't adapt well. One rating is too little data.

3. Final Answer:

   Collecting lots of user rating data to learn patterns. -> Option D

4. Quick Check:

   More data = better learning [OK]

Hint: More data helps machine learning find better patterns [OK]

Common Mistakes:

• Relying on fixed rules only
• Ignoring user preferences
• Using too little data for learning