ML workflow (collect, prepare, train, evaluate, deploy) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - ML workflow (collect, prepare, train, evaluate, deploy)

Which metric matters for ML workflow and WHY

In the ML workflow, the evaluation step uses metrics to check how well the model learned. The right metric depends on the problem:

• Accuracy shows overall correct predictions, good for balanced classes.
• Precision tells how many predicted positives are truly positive, important when false alarms are costly.
• Recall shows how many real positives the model found, critical when missing positives is bad.
• F1 score balances precision and recall when both matter.

Choosing the right metric helps decide if the model is ready to deploy or needs more training or data preparation.

Confusion Matrix Example

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    70    |   30    
      Negative           |    10    |   90    

      Total samples = 70 + 30 + 10 + 90 = 200

      Precision = TP / (TP + FP) = 70 / (70 + 10) = 0.875
      Recall = TP / (TP + FN) = 70 / (70 + 30) = 0.7
      Accuracy = (TP + TN) / Total = (70 + 90) / 200 = 0.8
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.778
    

Precision vs Recall Tradeoff with Examples

Imagine a spam email filter:

• High Precision: Few good emails marked as spam (false alarms low). Important so you don't miss important emails.
• High Recall: Most spam emails caught, but some good emails might be wrongly marked as spam.

Or a cancer detector:

• High Recall: Catch almost all cancer cases, even if some false alarms happen.
• High Precision: Fewer false alarms, but risk missing some cancer cases.

Choosing which to prioritize depends on what mistakes cost more in real life.

Good vs Bad Metric Values for ML Workflow

Good metrics mean the model learned well and can be deployed:

• Accuracy above 80% on balanced data is usually good.
• Precision and recall above 70% show balanced performance.
• F1 score close to 1 means excellent balance.

Bad metrics show problems needing more work:

• Accuracy near 50% on balanced data means guessing randomly.
• Very high precision but very low recall means many positives missed.
• Very high recall but very low precision means many false alarms.

Common Pitfalls in ML Workflow Metrics

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced (e.g., 95% accuracy by always predicting the majority class).
• Data leakage: Using future or test data in training inflates metrics falsely.
• Overfitting: Very high training accuracy but low test accuracy means the model memorized training data, not learned general patterns.
• Ignoring metric choice: Using accuracy when recall matters can lead to bad real-world results.

Self Check: Is 98% Accuracy but 12% Recall Good for Fraud Detection?

No, this is not good. Even though accuracy is high, recall is very low. This means the model misses 88% of fraud cases, which is dangerous because many frauds go undetected. For fraud detection, high recall is critical to catch as many frauds as possible, even if some false alarms happen.

Key Result

Choosing the right metric like precision or recall in the ML workflow evaluation step ensures the model meets real-world needs before deployment.

Practice

1. Which step in the ML workflow involves gathering raw data from various sources?

easy

A. Deploy

B. Train

C. Evaluate

D. Collect

Solution

1. Step 1: Understand the meaning of each ML workflow step

   Collect means to gather raw data needed for the model.

2. Step 2: Match the step to the description

   Since gathering data is the first step, it matches 'Collect'.

3. Final Answer:

   Collect -> Option D

4. Quick Check:

   Gathering data = Collect [OK]

Hint: Collect means get data first before anything else [OK]

Common Mistakes:

• Confusing 'Train' with data gathering
• Thinking 'Evaluate' means collecting data
• Mixing 'Deploy' with data collection

2. Which of the following is the correct order of the ML workflow steps?

easy

A. Collect, Prepare, Train, Evaluate, Deploy

B. Train, Collect, Prepare, Deploy, Evaluate

C. Prepare, Collect, Train, Evaluate, Deploy

D. Deploy, Evaluate, Train, Prepare, Collect

Solution

1. Step 1: Recall the standard ML workflow order

   The correct order is: Collect data, Prepare data, Train model, Evaluate model, Deploy model.

2. Step 2: Compare each option to the correct order

   Only Collect, Prepare, Train, Evaluate, Deploy matches this exact sequence.

3. Final Answer:

   Collect, Prepare, Train, Evaluate, Deploy -> Option A

4. Quick Check:

   Correct ML workflow order = Collect, Prepare, Train, Evaluate, Deploy [OK]

Hint: Remember: Collect first, then Prepare, Train, Evaluate, Deploy [OK]

Common Mistakes:

• Mixing up Collect and Prepare order
• Putting Deploy before Evaluate
• Starting with Train before data collection

3. Given this Python code snippet for training a model, what will be printed after evaluation?

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3, random_state=42)
model = DecisionTreeClassifier()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
accuracy = accuracy_score(y_test, predictions)
print(f"Accuracy: {accuracy:.2f}")

medium

A. Accuracy: 0.98

B. Accuracy: 0.85

C. Accuracy: 0.70

D. Accuracy: 1.00

Solution

1. Step 1: Understand the code flow

   The code loads iris data, splits it, trains a Decision Tree, predicts, and calculates accuracy.

2. Step 2: Recall typical Decision Tree accuracy on iris test split

   With random_state=42 and 30% test size, accuracy is 0.98.

3. Final Answer:

   Accuracy: 0.98 -> Option A

4. Quick Check:

   Decision Tree on iris test = 0.98 accuracy [OK]

Hint: Decision Trees on iris usually score 98% accuracy [OK]

Common Mistakes:

• Assuming accuracy is always 1.00
• Confusing train and test accuracy
• Ignoring random_state effect on split

4. You wrote this code to train a model but get an error. What is the main problem?

from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(X_train, y_train)
print(model.predict(X_test))

Assuming X_train, y_train, and X_test are defined but you forgot to prepare data properly.

medium

A. Model was not imported correctly

B. Data was not scaled or cleaned before training

C. Predict method is called before fit

D. X_test has different features than X_train

Solution

1. Step 1: Analyze the error cause

   Logistic Regression often requires data preparation like scaling or cleaning.

2. Step 2: Identify missing preparation step

   Since data preparation is missing, model may fail or give poor results.

3. Final Answer:

   Data was not scaled or cleaned before training -> Option B

4. Quick Check:

   Missing data prep causes training issues [OK]

Hint: Always prepare data before training models [OK]

Common Mistakes:

• Thinking import error caused failure
• Assuming predict before fit error
• Ignoring feature mismatch possibility

5. You want to deploy a model that predicts house prices. Which step should you do right after evaluating the model's accuracy?

hard

A. Collect more data

B. Prepare the data again

C. Deploy the model to a real environment

D. Train the model with new parameters

Solution

1. Step 1: Understand the ML workflow sequence

   After evaluating the model, the next step is to deploy it if accuracy is good.

2. Step 2: Match the step after evaluation

   Deploy means putting the model into real use for predictions.

3. Final Answer:

   Deploy the model to a real environment -> Option C

4. Quick Check:

   Evaluation done, next is Deploy [OK]

Hint: Deploy comes right after evaluation if model is good [OK]

Common Mistakes:

• Repeating data collection unnecessarily
• Preparing data again without new data
• Retraining without evaluation feedback