Training data preparation in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Training data preparation

Which metric matters for Training Data Preparation and WHY

When preparing training data, the key metric to watch is data quality. This means how clean, balanced, and relevant your data is. Good data helps your model learn well and make correct predictions.

Metrics like class balance (how evenly classes are represented) and missing value rate (how much data is incomplete) matter a lot. If your data is messy or biased, your model's accuracy, precision, and recall will suffer.

Confusion Matrix or Equivalent Visualization

Confusion Matrix Example (after training with good data):

          Predicted
          Pos   Neg
Actual Pos  90    10
       Neg  15    85

- Total samples = 90 + 10 + 15 + 85 = 200
- Precision = 90 / (90 + 15) = 0.857
- Recall = 90 / (90 + 10) = 0.9

If training data is poor, these numbers drop, showing the model learned wrong patterns.
    

Precision vs Recall Tradeoff with Examples

Good training data helps balance precision and recall. For example:

• Spam filter: High precision means few good emails marked as spam. Training data must include many examples of real spam and real emails.
• Medical diagnosis: High recall means catching most sick patients. Training data must have enough positive cases to teach the model.

If training data is biased or missing classes, the model may have high precision but low recall, or vice versa.

What "Good" vs "Bad" Metric Values Look Like for Training Data Preparation

Good training data:

• Balanced classes (e.g., 50% positive, 50% negative)
• Low missing data (<5%)
• Clear, correct labels
• Results in model metrics: accuracy > 85%, precision and recall both > 80%

Bad training data:

• Highly imbalanced classes (e.g., 95% negative, 5% positive)
• Lots of missing or noisy data (>20%)
• Incorrect or inconsistent labels
• Results in model metrics: accuracy high but recall or precision very low (e.g., recall < 50%)

Common Metrics Pitfalls in Training Data Preparation

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, 95% accuracy if model always predicts the majority class.
• Data leakage: When test data leaks into training, metrics look perfect but model fails in real use.
• Overfitting indicators: Very high training accuracy but low test accuracy means model memorized bad data instead of learning.
• Ignoring class balance: Leads to poor recall or precision on minority classes.

Self-Check Question

Your model has 98% accuracy but only 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous. High accuracy is misleading because fraud is rare (class imbalance). You need better training data to improve recall.

Key Result

Good training data leads to balanced precision and recall, avoiding misleading high accuracy from poor data.

Practice

1. What is the main purpose of training data preparation in machine learning?

easy

A. To clean and organize data for better model learning

B. To create the final model architecture

C. To deploy the model to production

D. To write the code for model training

Solution

1. Step 1: Understand the role of training data preparation

   Training data preparation involves cleaning and organizing data so the model can learn effectively.

2. Step 2: Differentiate from other steps in machine learning

   Creating model architecture, deployment, and coding are separate steps after data preparation.

3. Final Answer:

   To clean and organize data for better model learning -> Option A

4. Quick Check:

   Training data preparation = cleaning and organizing data [OK]

Hint: Focus on data cleaning and organizing for training [OK]

Common Mistakes:

• Confusing data preparation with model building
• Thinking deployment is part of data preparation
• Assuming coding is data preparation

2. Which of the following is the correct way to split data into training and testing sets in Python using scikit-learn?

easy

A. split_train_test(data, 0.2)

B. train_test(data, split=0.2)

C. train_test_split(data, test_size=0.2)

D. test_train_split(data, size=0.2)

Solution

1. Step 1: Recall the scikit-learn function for splitting data

   The correct function is train_test_split with parameters like test_size.

2. Step 2: Check the syntax of each option

   Only train_test_split(data, test_size=0.2) uses the correct function name and parameter syntax.

3. Final Answer:

   train_test_split(data, test_size=0.2) -> Option C

4. Quick Check:

   Correct function and parameter = train_test_split(data, test_size=0.2) [OK]

Hint: Remember scikit-learn's train_test_split function name [OK]

Common Mistakes:

• Using wrong function names
• Incorrect parameter names
• Mixing order of parameters

3. Given the code below, what will be the output of print(X_train.shape, X_test.shape)?

from sklearn.model_selection import train_test_split
import numpy as np
X = np.arange(20).reshape(10, 2)
X_train, X_test = train_test_split(X, test_size=0.3, random_state=42)

medium

A. (7, 2) (3, 2)

B. (3, 2) (7, 2)

C. (10, 2) (0, 2)

D. (5, 2) (5, 2)

Solution

1. Step 1: Understand the data shape and split ratio

   The data X has 10 rows and 2 columns. test_size=0.3 means 30% data for testing (3 rows) and 70% for training (7 rows).

2. Step 2: Calculate the shapes of training and testing sets

   Training set shape: (7, 2), Testing set shape: (3, 2).

3. Final Answer:

   (7, 2) (3, 2) -> Option A

4. Quick Check:

   70% train = 7 rows, 30% test = 3 rows [OK]

Hint: Calculate rows by multiplying total by split ratio [OK]

Common Mistakes:

• Swapping train and test sizes
• Ignoring the shape's second dimension
• Misunderstanding test_size meaning

4. Identify the error in the following code snippet for normalizing data using MinMaxScaler:

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
X = [[-1, 2], [-0.5, 6], [0, 10], [1, 18]]
X_scaled = scaler.fit_transform(X)
print(X_scaled)

medium

A. MinMaxScaler cannot handle negative values

B. MinMaxScaler requires data as a numpy array, not list

C. fit_transform should be called on scaler.fit(X).transform(X)

D. No error, code runs correctly

Solution

1. Step 1: Check input data type compatibility

   MinMaxScaler accepts lists or numpy arrays as input, so list input is valid.

2. Step 2: Verify method usage

   Calling scaler.fit_transform(X) is the correct way to fit and transform data in one step.

3. Final Answer:

   No error, code runs correctly -> Option D

4. Quick Check:

   MinMaxScaler works with lists and fit_transform method [OK]

Hint: MinMaxScaler accepts lists and arrays directly [OK]

Common Mistakes:

• Thinking input must be numpy array
• Misusing fit and transform methods
• Assuming scaler rejects negative values

5. You have a dataset with categorical text features and numeric features. Which sequence of steps correctly prepares the data for training a machine learning model?

hard

A. Split data, encode categorical features, normalize numeric features, then clean missing values

B. Clean missing values, encode categorical features, normalize numeric features, then split data

C. Normalize numeric features, clean missing values, split data, then encode categorical features

D. Encode categorical features, split data, clean missing values, then normalize numeric features

Solution

1. Step 1: Clean missing values first

   Cleaning missing data ensures no errors during encoding or normalization.

2. Step 2: Encode categorical features before normalization

   Categorical data must be converted to numbers before normalization.

3. Step 3: Normalize numeric features and then split data

   Normalization scales numeric data; splitting last avoids data leakage.

4. Final Answer:

   Clean missing values, encode categorical features, normalize numeric features, then split data -> Option B

5. Quick Check:

   Proper order: clean -> encode -> normalize -> split [OK]

Hint: Always clean first, encode before normalize, split last [OK]

Common Mistakes:

• Splitting data before cleaning causes leakage
• Normalizing before encoding categorical data
• Encoding after splitting leads to inconsistent categories