Training data preparation in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Training data preparation

This pipeline shows how raw data is cleaned and organized before training a machine learning model. It prepares the data so the model can learn well.

Data Flow - 6 Stages

1Raw data collection

1000 rows x 10 columns→Collect data from various sources with missing and noisy values→1000 rows x 10 columns

Row example: {"age": 25, "income": null, "gender": "M", ...}

↓

2Data cleaning

1000 rows x 10 columns→Fill missing values and remove duplicates→980 rows x 10 columns

Row example: {"age": 25, "income": 50000, "gender": "M", ...}

↓

3Feature selection

980 rows x 10 columns→Keep only relevant columns for prediction→980 rows x 6 columns

Columns kept: age, income, gender, education, hours_worked, target

↓

4Data encoding

980 rows x 6 columns→Convert categorical data to numbers→980 rows x 8 columns

Gender 'M' -> 1, 'F' -> 0; Education levels one-hot encoded

↓

5Data normalization

980 rows x 8 columns→Scale numeric features to 0-1 range→980 rows x 8 columns

Age scaled from 0 to 1, income scaled from 0 to 1

↓

6Train/test split

980 rows x 8 columns→Split data into training and testing sets→784 rows x 8 columns (train), 196 rows x 8 columns (test)

Training set example row: {age: 0.3, income: 0.5, gender: 1, ...}

Training Trace - Epoch by Epoch

Loss
0.7 |****
0.6 |****
0.5 |***
0.4 |**
0.3 |*
    +------------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Model starts learning with moderate loss and accuracy
2	0.50	0.72	Loss decreases and accuracy improves as model learns
3	0.40	0.80	Model continues to improve with lower loss and higher accuracy
4	0.35	0.85	Training converges with good accuracy and low loss
5	0.32	0.87	Final epoch shows stable loss and accuracy

Prediction Trace - 3 Layers

Layer 1: Input features

Layer 2: Hidden layer with ReLU activation

Layer 3: Output layer with sigmoid activation

Model Quiz - 3 Questions

What happens to missing values during data cleaning?

A They are filled or removed

B They are left as is

C They are converted to zeros

D They are duplicated

Key Insight

Preparing data carefully by cleaning, selecting, encoding, and normalizing helps the model learn better and faster. Good data preparation leads to lower loss and higher accuracy during training.