Latency optimization in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Latency optimization

This pipeline shows how we make a machine learning model faster to respond. We reduce waiting time by changing data and model steps carefully.

Data Flow - 6 Stages

1Data input

1000 rows x 10 columns→Raw data loaded→1000 rows x 10 columns

[[5.1, 3.5, ..., 1.4], [4.9, 3.0, ..., 1.4], ...]

↓

2Preprocessing

1000 rows x 10 columns→Normalize features to speed up training→1000 rows x 10 columns

[[0.52, 0.35, ..., 0.14], [0.49, 0.30, ..., 0.14], ...]

↓

3Feature selection

1000 rows x 10 columns→Keep top 5 important features to reduce input size→1000 rows x 5 columns

[[0.52, 0.35, 0.14, 0.22, 0.18], [0.49, 0.30, 0.14, 0.20, 0.17], ...]

↓

4Model training

1000 rows x 5 columns→Train smaller model with fewer layers→Model trained with 5 input features

Model with 2 hidden layers, 32 neurons each

↓

5Model quantization

Model weights in float32→Convert weights to int8 to reduce size and speed up→Model weights in int8

Weights size reduced from 10MB to 2.5MB

↓

6Prediction

1 row x 5 columns→Run fast prediction on optimized model→1 row x 1 column (prediction)

[0.87]

Training Trace - Epoch by Epoch

Loss
0.7 |****
0.6 |*** 
0.5 |**  
0.4 |*   
0.3 |    
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Starting training with smaller model
2	0.50	0.72	Loss decreased, accuracy improved
3	0.40	0.80	Model learning well with fewer features
4	0.35	0.85	Training converging nicely
5	0.30	0.88	Good balance of speed and accuracy

Prediction Trace - 4 Layers

Layer 1: Input layer

Layer 2: Hidden layer 1 (ReLU)

Layer 3: Hidden layer 2 (ReLU)

Layer 4: Output layer (Sigmoid)

Model Quiz - 3 Questions

Why do we select fewer features before training?

A To reduce input size and speed up training

B To increase model complexity

C To add more data columns

D To make the model slower

Key Insight

Reducing input features and model size, plus converting weights to smaller types, helps the model respond faster without losing much accuracy.