Evaluation metrics (RMSE, precision@k) in ML Python - Model Pipeline Trace

Model Pipeline - Evaluation metrics (RMSE, precision@k)

This pipeline shows how we measure how well a model predicts numbers using RMSE and how well it ranks items using precision@k.

Data Flow - 5 Stages

1Data in

1000 rows x 5 columns→Collect raw data with features and true labels→1000 rows x 5 columns

Features: user age, item category; Label: true rating (3.5) or relevant item (yes/no)

↓

2Preprocessing

1000 rows x 5 columns→Clean data and split into features and labels→1000 rows x 4 columns (features), 1000 rows x 1 column (labels)

Features: [25, electronics, ...], Label: 4.0

↓

3Model Trains

800 rows x 4 columns (features), 800 rows x 1 column (labels)→Train model to predict ratings or rank items→Trained model

Model learns to predict rating 4.0 or rank items for user

↓

4Model Predicts

200 rows x 4 columns (features)→Model predicts ratings or ranks items→200 rows x 1 column (predicted ratings) or 200 rows x k items (ranked list)

Predicted rating: 3.8, Ranked items: [item5, item2, item9]

↓

5Evaluation Metrics

200 rows x 1 column (predicted ratings), 200 rows x 1 column (true ratings)→Calculate RMSE for rating predictions and precision@k for ranked items→Single RMSE value, precision@k value

RMSE: 0.45, precision@5: 0.8

Training Trace - Epoch by Epoch

Loss
1.2 |*       
0.9 | **     
0.7 |  ***   
0.6 |   ***  
0.55|    *** 
    +--------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	N/A	Initial loss is high because model is just starting
2	0.9	N/A	Loss decreases as model learns patterns
3	0.7	N/A	Loss continues to decrease, model improving
4	0.6	N/A	Loss stabilizes, model converging
5	0.55	N/A	Small improvements, training nearing end

Prediction Trace - 5 Layers

Layer 1: Input features

Layer 2: Model prediction for rating

Layer 3: Calculate RMSE

Layer 4: Model prediction for ranking

Layer 5: Calculate precision@3

Model Quiz - 3 Questions

What does a lower RMSE value mean?

A Model is overfitting

B Predictions are more spread out

C Predictions are closer to true values

D Model is ignoring input features

Key Insight

RMSE helps us understand how close our predicted numbers are to the true numbers by measuring average error size. Precision@k tells us how good the model is at picking the right top items, which is important in recommendations.