Red teaming and adversarial testing in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Red teaming and adversarial testing

This pipeline shows how red teaming and adversarial testing help find weaknesses in AI models by feeding tricky inputs and checking model responses.

Data Flow - 6 Stages

1Data in

1000 rows x 10 columns→Collect normal and adversarial examples (inputs designed to fool the model)→1000 rows x 10 columns

Normal input: 'The cat sat on the mat.' Adversarial input: 'The c@t s@t on the m@t.'

↓

2Preprocessing

1000 rows x 10 columns→Clean text, tokenize, and convert to numbers→1000 rows x 50 tokens

Input text converted to token IDs like [12, 45, 78, ...]

↓

3Feature Engineering

1000 rows x 50 tokens→Embed tokens into vectors→1000 rows x 50 tokens x 128 features

Token 'cat' becomes a 128-dimensional vector

↓

4Model Trains

800 rows x 50 tokens x 128 features→Train model on normal and adversarial data→Trained model

Model learns to classify inputs correctly despite adversarial noise

↓

5Metrics Improve

Validation set 200 rows x 50 tokens x 128 features→Evaluate accuracy and robustness→Accuracy: 85%, Robustness score: 78%

Model correctly classifies 85% of inputs including adversarial ones

↓

6Prediction

1 row x 50 tokens x 128 features→Model predicts label for new input→Prediction: 'Safe' or 'Adversarial'

Input: 'The c@t s@t on the m@t.' Output: 'Adversarial'

Training Trace - Epoch by Epoch

Loss
1.2 |*       
0.9 | **     
0.7 |  ***   
0.5 |    ****
0.4 |     *****
     ----------------
      1  2  3  4  5  Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	0.55	Model starts learning but struggles with adversarial examples
2	0.9	0.65	Loss decreases, accuracy improves as model adapts
3	0.7	0.75	Better handling of adversarial inputs
4	0.5	0.82	Model robustness improves
5	0.4	0.85	Training converges with good accuracy and robustness

Prediction Trace - 5 Layers

Layer 1: Input preprocessing

Layer 2: Embedding layer

Layer 3: Neural network layers

Layer 4: Output layer

Layer 5: Prediction decision

Model Quiz - 3 Questions

What is the main goal of adversarial testing in this pipeline?

A To speed up model training

B To increase the size of the training data

C To find inputs that trick the model

D To reduce the number of model layers

Key Insight

Red teaming and adversarial testing help models learn to recognize tricky inputs, improving their safety and reliability by exposing weaknesses during training.