Red teaming and adversarial testing in Prompt Engineering / GenAI - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create an adversarial example by adding small noise to the input.

Prompt Engineering / GenAI

adversarial_input = original_input + [1]

Drag options to blanks, or click blank then click option'

Alabels

Bnoise

Cmodel

Dloss

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'labels' instead of noise to modify the input.

Adding the model or loss instead of noise.

Adding small noise to the original input creates an adversarial example that can fool the model.

2fill in blank

medium

Complete the code to calculate the loss used for adversarial training.

Prompt Engineering / GenAI

loss = loss_function([1], predictions)

Drag options to blanks, or click blank then click option'

Aoriginal_labels

Badversarial_labels

Cadversarial_input

Dmodel_output

Attempts:

3 left

💡 Hint

Common Mistakes

Using adversarial input or model output instead of labels.

Confusing inputs with labels.

The loss compares the model's predictions to the original labels to measure error.

3fill in blank

hard

Fix the error in the code to generate adversarial noise using gradient sign method.

Prompt Engineering / GenAI

noise = epsilon * [1](loss, input, retain_graph=True)

Drag options to blanks, or click blank then click option'

Abackward

Bzero_grad

Cdetach

Dgrad

Attempts:

3 left

💡 Hint

Common Mistakes

Using backward() instead of accessing the gradient.

Calling zero_grad() or detach() incorrectly.

The gradient of the loss with respect to the input is used to create adversarial noise.

4fill in blank

hard

Fill both blanks to create a dictionary of adversarial examples filtered by confidence score.

Prompt Engineering / GenAI

adv_examples = {input: output for input, output in dataset.items() if output [1] threshold and confidence_score(input) [2] 0.8}

Drag options to blanks, or click blank then click option'

A>

B<

C>=

D<=

Attempts:

3 left

💡 Hint

Common Mistakes

Using '<' instead of '>' for output filtering.

Using '<' instead of '>=' for confidence filtering.

We select outputs greater than the threshold and confidence scores greater or equal to 0.8.

5fill in blank

hard

Fill all three blanks to implement adversarial training step updating model parameters.

Prompt Engineering / GenAI

optimizer.[1]()
loss = loss_function(model([2]), labels)
loss.[3]()

Drag options to blanks, or click blank then click option'

Azero_grad

Badversarial_input

Cbackward

Doriginal_input

Attempts:

3 left

💡 Hint

Common Mistakes

Not clearing gradients before backward pass.

Using original input instead of adversarial input.

Calling backward on loss incorrectly.

We clear gradients, compute loss on adversarial input, then backpropagate to update model.

Practice

1. What is the main goal of red teaming in AI?

easy

A. To find weaknesses by testing with tricky inputs

B. To train the AI model with more data

C. To improve the speed of the AI model

D. To reduce the size of the AI model

Solution

1. Step 1: Understand red teaming purpose

   Red teaming is about testing AI models with challenging inputs to find weaknesses.

2. Step 2: Compare options

   Only To find weaknesses by testing with tricky inputs matches this goal; others relate to training, speed, or size, which are unrelated.

3. Final Answer:

   To find weaknesses by testing with tricky inputs -> Option A

4. Quick Check:

   Red teaming = find weaknesses [OK]

Hint: Red teaming means testing for weaknesses with tricky inputs [OK]

Common Mistakes:

• Confusing red teaming with training
• Thinking it improves speed or size
• Assuming it fixes bugs automatically

2. Which of the following is the correct way to describe an adversarial example?

easy

A. A normal input that the model handles well

B. A training example used to improve accuracy

C. A random input unrelated to the task

D. An input designed to confuse the AI model

Solution

1. Step 1: Define adversarial example

   An adversarial example is a carefully crafted input meant to confuse or trick the AI model.

2. Step 2: Match definition to options

   An input designed to confuse the AI model matches this exactly; others describe normal, random, or training inputs.

3. Final Answer:

   An input designed to confuse the AI model -> Option D

4. Quick Check:

   Adversarial example = tricky input [OK]

Hint: Adversarial examples are tricky inputs to confuse AI [OK]

Common Mistakes:

• Thinking adversarial means normal or random input
• Confusing training data with adversarial examples
• Assuming adversarial examples improve model accuracy

3. Consider this Python code snippet for adversarial testing:

def test_model(model, inputs):
    results = []
    for inp in inputs:
        pred = model.predict(inp)
        if pred == 'safe':
            results.append(True)
        else:
            results.append(False)
    return results

inputs = ['normal', 'tricky', 'normal']
class DummyModel:
    def predict(self, x):
        return 'safe' if x == 'normal' else 'unsafe'

model = DummyModel()
print(test_model(model, inputs))

What is the output?

medium

A. [False, True, False]

B. [True, True, True]

C. [True, False, True]

D. [False, False, False]

Solution

1. Step 1: Understand model predictions

   The DummyModel returns 'safe' for 'normal' inputs and 'unsafe' for others.

2. Step 2: Evaluate each input

   Inputs are ['normal', 'tricky', 'normal']. Predictions: 'safe', 'unsafe', 'safe'. Results: True, False, True.

3. Final Answer:

   [True, False, True] -> Option C

4. Quick Check:

   Predictions match results [OK]

Hint: Check each input prediction carefully [OK]

Common Mistakes:

• Mixing up 'safe' and 'unsafe' outputs
• Assuming all inputs are safe
• Ignoring the else condition

4. This code tries to detect adversarial inputs but has a bug:

def detect_adversarial(inputs, model):
    flagged = []
    for i in inputs:
        if model.predict(i) == 'safe':
            flagged.append(i)
    return flagged

class Model:
    def predict(self, x):
        return 'unsafe' if x == 'tricky' else 'safe'

inputs = ['normal', 'tricky', 'normal']
print(detect_adversarial(inputs, Model()))

What is the bug?

medium

A. The model.predict method is missing

B. It flags safe inputs instead of unsafe ones

C. The inputs list is empty

D. The function returns a boolean instead of a list

Solution

1. Step 1: Analyze detection logic

   The function flags inputs where model.predict returns 'safe'.

2. Step 2: Check model behavior

   Model returns 'unsafe' for 'tricky', 'safe' otherwise. So safe inputs are flagged, which is wrong.

3. Final Answer:

   It flags safe inputs instead of unsafe ones -> Option B

4. Quick Check:

   Flagging logic reversed [OK]

Hint: Check if flagged inputs match unsafe cases [OK]

Common Mistakes:

• Assuming model.predict is missing
• Thinking inputs list is empty
• Confusing return types

5. You want to improve an AI chatbot's safety by using red teaming and adversarial testing. Which combined approach is best?

hard

A. Use tricky inputs to find weaknesses, then retrain with those examples

B. Ignore tricky inputs and focus on normal conversation data

C. Only test with random inputs and fix errors found

D. Reduce model size to avoid complex errors

Solution

1. Step 1: Understand red teaming and adversarial testing roles

   They find weaknesses by using tricky inputs to test the model.

2. Step 2: Combine testing with retraining

   After finding weaknesses, retraining with those examples improves safety and reliability.

3. Final Answer:

   Use tricky inputs to find weaknesses, then retrain with those examples -> Option A

4. Quick Check:

   Test + retrain = better safety [OK]

Hint: Test with tricky inputs, then retrain to fix weaknesses [OK]

Common Mistakes:

• Only testing without retraining
• Ignoring tricky inputs
• Thinking smaller models fix safety