Red teaming and adversarial testing in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Purpose of Red Teaming in AI

What is the main purpose of red teaming in the context of AI systems?

A To increase the size of the training dataset by generating synthetic data

B To improve the speed of AI model training by optimizing hardware usage

C To identify vulnerabilities and weaknesses by simulating attacks or adversarial inputs

D To reduce the model size for deployment on mobile devices

💡 Hint

Think about how red teaming helps find problems before they happen.

Red teaming involves actively trying to find weaknesses or flaws in AI systems by simulating attacks or adversarial inputs. This helps improve system robustness and safety.

❓ Predict Output

intermediate

Output of Adversarial Example Generation Code

What will be the output of the following Python code snippet that generates an adversarial example for a simple model?

Prompt Engineering / GenAI

import numpy as np

def simple_model(x):
    return x * 2

original_input = np.array([1.0, 2.0, 3.0])
adversarial_perturbation = np.array([0.1, -0.2, 0.3])
adversarial_input = original_input + adversarial_perturbation
output = simple_model(adversarial_input)
print(output)

A [2.0 4.0 6.0]

B [0.2 0.4 0.6]

C [1.1 1.8 3.3]

D [2.2 3.6 6.6]

💡 Hint

Remember the model doubles the input values.

The adversarial input is the original input plus the perturbation. The model doubles each input value, so the output is the adversarial input multiplied by 2.

❓ Model Choice

advanced

Best Model Type for Adversarial Robustness

Which type of model architecture is generally considered more robust against adversarial attacks?

A Shallow linear models without nonlinearities

B Models trained with adversarial training techniques

C Deep neural networks without any regularization

D Untrained random weight neural networks

💡 Hint

Think about training methods that expose the model to attacks during learning.

Adversarial training involves training the model on adversarial examples, which helps it learn to resist such attacks and improves robustness.

❓ Metrics

advanced

Metric to Evaluate Adversarial Robustness

Which metric is most appropriate to evaluate the adversarial robustness of a classification model?

A Accuracy on adversarially perturbed test data

B Standard accuracy on clean test data

C Training loss on the training dataset

D Model size in megabytes

💡 Hint

Robustness means performance when under attack.

Accuracy on adversarially perturbed data shows how well the model performs when attacked, which directly measures robustness.

🔧 Debug

expert

Debugging Adversarial Attack Code

Consider this Python code snippet intended to create an adversarial example by adding a small perturbation to an input tensor. What error will this code raise?

Prompt Engineering / GenAI

import torch

input_tensor = torch.tensor([1.0, 2.0, 3.0])
perturbation = torch.tensor([0.1, 0.1])
adversarial_input = input_tensor + perturbation
print(adversarial_input)

A RuntimeError due to size mismatch in tensor addition

B TypeError because tensors must be converted to numpy arrays first

C No error, outputs tensor with added perturbation

D NameError because 'torch' is not imported

💡 Hint

Check if the tensors have the same shape before adding.

The input tensor has 3 elements, but the perturbation has 2 elements. Adding tensors of different sizes without broadcasting causes a RuntimeError.

Practice

1. What is the main goal of red teaming in AI?

easy

A. To find weaknesses by testing with tricky inputs

B. To train the AI model with more data

C. To improve the speed of the AI model

D. To reduce the size of the AI model

Solution

1. Step 1: Understand red teaming purpose

   Red teaming is about testing AI models with challenging inputs to find weaknesses.

2. Step 2: Compare options

   Only To find weaknesses by testing with tricky inputs matches this goal; others relate to training, speed, or size, which are unrelated.

3. Final Answer:

   To find weaknesses by testing with tricky inputs -> Option A

4. Quick Check:

   Red teaming = find weaknesses [OK]

Hint: Red teaming means testing for weaknesses with tricky inputs [OK]

Common Mistakes:

• Confusing red teaming with training
• Thinking it improves speed or size
• Assuming it fixes bugs automatically

2. Which of the following is the correct way to describe an adversarial example?

easy

A. A normal input that the model handles well

B. A training example used to improve accuracy

C. A random input unrelated to the task

D. An input designed to confuse the AI model

Solution

1. Step 1: Define adversarial example

   An adversarial example is a carefully crafted input meant to confuse or trick the AI model.

2. Step 2: Match definition to options

   An input designed to confuse the AI model matches this exactly; others describe normal, random, or training inputs.

3. Final Answer:

   An input designed to confuse the AI model -> Option D

4. Quick Check:

   Adversarial example = tricky input [OK]

Hint: Adversarial examples are tricky inputs to confuse AI [OK]

Common Mistakes:

• Thinking adversarial means normal or random input
• Confusing training data with adversarial examples
• Assuming adversarial examples improve model accuracy

3. Consider this Python code snippet for adversarial testing:

def test_model(model, inputs):
    results = []
    for inp in inputs:
        pred = model.predict(inp)
        if pred == 'safe':
            results.append(True)
        else:
            results.append(False)
    return results

inputs = ['normal', 'tricky', 'normal']
class DummyModel:
    def predict(self, x):
        return 'safe' if x == 'normal' else 'unsafe'

model = DummyModel()
print(test_model(model, inputs))

What is the output?

medium

A. [False, True, False]

B. [True, True, True]

C. [True, False, True]

D. [False, False, False]

Solution

1. Step 1: Understand model predictions

   The DummyModel returns 'safe' for 'normal' inputs and 'unsafe' for others.

2. Step 2: Evaluate each input

   Inputs are ['normal', 'tricky', 'normal']. Predictions: 'safe', 'unsafe', 'safe'. Results: True, False, True.

3. Final Answer:

   [True, False, True] -> Option C

4. Quick Check:

   Predictions match results [OK]

Hint: Check each input prediction carefully [OK]

Common Mistakes:

• Mixing up 'safe' and 'unsafe' outputs
• Assuming all inputs are safe
• Ignoring the else condition

4. This code tries to detect adversarial inputs but has a bug:

def detect_adversarial(inputs, model):
    flagged = []
    for i in inputs:
        if model.predict(i) == 'safe':
            flagged.append(i)
    return flagged

class Model:
    def predict(self, x):
        return 'unsafe' if x == 'tricky' else 'safe'

inputs = ['normal', 'tricky', 'normal']
print(detect_adversarial(inputs, Model()))

What is the bug?

medium

A. The model.predict method is missing

B. It flags safe inputs instead of unsafe ones

C. The inputs list is empty

D. The function returns a boolean instead of a list

Solution

1. Step 1: Analyze detection logic

   The function flags inputs where model.predict returns 'safe'.

2. Step 2: Check model behavior

   Model returns 'unsafe' for 'tricky', 'safe' otherwise. So safe inputs are flagged, which is wrong.

3. Final Answer:

   It flags safe inputs instead of unsafe ones -> Option B

4. Quick Check:

   Flagging logic reversed [OK]

Hint: Check if flagged inputs match unsafe cases [OK]

Common Mistakes:

• Assuming model.predict is missing
• Thinking inputs list is empty
• Confusing return types

5. You want to improve an AI chatbot's safety by using red teaming and adversarial testing. Which combined approach is best?

hard

A. Use tricky inputs to find weaknesses, then retrain with those examples

B. Ignore tricky inputs and focus on normal conversation data

C. Only test with random inputs and fix errors found

D. Reduce model size to avoid complex errors

Solution

1. Step 1: Understand red teaming and adversarial testing roles

   They find weaknesses by using tricky inputs to test the model.

2. Step 2: Combine testing with retraining

   After finding weaknesses, retraining with those examples improves safety and reliability.

3. Final Answer:

   Use tricky inputs to find weaknesses, then retrain with those examples -> Option A

4. Quick Check:

   Test + retrain = better safety [OK]

Hint: Test with tricky inputs, then retrain to fix weaknesses [OK]

Common Mistakes:

• Only testing without retraining
• Ignoring tricky inputs
• Thinking smaller models fix safety