Red teaming and adversarial testing in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Purpose of Red Teaming in AI

What is the main purpose of red teaming in the context of AI systems?

A To increase the size of the training dataset by generating synthetic data

B To improve the speed of AI model training by optimizing hardware usage

C To identify vulnerabilities and weaknesses by simulating attacks or adversarial inputs

D To reduce the model size for deployment on mobile devices

💡 Hint

Think about how red teaming helps find problems before they happen.

Red teaming involves actively trying to find weaknesses or flaws in AI systems by simulating attacks or adversarial inputs. This helps improve system robustness and safety.

❓ Predict Output

intermediate

Output of Adversarial Example Generation Code

What will be the output of the following Python code snippet that generates an adversarial example for a simple model?

Prompt Engineering / GenAI

import numpy as np

def simple_model(x):
    return x * 2

original_input = np.array([1.0, 2.0, 3.0])
adversarial_perturbation = np.array([0.1, -0.2, 0.3])
adversarial_input = original_input + adversarial_perturbation
output = simple_model(adversarial_input)
print(output)

A [2.0 4.0 6.0]

B [0.2 0.4 0.6]

C [1.1 1.8 3.3]

D [2.2 3.6 6.6]

💡 Hint

Remember the model doubles the input values.

The adversarial input is the original input plus the perturbation. The model doubles each input value, so the output is the adversarial input multiplied by 2.

❓ Model Choice

advanced

Best Model Type for Adversarial Robustness

Which type of model architecture is generally considered more robust against adversarial attacks?

A Shallow linear models without nonlinearities

B Models trained with adversarial training techniques

C Deep neural networks without any regularization

D Untrained random weight neural networks

💡 Hint

Think about training methods that expose the model to attacks during learning.

Adversarial training involves training the model on adversarial examples, which helps it learn to resist such attacks and improves robustness.

❓ Metrics

advanced

Metric to Evaluate Adversarial Robustness

Which metric is most appropriate to evaluate the adversarial robustness of a classification model?

A Accuracy on adversarially perturbed test data

B Standard accuracy on clean test data

C Training loss on the training dataset

D Model size in megabytes

💡 Hint

Robustness means performance when under attack.

Accuracy on adversarially perturbed data shows how well the model performs when attacked, which directly measures robustness.

🔧 Debug

expert

Debugging Adversarial Attack Code

Consider this Python code snippet intended to create an adversarial example by adding a small perturbation to an input tensor. What error will this code raise?

Prompt Engineering / GenAI

import torch

input_tensor = torch.tensor([1.0, 2.0, 3.0])
perturbation = torch.tensor([0.1, 0.1])
adversarial_input = input_tensor + perturbation
print(adversarial_input)

A RuntimeError due to size mismatch in tensor addition

B TypeError because tensors must be converted to numpy arrays first

C No error, outputs tensor with added perturbation

D NameError because 'torch' is not imported

💡 Hint

Check if the tensors have the same shape before adding.

The input tensor has 3 elements, but the perturbation has 2 elements. Adding tensors of different sizes without broadcasting causes a RuntimeError.