Message roles (system, user, assistant) in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Message roles (system, user, assistant)

Problem:You are building a chatbot using a generative AI model. The model uses three message roles: system, user, and assistant. The system message sets the behavior, the user message is the input, and the assistant message is the model's reply. Currently, the model responds well to user messages but sometimes ignores the system instructions, leading to inconsistent answers.

Current Metrics:User satisfaction score: 70%, Consistency score: 65%

Issue:The model does not consistently follow system instructions, causing lower consistency and user satisfaction.

Your Task

Improve the model's adherence to system instructions to increase consistency score to at least 85% while maintaining or improving user satisfaction above 75%.

Do not change the model architecture or training data.

Only modify how messages with roles are structured and sent to the model.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

messages = [
    {"role": "system", "content": "You are a helpful assistant. Always follow the user's instructions carefully and provide clear, concise answers."},
    {"role": "user", "content": "Explain how photosynthesis works in simple terms."}
]

response = model.chat(messages=messages)
print(response.content)

Placed a clear and explicit system message at the start to guide the assistant's behavior.

Ensured the system message is the first message in the conversation list.

Simplified and emphasized instructions in the system message to improve adherence.

Results Interpretation

Before: User satisfaction 70%, Consistency 65%
After: User satisfaction 78%, Consistency 87%

Proper use and placement of system messages significantly improves the model's ability to follow instructions, leading to better consistency and user satisfaction.

Bonus Experiment

Try adding multiple system messages with different instructions and observe how the model prioritizes them.

💡 Hint

Experiment with combining or splitting instructions across system messages and see which approach yields clearer assistant behavior.