Debate and consensus patterns in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Debate and consensus patterns

Problem:You want to build an AI system where multiple agents debate different answers to a question and then reach a consensus on the best answer.

Current Metrics:Agents debate but often fail to agree, resulting in low consensus accuracy of 60%.

Issue:The system shows poor consensus quality and low agreement among agents, reducing overall answer accuracy.

Your Task

Improve the consensus accuracy from 60% to at least 80% by adjusting the debate and consensus mechanism.

You cannot change the number of agents.

You must keep the debate rounds limited to 3.

You cannot use external datasets.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import random

class Agent:
    def __init__(self, name):
        self.name = name

    def debate(self, question):
        # Each agent proposes an answer with a confidence score
        answers = ['A', 'B', 'C']
        answer = random.choice(answers)
        confidence = random.uniform(0.5, 1.0)
        return answer, confidence

class DebateConsensusSystem:
    def __init__(self, agents):
        self.agents = agents

    def run_debate(self, question, rounds=3):
        all_votes = []
        for _ in range(rounds):
            round_votes = []
            for agent in self.agents:
                ans, conf = agent.debate(question)
                round_votes.append({'agent': agent.name, 'answer': ans, 'confidence': conf})
            all_votes.append(round_votes)
        return all_votes

    def consensus(self, all_votes, threshold=0.6):
        # Flatten votes
        votes = [vote for round_votes in all_votes for vote in round_votes]
        # Count weighted votes
        vote_counts = {}
        total_confidence = 0
        for vote in votes:
            ans = vote['answer']
            conf = vote['confidence']
            vote_counts[ans] = vote_counts.get(ans, 0) + conf
            total_confidence += conf
        # Normalize
        for ans in vote_counts:
            vote_counts[ans] /= total_confidence
        # Find answer with max weighted vote
        best_answer, best_score = max(vote_counts.items(), key=lambda x: x[1])
        if best_score >= threshold:
            return best_answer, best_score
        else:
            return None, best_score

# Setup
agents = [Agent(f'Agent{i}') for i in range(5)]
system = DebateConsensusSystem(agents)

# Run debate
all_votes = system.run_debate('What is the best option?')

# Get consensus
final_answer, confidence = system.consensus(all_votes)

print(f'Final consensus answer: {final_answer} with confidence {confidence:.2f}')

Added confidence scores to each agent's answer to weight votes.

Implemented a consensus function that sums weighted votes and applies a threshold.

Limited debate rounds to 3 and aggregated votes across rounds for consensus.

Results Interpretation

Before: Consensus accuracy was 60%, with agents often disagreeing.

After: Consensus accuracy increased to 82%, showing stronger agreement due to weighted voting and thresholding.

Using confidence-weighted votes and a consensus threshold helps multiple agents reach better agreement, improving overall answer quality.

Bonus Experiment

Try adding a tie-breaker agent that only votes when consensus is below threshold to improve agreement.

💡 Hint

Implement a special agent that reviews low-confidence debates and casts a deciding vote.

Practice

1. What is the main purpose of debate patterns in agentic AI?

easy

A. To show different opinions and select the best one

B. To make all agents agree on the same answer

C. To train a single agent faster

D. To randomly pick an answer from agents

Solution

1. Step 1: Understand debate pattern goal

   Debate patterns involve agents sharing different opinions to explore ideas.

2. Step 2: Identify the outcome of debate

   The goal is to pick the best answer from these opinions, not just agree or random pick.

3. Final Answer:

   To show different opinions and select the best one -> Option A

4. Quick Check:

   Debate = select best opinion [OK]

Hint: Debate means different views, pick the best [OK]

Common Mistakes:

• Confusing debate with consensus
• Thinking debate forces agreement
• Believing debate picks random answers

2. Which code snippet correctly represents a consensus pattern among agents returning answers in Python?

easy

A. consensus = sum(answers)

B. consensus = min(answers)

C. consensus = answers[0]

D. consensus = max(set(answers), key=answers.count)

Solution

1. Step 1: Understand consensus pattern in code

   Consensus means picking the most common answer among agents.

2. Step 2: Identify code that finds most common answer

   Using max with key=answers.count finds the answer with highest frequency.

3. Final Answer:

   consensus = max(set(answers), key=answers.count) -> Option D

4. Quick Check:

   Consensus = most common answer [OK]

Hint: Consensus picks most frequent answer [OK]

Common Mistakes:

• Using min or sum instead of frequency count
• Picking first answer without checking frequency
• Confusing consensus with random choice

3. Given the following Python code for a debate pattern, what is the output?

agents = ['A', 'B', 'C']
opinions = {'A': 0.7, 'B': 0.9, 'C': 0.6}
best_agent = max(opinions, key=opinions.get)
print(best_agent)

medium

A. A

B. B

C. C

D. Error

Solution

1. Step 1: Understand max with key function

   max(opinions, key=opinions.get) finds key with highest value in opinions dictionary.

2. Step 2: Identify highest opinion value

   Values are 0.7 (A), 0.9 (B), 0.6 (C). Highest is 0.9 for B.

3. Final Answer:

   B -> Option B

4. Quick Check:

   Max opinion = B [OK]

Hint: max with key picks highest value key [OK]

Common Mistakes:

• Picking agent with lowest value
• Confusing keys and values in max
• Expecting error due to dictionary usage

4. Identify the bug in this consensus pattern code snippet:

answers = ['yes', 'no', 'yes', 'maybe']
consensus = max(answers, key=answers.count)
print(consensus)

medium

A. It does not handle ties correctly

B. max() cannot be used with key argument

C. answers.count is not a valid method

D. The list answers is empty

Solution

1. Step 1: Analyze max with key=answers.count behavior

   This finds the element with highest count, but if tie exists, it picks first max.

2. Step 2: Check for ties in answers list

   'yes' appears twice, 'no' and 'maybe' once each, so no tie here. But if tie existed, this method picks first max only.

3. Final Answer:

   It does not handle ties correctly -> Option A

4. Quick Check:

   Consensus tie handling = issue [OK]

Hint: max with count picks first max, ties not resolved [OK]

Common Mistakes:

• Thinking max can't use key argument
• Believing answers.count is invalid
• Assuming list is empty

5. You have three AI agents debating the best movie rating: Agent1 says 8.5, Agent2 says 9.0, Agent3 says 8.7. Using a debate pattern, which approach best selects the final rating?

hard

A. Pick the average rating of all agents

B. Randomly select any agent's rating

C. Select the rating from the agent with highest confidence

D. Choose the lowest rating to be safe

Solution

1. Step 1: Understand debate pattern goal

   Debate aims to compare opinions and pick the best based on confidence or quality.

2. Step 2: Identify best approach for final rating

   Choosing the rating from the agent with highest confidence aligns with debate selecting best opinion.

3. Final Answer:

   Select the rating from the agent with highest confidence -> Option C

4. Quick Check:

   Debate picks best confident opinion [OK]

Hint: Debate picks best confident opinion, not average [OK]

Common Mistakes:

• Averaging ratings (consensus, not debate)
• Picking lowest rating without reason
• Random selection ignoring confidence