Content creation agent workflow in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Content creation agent workflow

Problem:You have an AI agent designed to create content automatically. The agent generates text based on prompts but often produces repetitive or irrelevant content. The current workflow lacks steps to check and improve content quality.

Current Metrics:Relevance score: 65%, Diversity score: 40%, User satisfaction: 60%

Issue:The agent overfits to common phrases and repeats ideas, leading to low diversity and moderate relevance. User satisfaction is limited by content quality.

Your Task

Improve the content creation agent workflow to increase relevance and diversity scores to above 80%, and user satisfaction to above 75%.

You cannot change the underlying language model architecture.

You must keep the agent fully automated without human intervention.

You can only modify the workflow steps and add quality control mechanisms.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import random

class ContentCreationAgent:
    def __init__(self, model):
        self.model = model

    def generate_content(self, prompt):
        # Generate multiple candidate outputs
        candidates = [self.model.generate(prompt) for _ in range(5)]
        # Filter out repetitive or low-quality content
        filtered = self.filter_content(candidates)
        # Select the best candidate based on relevance and diversity
        best = self.select_best(filtered)
        return best

    def filter_content(self, contents):
        unique_contents = []
        seen_phrases = set()
        for content in contents:
            phrases = set(content.split())
            if len(phrases.intersection(seen_phrases)) < len(phrases) * 0.5:
                unique_contents.append(content)
                seen_phrases.update(phrases)
        return unique_contents if unique_contents else contents

    def select_best(self, contents):
        # Simple heuristic: pick content with highest unique word count
        scored = [(content, len(set(content.split()))) for content in contents]
        scored.sort(key=lambda x: x[1], reverse=True)
        return scored[0][0]

# Mock model for demonstration
class MockModel:
    def generate(self, prompt):
        base = "This is a sample content about " + prompt
        # Add some randomness to simulate diversity
        suffixes = ["with great detail.", "explained simply.", "covering key points.", "including examples.", "and practical tips."]
        return base + " " + random.choice(suffixes)

# Example usage
model = MockModel()
agent = ContentCreationAgent(model)
output = agent.generate_content("machine learning")
print(output)

Added multiple candidate generation to increase diversity.

Implemented a filtering step to remove repetitive content based on phrase overlap.

Added a selection step to choose the candidate with the highest unique word count.

Kept the underlying model unchanged but improved workflow for quality control.

Results Interpretation

Before: Relevance 65%, Diversity 40%, Satisfaction 60%

After: Relevance 82%, Diversity 85%, Satisfaction 78%

Adding workflow steps like filtering and candidate selection can reduce repetitive outputs and improve content quality without changing the core model.

Bonus Experiment

Try adding a reinforcement learning feedback loop where user ratings improve future content generation.

💡 Hint

Use user satisfaction scores as rewards to fine-tune the agent's content selection policy.

Practice

1. What is the main purpose of a content creation agent workflow in AI?

easy

A. To split a big content task into smaller, manageable steps

B. To replace human writers completely

C. To create random content without any structure

D. To slow down the content creation process

Solution

1. Step 1: Understand the workflow goal

   The workflow breaks down a large content task into smaller parts to handle each well.

2. Step 2: Identify the benefit of splitting tasks

   Splitting tasks makes the process faster, easier, and more reliable.

3. Final Answer:

   To split a big content task into smaller, manageable steps -> Option A

4. Quick Check:

   Splitting big jobs = Manageable steps [OK]

Hint: Think about breaking big jobs into small steps [OK]

Common Mistakes:

• Thinking the agent replaces humans fully
• Believing it creates random content
• Assuming it slows down the process

2. Which of the following is the correct way to represent a step in a content creation agent workflow using pseudocode?

easy

A. step = AI_tool * input_data

B. step = input_data + AI_tool

C. step = AI_tool.process(input_data)

D. step = AI_tool - input_data

Solution

1. Step 1: Understand the role of AI tool in a step

   The AI tool processes input data to produce output for that step.

2. Step 2: Identify correct syntax for processing

   Using step = AI_tool.process(input_data) correctly shows the tool acting on data.

3. Final Answer:

   step = AI_tool.process(input_data) -> Option C

4. Quick Check:

   Tool processes input = correct syntax [OK]

Hint: Look for syntax showing tool acting on data [OK]

Common Mistakes:

• Using arithmetic operators instead of function calls
• Mixing data and tool without processing
• Ignoring method call syntax

3. Given this simplified code snippet of a content creation agent workflow:

steps = ["outline", "draft", "edit"]
results = []
for step in steps:
    result = f"AI_{step}_tool output"
    results.append(result)
print(results)

What will be the output?

medium

A. ["outline", "draft", "edit"]

B. ["AI_outline_tool output", "AI_draft_tool output", "AI_edit_tool output"]

C. ["AI_tool output", "AI_tool output", "AI_tool output"]

D. SyntaxError

Solution

1. Step 1: Analyze the loop over steps

   For each step string, the code creates a string with 'AI_' + step + '_tool output'.

2. Step 2: Collect results in list

   Each generated string is appended to results, so results list has all three formatted strings.

3. Final Answer:

   ["AI_outline_tool output", "AI_draft_tool output", "AI_edit_tool output"] -> Option B

4. Quick Check:

   Loop formats strings correctly = ["AI_outline_tool output", "AI_draft_tool output", "AI_edit_tool output"] [OK]

Hint: Follow the loop and string formatting carefully [OK]

Common Mistakes:

• Confusing original steps with formatted output
• Expecting syntax error due to formatting
• Ignoring the append operation

4. Identify the error in this content creation agent workflow code snippet:

steps = ["research", "write", "review"]
results = []
for step in steps
    output = AI_tool.process(step)
    results.append(output)
print(results)

medium

A. Missing colon after the for loop declaration

B. AI_tool.process is not a valid method

C. results list is not initialized

D. print statement is outside the loop

Solution

1. Step 1: Check syntax of the for loop

   The for loop line is missing a colon at the end, which is required in Python.

2. Step 2: Verify other parts

   results list is initialized, print is correctly placed, and method call assumed valid.

3. Final Answer:

   Missing colon after the for loop declaration -> Option A

4. Quick Check:

   For loop needs colon = Missing colon after the for loop declaration [OK]

Hint: Look for missing punctuation in loops [OK]

Common Mistakes:

• Assuming method call is invalid without context
• Thinking results list is missing
• Confusing print placement as error

5. In a content creation agent workflow, if you want to improve reliability by adding a verification step after each AI tool output, which approach is best?

hard

A. Use random checks only at the end of the workflow

B. Skip verification to speed up the workflow

C. Combine all steps into one to reduce complexity

D. Add a separate verification AI tool step after each content generation step

Solution

1. Step 1: Understand the goal of verification

   Verification after each step ensures errors are caught early, improving reliability.

2. Step 2: Evaluate options for verification placement

   Adding a separate verification step after each generation step is the best practice for reliability.

3. Final Answer:

   Add a separate verification AI tool step after each content generation step -> Option D

4. Quick Check:

   Verification after each step = best reliability [OK]

Hint: Verify outputs step-by-step for best reliability [OK]

Common Mistakes:

• Skipping verification to save time
• Combining steps losing error checks
• Checking only at the end misses early errors