Chains (sequential, router) in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Chains (sequential, router)

Which metric matters for Chains (sequential, router) and WHY

For chains that combine multiple models or steps, the key metric is overall accuracy or task success rate. This shows how well the entire chain completes the goal. For router chains, routing accuracy is also important to check if the right model is chosen for each input. We care about these because a chain is only as good as its weakest step or wrong routing.

Confusion matrix for router chain routing decisions

    | Actual Model Needed | Predicted Model Chosen |
    |---------------------|-----------------------|
    | Model A             | TP_A (correct)        |
    | Model B             | FP_A (wrongly chosen) |
    | Model B             | TP_B (correct)        |
    | Model A             | FP_B (wrongly chosen) |

    Total samples = TP_A + FP_A + TP_B + FP_B

    Precision for Model A = TP_A / (TP_A + FP_A)
    Recall for Model A = TP_A / (TP_A + FN_A)
    

This matrix helps measure if the router picks the right model for each input.

Precision vs Recall tradeoff in router chains

If the router has high precision but low recall for a model, it means it rarely picks that model wrongly but often misses inputs that need it. This can cause poor results if some inputs never reach the best model.

If recall is high but precision is low, the router picks the model often but sometimes wrongly, causing unnecessary processing or errors.

For sequential chains, a tradeoff is between speed and accuracy: adding more steps can improve accuracy but slow down the chain.

Good vs Bad metric values for Chains

• Good: Overall accuracy above 90%, router precision and recall above 85%, smooth step transitions without errors.
• Bad: Overall accuracy below 70%, router precision or recall below 50%, frequent step failures or wrong routing causing wrong outputs.

Common pitfalls in evaluating Chains

• Ignoring step errors: A chain may have good final accuracy but some steps fail silently, causing hidden issues.
• Data leakage: Training router or steps on overlapping data can inflate metrics falsely.
• Overfitting: Router or steps tuned too much on training data may fail on new inputs.
• Accuracy paradox: High accuracy can hide poor performance on rare but important cases.

Self-check question

Your router chain has 98% overall accuracy but only 12% recall for a critical model in the chain. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the router misses most inputs that need the critical model. This can cause many inputs to be handled incorrectly, hurting overall performance despite high accuracy.

Key Result

For chains, overall accuracy and router precision/recall are key to ensure correct step execution and routing.

Practice

1. What is the main purpose of a sequential chain in GenAI?

easy

A. To run all AI steps at the same time

B. To randomly select one AI step to run

C. To run multiple AI steps one after another in order

D. To stop the AI process after the first step

Solution

1. Step 1: Understand sequential chain behavior

   A sequential chain connects AI steps so they run one after another, passing output from one to the next.

2. Step 2: Compare options to definition

   Only To run multiple AI steps one after another in order describes running steps in order, matching the sequential chain purpose.

3. Final Answer:

   To run multiple AI steps one after another in order -> Option C

4. Quick Check:

   Sequential chain = run steps in order [OK]

Hint: Sequential means steps run one after another [OK]

Common Mistakes:

• Thinking sequential means random step selection
• Confusing sequential with parallel execution
• Assuming sequential chains stop early

2. Which of the following is the correct way to define a router chain in GenAI?

easy

A. router = SequentialChain(steps=[step1, step2])

B. router = RouterChain(steps=[step1, step2], router_function=choose_step)

C. router = RouterChain(steps=step1, step2)

D. router = ChainRouter(steps=[step1, step2])

Solution

1. Step 1: Recall router chain syntax

   A router chain requires a list of steps and a router function to decide which step to run.

2. Step 2: Check each option's syntax

   router = RouterChain(steps=[step1, step2], router_function=choose_step) correctly uses RouterChain with steps list and router_function parameter. Others have wrong class names or syntax.

3. Final Answer:

   router = RouterChain(steps=[step1, step2], router_function=choose_step) -> Option B

4. Quick Check:

   RouterChain needs steps list and router_function [OK]

Hint: RouterChain needs steps list and router_function param [OK]

Common Mistakes:

• Using SequentialChain instead of RouterChain
• Passing steps without brackets as list
• Using wrong class names like ChainRouter

3. Given the code below, what will be the output?

def router_func(input_text):
    if 'weather' in input_text.lower():
        return 'weather_step'
    else:
        return 'default_step'

steps = {
    'weather_step': lambda x: 'It is sunny',
    'default_step': lambda x: 'I do not understand'
}

router_chain = RouterChain(steps=steps, router_function=router_func)

result = router_chain.run('What is the weather today?')

medium

A. 'It is sunny'

B. 'I do not understand'

C. Error: router_function missing

D. 'What is the weather today?'

Solution

1. Step 1: Analyze router function behavior

   The router_func checks if 'weather' is in the input text (case-insensitive). Input contains 'weather', so it returns 'weather_step'.

2. Step 2: Determine which step runs

   The router_chain uses 'weather_step' key to run the lambda returning 'It is sunny'.

3. Final Answer:

   'It is sunny' -> Option A

4. Quick Check:

   Input contains 'weather' -> weather_step -> 'It is sunny' [OK]

Hint: Router picks step by input keyword match [OK]

Common Mistakes:

• Ignoring case in input text check
• Confusing step keys with output strings
• Assuming default_step runs always

4. Identify the error in this router chain code snippet:

steps = {
    'step1': lambda x: 'Hello',
    'step2': lambda x: 'Bye'
}

def router_func(input_text):
    if 'hello' in input_text:
        return 'step1'
    else:
        return 'step3'

router_chain = RouterChain(steps=steps, router_function=router_func)
result = router_chain.run('hello there')

medium

A. Lambda functions require two arguments

B. steps dictionary keys are not strings

C. router_function is missing in RouterChain

D. router_func returns a step key not in steps dictionary

Solution

1. Step 1: Check router_func return values

   router_func returns 'step1' if 'hello' in input, else 'step3'. Input contains 'hello', so returns 'step1'.

2. Step 2: Verify steps dictionary keys

   Steps dictionary has keys 'step1' and 'step2', but no 'step3'. Returning 'step3' would cause error if input changed.

3. Final Answer:

   router_func returns a step key not in steps dictionary -> Option D

4. Quick Check:

   Router returns unknown step key 'step3' [OK]

Hint: Router must return keys present in steps dict [OK]

Common Mistakes:

• Ignoring missing step keys in router return
• Assuming lambda needs multiple args
• Forgetting to pass router_function parameter

5. You want to build a GenAI system that first summarizes a text, then translates it to French, but only if the text is longer than 100 words. Which chain setup is best?

hard

A. Use a sequential chain with a router function that skips translation if text is short

B. Use a router chain that chooses between summarization or translation only

C. Use two separate sequential chains running independently

D. Use a sequential chain that always runs summarization then translation

Solution

1. Step 1: Understand task requirements

   The system must summarize first, then translate only if text is longer than 100 words.

2. Step 2: Choose chain type matching conditional flow

   A sequential chain with a router function can run summarization step first, then decide to run translation step based on text length.

3. Step 3: Evaluate other options

   Router chain alone can't enforce sequential order; two separate chains won't coordinate; always running translation ignores condition.

4. Final Answer:

   Use a sequential chain with a router function that skips translation if text is short -> Option A

5. Quick Check:

   Sequential + router for conditional step flow [OK]

Hint: Combine sequential steps with router for conditional logic [OK]

Common Mistakes:

• Using router chain alone without sequence
• Running translation always ignoring condition
• Splitting steps into independent chains