Retry and fallback logic in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Retry and fallback logic

Which metric matters for Retry and fallback logic and WHY

For retry and fallback logic in AI systems, the key metrics are success rate and latency. Success rate shows how often the system recovers from failures by retrying or using fallback. Latency measures the time delay caused by retries or fallback steps. We want a high success rate to keep the system reliable, but also low latency so users don't wait too long. Balancing these metrics helps ensure the system is both dependable and fast.

Confusion matrix or equivalent visualization

Retry/Fallback Outcome Matrix:

| Outcome                     | Count |
|-----------------------------|-------|
| Success First Try (No Retry) | 800   |
| Success After Retry           | 150   |
| Success After Fallback        | 30    |
| Failure After All Attempts    | 20    |

Total Requests = 1000

Success Rate = (800 + 150 + 30) / 1000 = 0.98 (98%)
Failure Rate = 20 / 1000 = 0.02 (2%)

Average Latency = (800 * 1s + 150 * 3s + 30 * 5s + 20 * 5s) / 1000 = 1.5 seconds

Explanation:
- 1s = normal response time
- 3s = retry delay included
- 5s = fallback delay included
    

Precision vs Recall tradeoff with concrete examples

In retry and fallback logic, the tradeoff is between retry aggressiveness and system responsiveness.

• More retries: Increase success rate (like recall) by catching more failures, but increase latency (slow response).
• Fewer retries: Faster responses but risk more failures (lower success rate).

Example: A voice assistant that retries too much may respond correctly more often but annoy users with delays. If it retries less, it responds faster but may fail more.

What "good" vs "bad" metric values look like for retry and fallback logic

• Good: Success rate > 95%, average latency < 2 seconds. This means most requests succeed quickly.
• Bad: Success rate < 90%, average latency > 5 seconds. Many failures or long waits frustrate users.
• Warning: Success rate near 100% but latency very high (>10 seconds) means retries/fallbacks work but slow the system too much.

Metrics pitfalls

• Ignoring latency: High success rate alone can hide poor user experience if retries cause long delays.
• Data leakage: Using future information to decide retries can inflate success rate unrealistically.
• Overfitting retry logic: Tuning retries only on test data may fail in real-world diverse failures.
• Counting partial successes: Treating fallback partial results as full success can mislead metrics.

Self-check question

Your AI system has a 98% success rate but an average latency of 8 seconds due to many retries and fallbacks. Is this good for production? Why or why not?

Answer: No, because although the system succeeds often, the high latency means users wait too long. This hurts user experience and may cause frustration. You should reduce retries or optimize fallback to lower latency while keeping success rate high.

Key Result

High success rate with low latency is key to effective retry and fallback logic.

Practice

1.

What is the main purpose of retry logic in an AI system?

easy

A. To replace the task with a different unrelated task

B. To permanently stop a task after the first failure

C. To ignore errors and continue without any checks

D. To try a task multiple times to handle temporary failures

Solution

1. Step 1: Understand retry logic concept

   Retry logic means trying the same task again if it fails temporarily, like retrying a phone call if the line is busy.

2. Step 2: Match retry logic to options

   Only To try a task multiple times to handle temporary failures describes trying multiple times to handle temporary failures, which fits retry logic.

3. Final Answer:

   To try a task multiple times to handle temporary failures -> Option D

4. Quick Check:

   Retry logic = multiple attempts [OK]

Hint: Retry means try again after failure [OK]

Common Mistakes:

• Confusing retry with fallback
• Thinking retry stops after one failure
• Assuming retry changes the task

2.

Which of the following is the correct Python syntax to retry a function fetch_data() up to 3 times?

for _ in range(3):
    try:
        fetch_data()
        break
    except Exception:
        pass

easy

A. for _ in range(3): try: fetch_data() break except Exception: pass

B. for _ in range(3): fetch_data() break except Exception: pass

C. while True: fetch_data() break except Exception: pass

D. for i in range(3): fetch_data() except: break

Solution

1. Step 1: Check syntax for retry loop

   The code uses a for loop to try 3 times, with try-except to catch errors and break if successful.

2. Step 2: Identify correct syntax

   for _ in range(3): try: fetch_data() break except Exception: pass matches the correct Python syntax with try-except inside the loop and break on success.

3. Final Answer:

   for _ in range(3): try: fetch_data() break except Exception: pass -> Option A

4. Quick Check:

   Correct retry loop syntax = for _ in range(3): try: fetch_data() break except Exception: pass [OK]

Hint: Look for try-except inside a for loop with break [OK]

Common Mistakes:

• Missing try-except block
• Incorrect loop syntax
• Using 'except' without 'try'

3.

Consider this code snippet implementing retry and fallback logic:

def get_data():
    for _ in range(2):
        try:
            return fetch_from_primary()
        except Exception:
            pass
    return fetch_from_backup()

If fetch_from_primary() fails both times, what will get_data() return?

medium

A. The result of fetch_from_primary()

B. The result of fetch_from_backup()

C. None

D. An exception is raised

Solution

1. Step 1: Analyze retry attempts

   The function tries fetch_from_primary() twice inside the loop, catching exceptions and continuing if it fails.

2. Step 2: Understand fallback behavior

   If both retries fail, the function calls and returns fetch_from_backup() as a fallback.

3. Final Answer:

   The result of fetch_from_backup() -> Option B

4. Quick Check:

   Retries fail -> fallback used = The result of fetch_from_backup() [OK]

Hint: If retries fail, fallback result is returned [OK]

Common Mistakes:

• Assuming primary always returns result
• Ignoring fallback call
• Thinking exception propagates

4.

Identify the bug in this retry and fallback code snippet:

def get_info():
    for i in range(3):
        try:
            return fetch_data()
        except:
            continue
    return fallback_data()

medium

A. The except block catches all exceptions without specifying type

B. The function returns fallback_data() even if fetch_data() succeeds

C. The except block should raise the exception instead of continue

D. The loop variable i is unused and should be removed

Solution

1. Step 1: Review exception handling

   The except block catches all exceptions without specifying the exception type, which is bad practice and can hide bugs.

2. Step 2: Identify best practice

   It's better to catch specific exceptions to avoid masking unexpected errors.

3. Final Answer:

   The except block catches all exceptions without specifying type -> Option A

4. Quick Check:

   Catch specific exceptions, not all [OK]

Hint: Avoid bare except; specify exception type [OK]

Common Mistakes:

• Using bare except blocks
• Ignoring exception types
• Assuming unused variables cause bugs

5.

You want to design an AI agent that tries to fetch user data from a primary server up to 3 times. If all retries fail, it should fetch from a backup server. Which code snippet correctly implements this retry and fallback logic?

Option A:
for _ in range(3):
    try:
        data = fetch_primary()
    except:
        data = fetch_backup()
        break
Option B:
for _ in range(3):
    try:
        data = fetch_primary()
        break
    except:
        pass
else:
    data = fetch_backup()
Option C:
try:
    data = fetch_primary()
except:
    data = fetch_backup()
Option D:
while True:
    try:
        data = fetch_primary()
        break
    except:
        data = fetch_backup()
        break

hard

A. Retries primary, but fallback runs immediately on first failure

B. No retries, fallback runs immediately on first failure

C. Retries primary 3 times, then fallback if all fail

D. Retries once, fallback runs immediately after first failure

Solution

1. Step 1: Understand retry and fallback requirements

   The agent must retry fetching from primary 3 times, then fallback only if all retries fail.

2. Step 2: Analyze each option's behavior

   Retries primary 3 times, then fallback if all fail uses a for loop with try-except and an else clause that runs fallback only if loop completes without break (all retries failed). This matches requirements.

3. Final Answer:

   Retries primary 3 times, then fallback if all fail -> Option C

4. Quick Check:

   Retry 3 times + fallback after = Retries primary 3 times, then fallback if all fail [OK]

Hint: Use for-else to run fallback after retries fail [OK]

Common Mistakes:

• Running fallback too early
• Not retrying enough times
• Missing else clause for fallback