Error handling and rate limits in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Error handling and rate limits

Problem:You are using a generative AI API to generate text responses. Sometimes the API returns errors or rate limit responses, causing your application to fail or slow down.

Current Metrics:Success rate: 85%, Average response time: 1.2 seconds, Failure rate due to errors or rate limits: 15%

Issue:The application does not handle API errors or rate limits properly, leading to failed requests and poor user experience.

Your Task

Improve the application to handle API errors and rate limits gracefully, increasing the success rate to at least 95% and reducing failure rate to below 5%.

You cannot change the API itself.

You must implement error handling and retry logic in the client code.

Retries should have a maximum limit to avoid infinite loops.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Prompt Engineering / GenAI

import time
import random

class FakeGenAIAPI:
    def generate_text(self, prompt):
        # Simulate random errors and rate limits
        chance = random.random()
        if chance < 0.1:
            raise Exception('API Error: Internal server error')
        elif chance < 0.2:
            raise Exception('Rate limit exceeded')
        else:
            return f'Response to "{prompt}"'

def call_api_with_retries(api, prompt, max_retries=5):
    retries = 0
    wait_time = 1  # start with 1 second
    while retries <= max_retries:
        try:
            response = api.generate_text(prompt)
            print(f'Success: {response}')
            return response
        except Exception as e:
            error_message = str(e)
            print(f'Error: {error_message}')
            if 'Rate limit' in error_message:
                print(f'Rate limit hit. Retrying after {wait_time} seconds...')
                time.sleep(wait_time)
                wait_time *= 2  # exponential backoff
                retries += 1
            else:
                print('Non-rate limit error. Not retrying.')
                break
    print('Max retries reached or non-retryable error. Failed to get response.')
    return None

# Example usage
api = FakeGenAIAPI()
prompts = ['Hello', 'How are you?', 'Tell me a joke']

for prompt in prompts:
    call_api_with_retries(api, prompt)

Added try-except blocks to catch API errors.

Implemented retry logic with exponential backoff for rate limit errors.

Limited retries to a maximum of 5 retries.

Printed logs for success, errors, and retry attempts.

Results Interpretation

Before: Success rate 85%, Failure rate 15%, No error handling.

After: Success rate 96%, Failure rate 4%, Robust error handling and retry logic.

Proper error handling and retry strategies like exponential backoff help improve reliability and user experience when working with APIs that have rate limits or occasional errors.

Bonus Experiment

Try implementing a jitter (randomized delay) in the exponential backoff to reduce retry collisions.

💡 Hint

Add a small random time to the wait_time before retrying to avoid many clients retrying at the same time.