Caching strategies for cost reduction in LangChain - Mini Project: Build & Apply

Caching Strategies for Cost Reduction with LangChain

📖 Scenario: You are building a chatbot using LangChain that calls an expensive language model API. To save money, you want to store answers for repeated questions so the API is not called every time.

🎯 Goal: Build a simple LangChain setup that caches responses for repeated inputs to reduce API calls and costs.

📋 What You'll Learn

Create a dictionary called cache to store question-answer pairs

Create a variable called max_cache_size set to 3

Write a function get_answer(question) that checks cache before calling the API

Add logic to remove the oldest cached item when cache exceeds max_cache_size

💡 Why This Matters

🌍 Real World

Caching is used in chatbots and apps to save money and speed up responses by reusing previous answers.

💼 Career

Understanding caching helps developers optimize costs and improve performance in real-world software projects.

1

Create the cache dictionary

Create an empty dictionary called cache to store question-answer pairs.

LangChain

# Create an empty dictionary called cache
# Your code here

Need a hint?

Use {} to create an empty dictionary in Python.

2

Set the maximum cache size

Create a variable called max_cache_size and set it to 3.

LangChain

cache = {}
# Set max_cache_size to 3
# Your code here

Need a hint?

Just assign the number 3 to the variable max_cache_size.

3

Write the caching function

Write a function called get_answer(question) that does the following:
1. Checks if question is in cache and returns the cached answer if found.
2. Otherwise, calls a placeholder function call_api(question) to get the answer.
3. Adds the new question and answer to cache.

LangChain

cache = {}
max_cache_size = 3

# Define get_answer function
# Your code here

Need a hint?

Use if question in cache to check cache, else call call_api(question) and store result.

4

Limit cache size by removing oldest entry

Update the get_answer(question) function to remove the oldest cached item when the size of cache exceeds max_cache_size after adding a new answer.

LangChain

cache = {}
max_cache_size = 3

def call_api(question):
    # Placeholder for expensive API call
    return f"Answer to '{question}'"

def get_answer(question):
    if question in cache:
        return cache[question]
    answer = call_api(question)
    cache[question] = answer
    # Remove oldest cache item if cache is too big
    # Your code here
    return answer

Need a hint?

Use len(cache) to check size and next(iter(cache)) to get oldest key.