Why observability is essential for LLM apps in LangChain - See It in Action

Why Observability is Essential for LLM Apps

📖 Scenario: You are building a simple LangChain app that uses a large language model (LLM) to answer questions. To make sure your app works well and you can fix problems quickly, you want to add observability features. Observability means you can see what is happening inside your app, like tracking inputs, outputs, and errors.

🎯 Goal: Build a basic LangChain app with observability by setting up the data, adding configuration for logging, implementing the core logic to run the LLM with logging, and completing the app with error handling and final logging setup.

📋 What You'll Learn

Create a LangChain LLM instance with a fixed model name

Add a configuration variable to enable logging

Use LangChain's callback manager to log inputs and outputs

Add error handling to log exceptions

💡 Why This Matters

🌍 Real World

Observability helps developers understand how their LLM apps behave in real time. It shows what inputs the model receives, what outputs it produces, and if any errors happen. This is like having a dashboard for your app's health.

💼 Career

Many companies use LLMs in production. Knowing how to add observability is important for maintaining, debugging, and improving these apps. It is a key skill for AI engineers and developers working with LangChain or similar frameworks.

1

Set up the LLM instance

Create a LangChain LLM instance called llm using OpenAI with the model name "gpt-3.5-turbo".

LangChain

# Your code here

Hint

Use ChatOpenAI from langchain.chat_models and set model_name to "gpt-3.5-turbo".

2

Add logging configuration

Create a boolean variable called enable_logging and set it to True to enable observability logging.

LangChain

from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(model_name="gpt-3.5-turbo")

# Your code here

Hint

Just create a variable enable_logging and set it to True.

3

Implement LLM call with logging

Import CallbackManager and StdOutCallbackHandler from langchain.callbacks. Create a callback_manager that uses StdOutCallbackHandler() only if enable_logging is True. Then create a new llm_with_logging instance of ChatOpenAI with the same model name and the callback_manager. Finally, call llm_with_logging with the prompt "What is observability?" and assign the result to response.

LangChain

from langchain.chat_models import ChatOpenAI
from langchain.callbacks import CallbackManager, StdOutCallbackHandler

llm = ChatOpenAI(model_name="gpt-3.5-turbo")
enable_logging = True

# Your code here

Hint

Use CallbackManager with StdOutCallbackHandler() only if enable_logging is True. Pass it to the new LLM instance.

4

Add error handling and finalize observability

Wrap the LLM call in a try block. If an exception occurs, catch it with except Exception as e and print "Error:" followed by the exception e. This completes the observability setup by logging errors.

LangChain

from langchain.chat_models import ChatOpenAI
from langchain.callbacks import CallbackManager, StdOutCallbackHandler

llm = ChatOpenAI(model_name="gpt-3.5-turbo")
enable_logging = True

callback_manager = CallbackManager([StdOutCallbackHandler()]) if enable_logging else None

llm_with_logging = ChatOpenAI(model_name="gpt-3.5-turbo", callback_manager=callback_manager)

# Your code here

Hint

Use a try block around the LLM call and catch exceptions to print errors.

Practice

1. Why is observability important in LangChain apps that use large language models (LLMs)?

easy

A. It makes the app run faster by skipping API calls.

B. It automatically writes the code for the app without user input.

C. It replaces the need for training the language model.

D. It helps track what happens inside the app to find errors and improve responses.

Solution

1. Step 1: Understand observability's role in LLM apps

   Observability means seeing inside the app's processes to understand behavior and issues.

2. Step 2: Connect observability to error detection and improvement

   By tracking app actions, developers can find errors and improve responses effectively.

3. Final Answer:

   It helps track what happens inside the app to find errors and improve responses. -> Option D

4. Quick Check:

   Observability = Track and improve app behavior [OK]

Hint: Observability means watching app actions to fix and improve [OK]

Common Mistakes:

• Thinking observability writes code automatically
• Believing observability replaces model training
• Assuming observability speeds up API calls

2. Which of the following is the correct way to add a callback for observability in a LangChain LLM chain?

easy

A. chain = LLMChain(llm=llm, prompt=prompt, handlers=MyCallbackHandler())

B. chain = LLMChain(llm=llm, prompt=prompt, callbacks=[MyCallbackHandler()])

C. chain = LLMChain(llm=llm, prompt=prompt, callback=MyCallbackHandler())

D. chain = LLMChain(llm=llm, prompt=prompt, observers=[MyCallbackHandler()])

Solution

1. Step 1: Recall LangChain callback syntax

   LangChain expects a list of callback handlers passed as the 'callbacks' parameter.

2. Step 2: Match correct parameter and value type

   chain = LLMChain(llm=llm, prompt=prompt, callbacks=[MyCallbackHandler()]) uses 'callbacks' with a list containing the handler instance, which is correct.

3. Final Answer:

   chain = LLMChain(llm=llm, prompt=prompt, callbacks=[MyCallbackHandler()]) -> Option B

4. Quick Check:

   Callbacks param = list of handlers [OK]

Hint: Callbacks parameter takes a list of handler instances [OK]

Common Mistakes:

• Using 'callback' instead of 'callbacks'
• Passing a single handler without list brackets
• Using wrong parameter names like 'handlers' or 'observers'

3. Given this LangChain code snippet, what will be printed when the chain runs?

from langchain.callbacks.base import BaseCallbackHandler

class PrintCallback(BaseCallbackHandler):
    def on_llm_start(self, serialized, prompts, **kwargs):
        print(f"LLM started with prompt: {prompts[0]}")

chain = LLMChain(llm=llm, prompt=prompt, callbacks=[PrintCallback()])
chain.run("Hello")

medium

A. LLM started with prompt: Hello

B. LLM started with prompt: ["Hello"]

C. No output printed

D. Error: on_llm_start method missing required arguments

Solution

1. Step 1: Understand the on_llm_start callback parameter

   The 'prompts' argument is a list of prompt strings, so prompts[0] is the first prompt string.

2. Step 2: Analyze the print statement output

   The print outputs the string with prompts[0], which is the string "Hello" passed to run, but wrapped in a list originally.

3. Final Answer:

   LLM started with prompt: Hello -> Option A

4. Quick Check:

   Print prompt string = "Hello" [OK]

Hint: Callbacks receive prompts as list; print first item for prompt text [OK]

Common Mistakes:

• Thinking prompts is a string, not a list
• Expecting no output from callback
• Confusing method parameters causing errors

4. You added a callback to your LangChain app but no logs appear when running the chain. What is the most likely cause?

medium

A. The prompt variable is empty.

B. The callback was added as a single object, not inside a list.

C. The callback class does not implement any event methods like on_llm_start.

D. The LLM model is not connected to the internet.

Solution

1. Step 1: Check callback implementation

   If the callback class lacks event methods like on_llm_start, no logs will be triggered.

2. Step 2: Verify callback registration

   Even if callbacks are registered correctly, without event methods, no output occurs.

3. Final Answer:

   The callback class does not implement any event methods like on_llm_start. -> Option C

4. Quick Check:

   Callbacks need event methods to log [OK]

Hint: Callbacks must implement event methods to produce logs [OK]

Common Mistakes:

• Assuming single object instead of list stops logs
• Blaming internet connection for no logs
• Thinking empty prompt causes no callback logs

5. You want to monitor both the input prompts and the cost of API calls in your LangChain app. Which observability approach best achieves this?

hard

A. Use a callback handler that logs prompts on on_llm_start and tracks token usage on on_llm_end.

B. Add print statements inside the prompt template and ignore callbacks.

C. Only log the final output text after the chain finishes.

D. Use a callback that only tracks errors during chain execution.

Solution

1. Step 1: Identify observability needs

   You want to see input prompts and monitor API call costs (token usage).

2. Step 2: Match callback events to needs

   on_llm_start can log prompts; on_llm_end can provide token usage info for cost tracking.

3. Step 3: Evaluate options

   Use a callback handler that logs prompts on on_llm_start and tracks token usage on on_llm_end. covers both prompt logging and cost monitoring via callbacks, which fits best.

4. Final Answer:

   Use a callback handler that logs prompts on on_llm_start and tracks token usage on on_llm_end. -> Option A

5. Quick Check:

   Callbacks on start/end = prompt + cost tracking [OK]

Hint: Use callbacks on start and end to log prompts and costs [OK]

Common Mistakes:

• Ignoring callbacks and using print statements only
• Logging only outputs misses input and cost info
• Tracking only errors misses prompt and cost data