LangServe for API deployment in LangChain

Introduction

LangServe helps you turn your language models into easy-to-use APIs. It makes sharing and using AI models simple without complex setup.

You want to let others use your language model through a web API.

You need a quick way to deploy a chatbot or text generator online.

You want to test your language model with different inputs remotely.

You want to build an app that talks to your language model over the internet.

Syntax

LangChain

from langchain.serving import LangServe

app = LangServe(llm=your_language_model)

app.run(host='0.0.0.0', port=8000)

LangServe wraps your language model into a web server.

You can customize the host and port to control where the API listens.

Examples

This example creates a LangServe API using OpenAI's GPT-4 model and runs it on default settings.

LangChain

from langchain.serving import LangServe
from langchain.llms import OpenAI

llm = OpenAI(model_name='gpt-4')
app = LangServe(llm=llm)
app.run()

Here, a simple custom model echoes the prompt. LangServe runs it on port 9000.

LangChain

from langchain.serving import LangServe

class MyModel:
    def __call__(self, prompt):
        return f"Echo: {prompt}"

app = LangServe(llm=MyModel())
app.run(port=9000)

Sample Program

This program creates a simple API that replies with the input text prefixed by 'You said:'. It runs locally on port 8080.

LangChain

from langchain.serving import LangServe

class SimpleEchoModel:
    def __call__(self, prompt: str) -> str:
        return f"You said: {prompt}"

app = LangServe(llm=SimpleEchoModel())

if __name__ == '__main__':
    app.run(host='127.0.0.1', port=8080)

Important Notes

Make sure your language model class has a __call__ method that takes a prompt string and returns a string.

Use host='0.0.0.0' to allow external access to your API.

Check your firewall or network settings if others cannot reach your API.

Summary

LangServe quickly turns language models into web APIs.

You can deploy your model locally or on a server for others to use.

It requires a simple class with a __call__ method to work.