langchain-llama-server 0.1.2

LangChain integrations for llama-server inference

langchain-llama-server

This LangChain integration supports llama-server for remote inference over HTTP/S.

Normally you can use ChatOpenAI from langchain-openai for OpenAI compatible APIs. But, anything beyond the OpenAI spec is not supported. Hence why this integration exists.

This package was created to fill in the gaps between the OpenAI spec and extra fields llama-server returns:

• Access reasoning_content on both streaming and non-streaming chat completions.
• Capture timings, __verbose and other llama-server specific response fields.
• Uses BaseChatOpenAI from langchain-openai to implement a chat_model called ChatLlamaServer

Similar to other server integrations:

• ChatOllama
• ChatXAI
  • ChatXAI also works with llama-server for reasoning_content because they both implement the OpenAI spec closely
• ChatOpenRouter

TODO

• LlamaServerLLM ?
• LlamaServerEmbedding ?

What about ChatLlamaCpp?

The langchain_community package provides ChatLlamaCpp? This chat model targets in-process hosting using llama.cpp. It doesn't support llama-server for remote hosting.