LiteChat 🚀

LiteChat is a lightweight, OpenAI-compatible interface for running local LLM inference servers. It provides seamless integration with various open-source models while maintaining OpenAI-style API compatibility.

Features ✨

🔄 OpenAI API compatibility
🌐 Web search integration
💬 Conversation memory
🔄 Streaming responses
🛠️ Easy integration with HuggingFace models
📦 Compatible with both litellm and OpenAI clients
🎯 Type-safe model selection

Installation 🛠️

pip install litechat playwright
playwright install

Available Models 🤖

LiteChat supports the following models:

Qwen/Qwen2.5-Coder-32B-Instruct: Specialized coding model
Qwen/Qwen2.5-72B-Instruct: Large general-purpose model
meta-llama/Llama-3.3-70B-Instruct: Latest Llama 3 model
CohereForAI/c4ai-command-r-plus-08-2024: Cohere's command model
Qwen/QwQ-32B-Preview: Preview version of QwQ
nvidia/Llama-3.1-Nemotron-70B-Instruct-HF: NVIDIA's Nemotron model
meta-llama/Llama-3.2-11B-Vision-Instruct: Vision-capable Llama model
NousResearch/Hermes-3-Llama-3.1-8B: Lightweight Hermes model
mistralai/Mistral-Nemo-Instruct-2407: Mistral's instruction model
microsoft/Phi-3.5-mini-instruct: Microsoft's compact Phi model

Model Selection Helpers 🎯

LiteChat provides helper functions for type-safe model selection:

from litechat import litechat_model, litellm_model

# For use with LiteChat native client
model = litechat_model("Qwen/Qwen2.5-72B-Instruct")

# For use with LiteLLM
model = litellm_model("Qwen/Qwen2.5-72B-Instruct")  # Returns "openai/Qwen/Qwen2.5-72B-Instruct"

Quick Start 🚀

Starting the Server

You can start the LiteChat server in two ways:

Using the CLI:

litechat_server

Programmatically:

from litechat import litechat_server

if __name__ == "__main__":
    litechat_server(host="0.0.0.0", port=11437)

Using with OpenAI Client

import os
from openai import OpenAI

os.environ['OPENAI_BASE_URL'] = "http://localhost:11437/v1"
os.environ['OPENAI_API_KEY'] = "key123" # required, but not used

client = OpenAI()
response = client.chat.completions.create(
    model=litechat_model("NousResearch/Hermes-3-Llama-3.1-8B"),
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "What is the capital of France?"}
    ]
)
print(response.choices[0].message.content)

Using with LiteLLM

import os

from litellm import completion
from litechat import OPENAI_COMPATIBLE_BASE_URL, litellm_model

os.environ["OPENAI_API_KEY"] = "key123"

response = completion(
    model=litellm_model("NousResearch/Hermes-3-Llama-3.1-8B"),
    messages=[{"content": "Hello, how are you?", "role": "user"}],
    api_base=OPENAI_COMPATIBLE_BASE_URL
)
print(response)

Using LiteChat's Native Client

from litechat import completion, genai, pp_completion
from litechat import litechat_model

# Basic completion
response = completion(
    prompt="What is quantum computing?",
    model="nvidia/Llama-3.1-Nemotron-70B-Instruct-HF",
    web_search=True  # Enable web search
)

# Stream with pretty printing
pp_completion(
    prompt="Explain the theory of relativity",
    model="Qwen/Qwen2.5-72B-Instruct",
    conversation_id="physics_chat"  # Enable conversation memory
)

# Get direct response
result = genai(
    prompt="Write a poem about spring",
    model="meta-llama/Llama-3.3-70B-Instruct",
    system_prompt="You are a creative poet"
)

Advanced Features 🔧

Web Search Integration

Enable web search to get up-to-date information:

response = completion(
    prompt="What are the latest developments in AI?",
    web_search=True
)

Conversation Memory

Maintain context across multiple interactions:

response = completion(
    prompt="Tell me more about that",
    conversation_id="unique_conversation_id"
)

Streaming Responses

Get token-by-token streaming:

for chunk in completion(
    prompt="Write a long story",
    stream=True
):
    print(chunk.choices[0].delta.content, end="", flush=True)

API Reference 📚

LiteAI Client

from litechat import LiteAI, litechat_model

client = LiteAI(
    api_key="optional-key",  # Optional API key
    base_url="http://localhost:11437",  # Server URL
    system_prompt="You are a helpful assistant",  # Default system prompt
    web_search=False,  # Enable/disable web search by default
    model=litechat_model("nvidia/Llama-3.1-Nemotron-70B-Instruct-HF")  # Default model
)

Completion Function Parameters

messages: List of conversation messages or direct prompt string
model: HuggingFace model identifier (use litechat_model() for type safety)
system_prompt: System instruction for the model
temperature: Control randomness (0.0 to 1.0)
stream: Enable streaming responses
web_search: Enable web search
conversation_id: Enable conversation memory
max_tokens: Maximum tokens in response
tools: List of available tools/functions

Contributing 🤝

Contributions are welcome! Please feel free to submit a Pull Request.

License 📄

This project is licensed under the MIT License - see the LICENSE file for details.

Support 💬

For support, please open an issue on the GitHub repository or reach out to the maintainers.

Metadata

Release files for litechat 0.0.69

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for litechat 0.0.69
File	Size	Uploaded	Provenance
litechat-0.0.69.tar.gz	66.2 kB	Feb 2, 2025	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for litechat 0.0.69
File	Interpreter	ABI	Platform	Reset
litechat-0.0.69-py3-none-any.whl 32.3 kB Feb 2, 2025 Without provenance	Python 3	none	any	Details

Total release size: 98.6 kB

Release files / litechat-0.0.69.tar.gz

Download URL	litechat-0.0.69.tar.gz
Size	66.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`e11c8d01f1f0701758534914cc8e8f5386e7ba18560943824d3734bfe9ef643c`
BLAKE2b-256 checksum How to use checksums	`a291a738488211d658679e4d0b27b3110df6664bf017f5de12f86bf16c21d841`
Upload date	Feb 2, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.8.4 CPython/3.11.0rc1 Linux/6.8.0-52-generic`

Release files / litechat-0.0.69-py3-none-any.whl

Download URL	litechat-0.0.69-py3-none-any.whl
Size	32.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`c82c0000fa27438ce4e2e3b82fcebb8a03cd1bfd965bb303e2407ec08a03c3ed`
BLAKE2b-256 checksum How to use checksums	`0de11b3235510dd20c52e663071219474369ec50aa7002912a00ba44cacb28c0`
Upload date	Feb 2, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.8.4 CPython/3.11.0rc1 Linux/6.8.0-52-generic`

Security for litechat 0.0.69

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.