A library helps to chat with all kinds of LLMs consistently.
Project description
llmlite
A library helps to communicate with all kinds of LLMs consistently.
Model | State | System Prompt | Note |
---|---|---|---|
ChatGPT | Done ✅ | Yes | |
Llama-2 | Done ✅ | Yes | |
CodeLlama | Done ✅ | Yes | |
ChatGLM2 | Done ✅ | No | |
ChatGLM3 | WIP ⏳ | Yes | |
Baichuan2 | Done ✅ | Yes | |
Claude-2 | RoadMap 📋 | issue#7 | |
Falcon | RoadMap 📋 | issue#8 | |
StableLM | RoadMap 📋 | issue#11 | |
Baichuan2 | RoadMap 📋 | issue#34 | |
... | ... | ... | ... |
llmlite also supports different inference backends as below:
backend | State | Note |
---|---|---|
huggingface | Done ✅ | Support by huggingface pipeline |
vLLM | Done ✅ | |
... | ... | ... |
How to install
pip install llmlite==0.0.9
How to use
Chat
from llmlite.apis import ChatLLM, ChatMessage
chat = ChatLLM(
model_name_or_path="meta-llama/Llama-2-7b-chat-hf", # required
task="text-generation",
backend="vllm",
)
result = chat.completion(
messages=[
ChatMessage(role="system", content="You're a honest assistant."),
ChatMessage(role="user", content="There's a llama in my garden, what should I do?"),
]
)
# Output: Oh my goodness, a llama in your garden?! 😱 That's quite a surprise! 😅 As an honest assistant, I must inform you that llamas are not typically known for their gardening skills, so it's possible that the llama in your garden may have wandered there accidentally or is seeking shelter. 🐮 ...
llmlite
also supports other parameters like temperature
, max_length
, do_sample
, top_k
, top_p
to help control the length, randomness and diversity of the generated text.
See examples for reference.
Prompting
You can use llmlite
to help you generate full prompts, for instance:
from llmlite.apis import ChatMessage, LlamaChat
messages = [
ChatMessage(role="system", content="You're a honest assistant."),
ChatMessage(role="user", content="There's a llama in my garden, what should I do?"),
]
LlamaChat.prompt(messages)
# Output:
# <s>[INST] <<SYS>>
# You're a honest assistant.
# <</SYS>>
# There's a llama in my garden, what should I do? [/INST]
Logging
Set the env variable LOG_LEVEL
for log configuration, default to INFO
, others like DEBUG, INFO, WARNING etc..
Roadmap
- Adapter support
- Quantization
- Streaming
Contributions
🚀 All kinds of contributions are welcomed ! Please follow Contributing.
Contributors
🎉 Thanks to all these contributors.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.