Skip to main content

A library helps to chat with all kinds of LLMs consistently.

Project description

llmlite

A library helps to communicate with all kinds of LLMs consistently.

Model State System Prompt Note
ChatGPT Done ✅ Yes
Llama-2 Done ✅ Yes
CodeLlama Done ✅ Yes
ChatGLM2 Done ✅ No
ChatGLM3 WIP ⏳ Yes
Baichuan2 Done ✅ Yes
Claude-2 RoadMap 📋 issue#7
Falcon RoadMap 📋 issue#8
StableLM RoadMap 📋 issue#11
Baichuan2 RoadMap 📋 issue#34
... ... ... ...

llmlite also supports different inference backends as below:

backend State Note
huggingface Done ✅ Support by huggingface pipeline
vLLM Done ✅
... ... ...

How to install

pip install llmlite==0.0.9

How to use

Chat

from llmlite.apis import ChatLLM, ChatMessage

chat = ChatLLM(
    model_name_or_path="meta-llama/Llama-2-7b-chat-hf", # required
    task="text-generation",
    backend="vllm",
    )

result = chat.completion(
  messages=[
    ChatMessage(role="system", content="You're a honest assistant."),
    ChatMessage(role="user", content="There's a llama in my garden, what should I do?"),
  ]
)

# Output: Oh my goodness, a llama in your garden?! 😱 That's quite a surprise! 😅 As an honest assistant, I must inform you that llamas are not typically known for their gardening skills, so it's possible that the llama in your garden may have wandered there accidentally or is seeking shelter. 🐮 ...

llmlite also supports other parameters like temperature, max_length, do_sample, top_k, top_p to help control the length, randomness and diversity of the generated text.

See examples for reference.

Prompting

You can use llmlite to help you generate full prompts, for instance:

from llmlite.apis import ChatMessage, LlamaChat

messages = [
    ChatMessage(role="system", content="You're a honest assistant."),
    ChatMessage(role="user", content="There's a llama in my garden, what should I do?"),
]

LlamaChat.prompt(messages)

# Output:
# <s>[INST] <<SYS>>
# You're a honest assistant.
# <</SYS>>

# There's a llama in my garden, what should I do? [/INST]

Logging

Set the env variable LOG_LEVEL for log configuration, default to INFO, others like DEBUG, INFO, WARNING etc..

Roadmap

  • Adapter support
  • Quantization
  • Streaming

Contributions

🚀 All kinds of contributions are welcomed ! Please follow Contributing.

Contributors

🎉 Thanks to all these contributors.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llmlite-0.0.15.tar.gz (11.1 kB view hashes)

Uploaded Source

Built Distribution

llmlite-0.0.15-py3-none-any.whl (16.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page