Large Language Model (LLM) Inference API and Chatbot 🦙
Inference API for LLMs like LLaMA and Falcon powered by Lit-GPT from Lightning AI
pip install llm-inference
Install from main branch
pip install git+https://github.com/aniketmaurya/llm-inference.git@main
Note: You need to manually install Lit-GPT and setup the model weights to use this project.
pip install lit_gpt@git+https://github.com/aniketmaurya/install-lit-gpt.git@install
For Inference
from llm_inference import LLMInference, prepare_weights
from rich import print
path = prepare_weights("EleutherAI/pythia-70m")
model = LLMInference(checkpoint_dir=path)
print(model("New York is located in"))
How to use the Chatbot
from llm_chain import LitGPTConversationChain, LitGPTLLM
from llm_inference import prepare_weights
from rich import print
path = str(prepare_weights("lmsys/longchat-13b-16k"))
llm = LitGPTLLM(checkpoint_dir=path, quantize="bnb.nf4") # 8.4GB GPU memory
bot = LitGPTConversationChain.from_llm(llm=llm, verbose=True)
print(bot.send("hi, what is the capital of France?"))
Launch Chatbot App
1. Download weights
from llm_inference import prepare_weights
path = prepare_weights("lmsys/longchat-13b-16k")
2. Launch Gradio App
python examples/chatbot/gradio_demo.py
For deploying as a REST API
Create a Python file app.py and initialize the ServeLLaMA App.
# app.py
from llm_inference.serve import ServeLLaMA, Response, PromptRequest
import lightning as L
component = ServeLLaMA(input_type=PromptRequest, output_type=Response)
app = L.LightningApp(component)
lightning run app app.py
Metadata
Release files for llm-inference 0.0.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_inference-0.0.6.tar.gz | 810.4 kB | Without provenance | Details for llm_inference-0.0.6.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llm_inference-0.0.6-py3-none-any.whl | Python 3 | none | any | Details for llm_inference-0.0.6-py3-none-any.whl |
Total release size: 822.4 kB
Release files / llm_inference-0.0.6.tar.gz
| Download URL | llm_inference-0.0.6.tar.gz |
|---|---|
| Size | 810.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5bc77680d1df2f9af5b4cc9187fd47c473a2c98a32d0c59eb7427949ad19cbfe
|
|
BLAKE2b-256 checksum How to use checksums |
541756ce7b12de3af15ae7acf9358760ad9689e36392da2491888fa7299ad6b5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.17
|
Release files / llm_inference-0.0.6-py3-none-any.whl
| Download URL | llm_inference-0.0.6-py3-none-any.whl |
|---|---|
| Size | 12.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
08fe641c4511b1ad465a503c02b58752cf785ee00f4516a3b63e99103d1ea6ac
|
|
BLAKE2b-256 checksum How to use checksums |
c6cd4bfeab074dd703d4977641711c16e5f4641d550c55d2890b797c71f90af7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.17
|
Security for llm-inference 0.0.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.