Large Language Models Inference API and Applications
Project description
Large Language Model (LLM) Inference API and Chatbot 🦙
Inference API for LLaMA
pip install llm-inference
# to use chatbot
pip install llm-inference[chatbot]
Install from main branch
pip install git+https://github.com/aniketmaurya/llm-inference.git@main
Note: You need to manually install Lit-GPT and setup the model weights to use this project.
pip install lit-gpt@git+https://github.com/Lightning-AI/lit-gpt.git@main
For Inference
from llm_inference import LLMInference
import os
WEIGHTS_PATH = os.environ["WEIGHTS"]
checkpoint_dir = f"checkpoints/tiiuae/falcon-7b"
model = LLMInference(checkpoint_dir=checkpoint_dir, precision="bf16-true")
print(model("New York is located in"))
For deploying as a REST API
Create a Python file app.py
and initialize the ServeLLaMA
App.
# app.py
from llm_inference.serve import ServeLLaMA, Response, PromptRequest
import lightning as L
component = ServeLLaMA(input_type=PromptRequest, output_type=Response)
app = L.LightningApp(component)
lightning run app app.py
How to use the Chatbot
from chatbot import LLaMAChatBot
checkpoint_dir = "weights"
bot = LLaMAChatBot(
checkpoint_dir=checkpoint_dir)
print(bot.send("hi, what is the capital of France?"))
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
llm_inference-0.0.0.tar.gz
(414.6 kB
view hashes)
Built Distribution
Close
Hashes for llm_inference-0.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6c263c8b3211c7bf70a3a97c06d50b783a578c0d490c06abeb91ae82aec99fb |
|
MD5 | fca0ab9a8ff618d015b09c3965578966 |
|
BLAKE2b-256 | b589522eac695151e3c829b40c8717e04615c2f4321bd1568fec0ca50060a1cf |