Large Language Models Inference API and Applications
Project description
Large Language Model (LLM) Inference API and Chatbot 🦙
Inference API for LLMs like LLaMA and Falcon powered by Lit-GPT from Lightning AI
pip install llm-inference
Install from main branch
pip install git+https://github.com/aniketmaurya/llm-inference.git@main
Note: You need to manually install Lit-GPT and setup the model weights to use this project.
pip install lit_gpt@git+https://github.com/aniketmaurya/install-lit-gpt.git@install
For Inference
from llm_inference import LLMInference, prepare_weights
from rich import print
path = prepare_weights("EleutherAI/pythia-70m")
model = LLMInference(checkpoint_dir=path)
print(model("New York is located in"))
For deploying as a REST API
Create a Python file app.py
and initialize the ServeLLaMA
App.
# app.py
from llm_inference.serve import ServeLLaMA, Response, PromptRequest
import lightning as L
component = ServeLLaMA(input_type=PromptRequest, output_type=Response)
app = L.LightningApp(component)
lightning run app app.py
How to use the Chatbot
from chatbot import LitGPTChatBot
checkpoint_dir = "weights"
bot = LitGPTChatBot(
checkpoint_dir=checkpoint_dir)
print(bot.send("hi, what is the capital of France?"))
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
llm_inference-0.0.5.dev0.tar.gz
(809.4 kB
view hashes)
Built Distribution
Close
Hashes for llm_inference-0.0.5.dev0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4030d66eb706db7ff8381bbdb9ed1502502401654487a136e73f6e15b50eddfe |
|
MD5 | f4b5b20ed1a990bfdd3ff434fdbf221e |
|
BLAKE2b-256 | a3e5934dc43fe58bb9d1eb54ec149963aaa932f004a435976ed5310c88f44fb8 |