llm-inference 0.0.4

Large Language Models Inference API and Applications

Large Language Model (LLM) Inference API and Chatbot 🦙

Inference API for LLaMA

pip install llm-inference

# to use chatbot
pip install llm-inference[chatbot]

Install from main branch

pip install git+https://github.com/aniketmaurya/llm-inference.git@main

Note: You need to manually install Lit-GPT and setup the model weights to use this project.

pip install lit-gpt@git+https://github.com/Lightning-AI/lit-gpt.git@main

For Inference

from llm_inference import LLMInference
import os

WEIGHTS_PATH = os.environ["WEIGHTS"]

checkpoint_dir = f"checkpoints/tiiuae/falcon-7b"

model = LLMInference(checkpoint_dir=checkpoint_dir, precision="bf16-true")

print(model("New York is located in"))

For deploying as a REST API

Create a Python file app.py and initialize the ServeLLaMA App.

# app.py
from llm_inference.serve import ServeLLaMA, Response, PromptRequest

import lightning as L

component = ServeLLaMA(input_type=PromptRequest, output_type=Response)
app = L.LightningApp(component)

lightning run app app.py

How to use the Chatbot

from chatbot import LLaMAChatBot

checkpoint_dir = "weights"

bot = LLaMAChatBot(
    checkpoint_dir=checkpoint_dir)

print(bot.send("hi, what is the capital of France?"))