chatstream 0.2.0

A streaming chat toolkit for pre-trained large language models(LLM)

ChatStream

English | 日本語

ChatStream is a chat toolkit for pre-trained large language models.

It can be embedded in FastAPI/Starlette based web applications/web APIs to perform sequential sentence generation with pre-trained language models under load control.

Installation

pip install chatstream

Quick Start

Install required packages

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117
pip install transformers
pip install "uvicorn[standard]" gunicorn 

Implementing a ChatStream server

Implement a streaming chat server for pre-trained models.

import torch
from fastapi import FastAPI, Request
from fastsession import FastSessionMiddleware, MemoryStore
from transformers import AutoTokenizer, AutoModelForCausalLM

from chatstream import ChatStream,ChatPromptTogetherRedPajamaINCITEChat as ChatPrompt

model_path = "togethercomputer/RedPajama-INCITE-Chat-3B-v1"
device = "cuda" # "cuda" / "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16)
model.to(device)

chat_stream = ChatStream(
    num_of_concurrent_executions=2,# max_concurrent_executions for sentence generation
    max_queue_size=5,# size of queue
    model=model,
    tokenizer=tokenizer,
    device=device,
    chat_prompt_clazz=ChatPrompt,
)

app = FastAPI()

# Specify session middleware to keep per-user ChatPrompt in the HTTP session
app.add_middleware(FastSessionMiddleware,
                   secret_key="your-session-secret-key",
                   store=MemoryStore(),
                   http_only=True,
                   secure=False,
                   )


@app.post("/chat_stream")
async def stream_api(request: Request):.
    # Just pass a FastAPI Request object to `handle_starlette_request` to automatically queue and control concurrency
    response = await chat_stream.handle_starlette_request(request)
    return response


@app.on_event("startup")
async def startup():.
    # start the queueing system by doing `start_queue_worker` at the same time the web server starts up
    await chat_stream.start_queue_worker()

Table of Contents

• What is ChatStream

• -- Importing Prompt Class ChatPrompt

• -- Loading model classes

• -- HTTP session middleware configuration

• -- Create and initialize ChatStream

• Implementation of Web API Endpoints

  • -- Endpoint Implementation
  • -- Receive streaming transmission completion callback
  • -- Read requests from users
  • -- How to set up an HTTP session

• Queueing System and Concurrency Limit

  • -- What is the queueing system
  • -- Starting the Queueing System

• Start the Web server (ASGI server)

  • -- uvicorn (start from inside)
  • -- uvicorn (start from outside)
  • -- gunicorn

• Console chat implementation

  • -- Run a simple console chat to check the model behavior

• Configuration during development

  • -- CORS middleware settings
  • -- Using Mock Response (Fast Startup)
  • -- Logging Settings
  • -- Reading requests from users
  • -- Attach progress bar to time-consuming model loading

• Advanced Settings

  • Chat History Persistence
    • - Implement custom request handler
  • Configuration for large scale access
    • Interfacing with login authentication using OAuth
    • - Load Balancing on Multi-GPU
    • - Load Balancing with Multi-GPU Server