fastspeed 0.1.1

Fastspeed is a toolkit for serving LLM.

Fastspeed

Fastspeed is a toolkit for serving LLM.

Requirements

• Python >= 3.6

Installation

pip install fastspeed

Example

• server

fastspeed --model model_name_or_path  # default model_hub=modelscope
fastspeed --model model_name_or_path --model_hub hf

• client

curl -X 'POST' \
  'http://127.0.0.1:8000/chat/completions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "model": ""
}'

Change Log

0.1.1 (2024-09-15)

• Suport half, cuda and trust_remote_code=True

0.1.0 (2024-08-29)

• Fastapi with single GPU, model from huggingface or modelscope