The inference engine for PygmalionAI models
Project description
Breathing Life into Language
Aphrodite is the official backend engine for PygmalionAI. It is designed to serve as the inference endpoint for the PygmalionAI website, and to allow serving the Pygmalion models to a large number of users with blazing fast speeds (thanks to FasterTransformer and vLLM).
Aphrodite builds upon and integrates the exceptional work from various projects.
The compute necessary for Aphrodite's development is provided by Arc Compute.
Features
- Continuous Batching
- Efficient K/V management with PagedAttention
- Optimized CUDA kernels for improved inference
- Quantization support via GPTQ, GGUF, AWQ, QuIP#, and SqueezeLLM.
- Distributed inference
- Variety of sampling methods (Mirostat, Locally Typical Sampling, Tail-Free Sampling, etc)
- 8-bit KV Cache for higher context lengths and throughput.
Quickstart
pip install aphrodite-engine
python -m aphrodite.endpoints.openai.api_server --model PygmalionAI/pygmalion-2-7b
[!CAUTION] If the installation reports CUDA kernel errors, please run
pip install aphrodite-engine=0.4.5
instead.
This will create a OpenAI-compatible API server that can be accessed at port 2242 of the localhost. You can plug in the API into a UI that supports Kobold, such as SillyTavern.
Docker
sudo mkdir ~/workspace
sudo docker run --gpus '"all"' --shm-size 10g -p 2242:2242 -p 7650:7650 -p 8080:8080 --rm -it -v '/home/workspace' alignmentlabai/aphrodite-engine:latest
Performance
Speeds vary with different GPUs, model sizes, quantization schemes, batch sizes, etc. Here are some baseline benchmarks conducted by requesting as many completions as possible from the API server. Keep in mind that these are the theoritical peak throughput with parallel decoding, with as high a batch size as possible. Per-request generation speed is a fraction of this, at 30-40 t/s.
High Batch Size Performance
[!NOTE]
The numbers below are the theoritical peak achieved by only requesting output tokens at very high batch sizes. At lower batch sizes with much larger prompts, the results will be vastly different. Throughput refers to output tokens per second.
Model | Quantization | bits | GPU | Throughput (T/s) |
---|---|---|---|---|
Mistral 7B | None | 16 | RTX 4090 | 5489.3 |
AWQ | 4 | RTX 4090 | 4078.8 | |
GPTQ | 4 | RTX 4090 | 7850.4 | |
8 | RTX 4090 | 7658.0 | ||
GGUF | Q8 | RTX 4090 | 5141.2 | |
Q6KM | RTX 4090 | 5791.7 | ||
Q5KM | RTX 4090 | 5786.2 | ||
Q4KM | RTX 4090 | 5815.8 | ||
SqueezeLLM | 4 | RTX 4090 | 549.5 | |
Llama-2 7B | None | 16 | RTX 4090 | 2576.2 |
AWQ | 4 | RTX 4090 | 3551.3 | |
GPTQ | 4 | RTX 4090 | 2919.1 | |
GGUF | Q4KM | RTX 4090 | 2726.6 | |
Q5KM | RTX 4090 | 2763.4 | ||
Q6KM | RTX 4090 | 2694.7 | ||
Q8 | RTX 4090 | 2647.0 | ||
SqueezeLLM | 4 | RTX 4090 | 580.3 |
Batch Size 1
These are the speeds a user would normally get if they request a single output with a sizable prompt and output length. Essentially, normal chatting experience.
The following results were gathered by sending a request with 2000 prompt tokens and requesting 1024 tokens with ignore_eos=True
.
Model | Quantization | bits | GPU | Throughput (T/s) |
---|---|---|---|---|
Mistral 7B | None | 16 | RTX 4090 | 54.0 |
AWQ | 4 | RTX 4090 | 128.2 | |
GPTQ | 8 | RTX 4090 | 92.8 | |
4 | RTX 4090 | 146.8 | ||
GGUF | Q8 | RTX 4090 | 91.0 | |
Q6KM | RTX 4090 | 105.4 | ||
Q5KM | RTX 4090 | 117.8 | ||
Q4KM | RTX 4090 | 128.9 | ||
Llama-2 7B | None | 16 | RTX 4090 | 55.2 |
GPTQ | 8 | RTX 4090 | 90.2 | |
4 | RTX 4090 | 128.0 | ||
AWQ | 4 | RTX 4090 | 116.3 | |
GGUF | Q8 | RTX 4090 | 88.1 | |
Q6KM | RTX 4090 | 99.4 | ||
Q5KM | RTX 4090 | 109.9 | ||
Q4KM | RTX 4090 | 118.9 |
Requirements
- Operating System: Linux (or WSL for Windows)
- Python: at least 3.8
Build Requirements:
- CUDA >=12
For supported GPUs, see here.
Installation
Usage
For usage, please refer to the wiki page for detailed instructions. Aphrodite provides many different options for LLM inference, so please read through the list of options here.
Notes
-
By design, Aphrodite takes up 90% of your GPU's VRAM. If you're not serving an LLM at scale, you may want to limit the amount of memory it takes up. You can do this in the API example by launching the server with the
--gpu-memory-utilization 0.6
(0.6 means 60%). -
You can view the full list of commands by running
python -m aphrodite.endpoints.openai.api_server --help
. -
Context Length extension via the RoPE method is supported for most models. Use the command-line flag
--max-model-len
to specify a desired context length and the engine will adjust the RoPE scaling accordingly. -
Please refer to the FAQ & Issues if you run into problems. If you don't find an answer there, please make an issue.
Acknowledgements
Aphrodite Engine would have not been possible without the phenomenal work of other open-source projects. Credits go to:
- vLLM (CacheFlow)
- FasterTransformer
- xFormers
- AutoAWQ
- AutoGPTQ
- SqueezeLLM
- ExLlama
- Exllamav2
- KoboldAI
- Text Generation WebUI
- Megatron-LM
- FastChat
- Ray
- SkyPilot
- OpenAI Python Library
Contributing
Everyone is welcome to contribute. You can support the project by opening Pull Requests for new features, fixes, or general UX improvements.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for aphrodite_engine-0.4.9-cp311-cp311-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c006480dfd1ad368298c0bb511e22774fc203c80e96e875ffd1deb97a7afea0 |
|
MD5 | cba5643d8ebc2fdfa4e6b6ac081cc5a1 |
|
BLAKE2b-256 | f7d545326898a61153d13d0d82388967e9150655f2e29e8d0331e642ca1b8f2c |
Hashes for aphrodite_engine-0.4.9-cp310-cp310-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36bcb7dcbad2f3ce9a8a28842b66090702c6876ae27b88efa868cca8c4d5d516 |
|
MD5 | 7630a01724af778a0f622f8d0e69957a |
|
BLAKE2b-256 | c46df2f8463cc51007c605c052dea1842362a9b37cd78a3b8594b420a429456f |
Hashes for aphrodite_engine-0.4.9-cp39-cp39-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7837334d86fe874c432e0972e08e13da7194077d978338c20896cc5b42f8da35 |
|
MD5 | 3f12023292d281a29faadd0b9a3f11a6 |
|
BLAKE2b-256 | 75c8e45559d7ab6ef7f196d45a20b01011fb782055a0f17b7ce47e47a1d00fd2 |
Hashes for aphrodite_engine-0.4.9-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 026f880b6c85d46970e26803581e6a520c4152c5eea79008dc5c8495b980c7c5 |
|
MD5 | 8ad8cdc69b92f25001edb2aad2ef48c2 |
|
BLAKE2b-256 | fd8af2e46e9cefd765d9ae59abb7aaab5e66a4ed6f8df7baf63db6a06d5a9b17 |