Sonar
Sonar is an inference engine for Hugging Face-compatible language and multimodal models. It provides continuous batching, paged KV-cache management, optimized kernels, quantization, speculative decoding, and distributed serving.
Sonar is based on vLLM. It includes additional model and quantization formats, sampling methods, kernels, platforms, and deployment features. It serves production workloads for the Dolphin Inference Network and PygmalionAI.
Documentation
Read the documentation at sonar.dphn.ai.
- Installation
- Supported models
- Quantization support
- Server arguments
- Optimization
- Parallelism
- Production deployment
The model, quantization, and server-argument references are generated from the current source tree.
Install
Run the automatic installer:
curl -fsSL https://sonar.dphn.ai/install.sh | bash
The installer supports Linux x86-64 (NVIDIA CUDA, AMD ROCm, or CPU), Linux Arm64 (CPU), and Apple silicon macOS (Metal).
Use the complete installation guide for AMD ROCm, Intel XPU, CPU, Apple silicon, Google TPU, Docker, WSL 2, source builds, and nightly wheels.
Serve a model
aphrodite serve Qwen/Qwen3-0.6B \
--served-model-name qwen3
The server listens on http://127.0.0.1:2242 by default. It provides
OpenAI-compatible APIs, health checks, metrics, and an OpenAPI schema.
See the OpenAI-compatible API guide for streaming, embeddings, tool calls, reasoning output, and Sonar request parameters.
Key features
- Continuous batching and paged KV-cache management
- Prefix caching enabled by default
- Tensor, pipeline, data, and expert parallelism
- Multi-node multiprocessing without a Ray cluster
- Prefill/decode disaggregation through NIXL and other KV connectors
- Quantized weights and FP8 KV cache
- Speculative decoding with MTP, EAGLE, DSpark, DFlash, n-gram, and other methods
- Structured output, reasoning parsers, and automatic tool calling
- Image, audio, and video model support
- LoRA adapter serving
- Prometheus metrics and health endpoints
- OpenAI, Anthropic, pooling, scoring, reranking, transcription, and Kobold APIs
Support depends on the model, device, data type, and quantization method. Check the generated model matrix and quantization matrix before deployment.
Read the optimization guide before you tune scheduler, cache, compilation, quantization, or speculative decoding settings.
Development
git clone https://github.com/dphnAI/sonar.git
cd sonar
uv venv --python 3.13 --seed --prompt sonar
source .venv/bin/activate
APHRODITE_USE_PRECOMPILED=1 \
uv pip install --editable . --torch-backend=cu130
1 maintainer
Metadata
1 maintainer
Release files for aphrodite-engine 0.24.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| aphrodite_engine-0.24.1-cp38-abi3-manylinux_2_34_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.34+ x86-64 | Details |
Release files / aphrodite_engine-0.24.1-cp38-abi3-manylinux_2_34_x86_64.whl
| Download URL | aphrodite_engine-0.24.1-cp38-abi3-manylinux_2_34_x86_64.whl |
|---|---|
| Size | 336.0 MB |
| Tags | CPython 3.8 Linux glibc 2.34+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
66bb528c5daa96ccb5c0bbf5c942a02c5d1b67056bcf819d934ccd8730b40269
|
|
BLAKE2b-256 checksum How to use checksums |
c9c2656bd711187bdf2b9e652e60e5afaa1ab5f8d06d9324028ada683b86335e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
