Sonar
Sonar is an inference engine for Hugging Face-compatible language and multimodal models. It provides continuous batching, paged KV-cache management, optimized kernels, quantization, speculative decoding, and distributed serving.
Sonar is based on vLLM. It includes additional model and quantization formats, sampling methods, kernels, platforms, and deployment features. It serves production workloads for the Dolphin Inference Network and PygmalionAI.
Documentation
Read the documentation at sonar.dphn.ai.
- Installation
- Supported models
- Quantization support
- Server arguments
- Optimization
- Parallelism
- Production deployment
The model, quantization, and server-argument references are generated from the current source tree.
Install
Run the automatic installer:
curl -fsSL https://sonar.dphn.ai/install.sh | bash
The installer supports Linux x86-64 (NVIDIA CUDA, AMD ROCm, or CPU), Linux Arm64 (CPU), and Apple silicon macOS (Metal).
Use the complete installation guide for AMD ROCm, Intel XPU, CPU, Apple silicon, Google TPU, Docker, WSL 2, source builds, and nightly wheels.
Serve a model
aphrodite serve Qwen/Qwen3-0.6B \
--served-model-name qwen3
The server listens on http://127.0.0.1:2242 by default. It provides
OpenAI-compatible APIs, health checks, metrics, and an OpenAPI schema.
See the OpenAI-compatible API guide for streaming, embeddings, tool calls, reasoning output, and Sonar request parameters.
Key features
- Continuous batching and paged KV-cache management
- Prefix caching enabled by default
- Tensor, pipeline, data, and expert parallelism
- Multi-node multiprocessing without a Ray cluster
- Prefill/decode disaggregation through NIXL and other KV connectors
- Quantized weights and FP8 KV cache
- Speculative decoding with MTP, EAGLE, DSpark, DFlash, n-gram, and other methods
- Structured output, reasoning parsers, and automatic tool calling
- Image, audio, and video model support
- LoRA adapter serving
- Prometheus metrics and health endpoints
- OpenAI, Anthropic, pooling, scoring, reranking, transcription, and Kobold APIs
Support depends on the model, device, data type, and quantization method. Check the generated model matrix and quantization matrix before deployment.
Read the optimization guide before you tune scheduler, cache, compilation, quantization, or speculative decoding settings.
Development
git clone https://github.com/dphnAI/sonar.git
cd sonar
uv venv --python 3.13 --seed --prompt sonar
source .venv/bin/activate
APHRODITE_USE_PRECOMPILED=1 \
uv pip install --editable . --torch-backend=cu130
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file aphrodite_engine-0.24.1-cp38-abi3-manylinux_2_34_x86_64.whl.
File metadata
- Download URL: aphrodite_engine-0.24.1-cp38-abi3-manylinux_2_34_x86_64.whl
- Upload date:
- Size: 336.0 MB
- Tags: CPython 3.8+, manylinux: glibc 2.34+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
66bb528c5daa96ccb5c0bbf5c942a02c5d1b67056bcf819d934ccd8730b40269
|
|
| MD5 |
40ed89816ce70490d26fdf36a1b5e798
|
|
| BLAKE2b-256 |
c9c2656bd711187bdf2b9e652e60e5afaa1ab5f8d06d9324028ada683b86335e
|
