Skip to main content

haru_vllm

Serve Haru with vLLM-Omni: many concurrent full-duplex conversations on one GPU, over the realtime WebSocket API.

This is a plugin. Installing it registers the Haru model, pipeline and duplex policy with vLLM-Omni; nothing in vLLM-Omni is patched.

Install and run

pip install vllm==0.32.0 vllm-omni==0.32.0
pip install -e ../haru -e .             # from this directory

vllm serve higurashi-ai/Haru-4B --omni --deploy-config "$(python -m haru_vllm.deploy)"

The server listens on ws://localhost:8000/v1/realtime?duplex=1. To use it from the web demo:

python -m haru.server --backend vllm --vllm-url http://localhost:8000

python -m haru_vllm.deploy prints the path of the bundled deploy configuration (deploy/haru.yaml). Copy and edit it to change the number of concurrent conversations (duplex_session.max_sessions and max_num_seqs) or the cache size.

Tested with vLLM 0.32.0 and vLLM-Omni 0.32.0, the releases on PyPI. It also runs with vLLM 0.33.0 and the vLLM-Omni main branch at commit 04e03a4. The plugin uses model-runner hooks of vLLM-Omni that may change between versions.

Protocol

The endpoint speaks vLLM-Omni's realtime duplex protocol (OpenAI Realtime events). One conversation:

  1. session.update opens the session. instructions is the character card, ref_audio a data:audio/...;base64, URI of the voice. Audio input is pcm_f32le at 24 kHz.
  2. Append one frame (3200 samples) with input_audio_buffer.append. This first append starts the conversation: the prompt runs and the model's frame 0 comes back. Its audio is not used; send silence.
  3. From then on, every appended frame k is answered by model frame k + 1: one response.output_audio.delta (PCM16, 24 kHz, 3200 samples) and, when the frame has text, a response.output_audio_transcript.delta.

Options in session.extra_body, fixed when the session is created:

Key Default Meaning
guidance_scale 1.5 Audio guidance, 1 to 2
top_k, top_p 25, 1.0 Text sampling (with the session's temperature, default 0.7)
eop_bias 0 Added to the logit of the "start speaking" token
speak_first false The model opens the conversation
dry true Repetition penalty on the text stream
seed random Seed of the audio sampling

How it is built

One autoregressive stage. Each conversation is a single resumable engine request whose prompt grows by one position per frame; the language model runs on vLLM's engine (paged attention cache, Gated DeltaNet state, CUDA graphs). The audio parts run in model hooks of the same step: the user frame is encoded before the forward pass, and the MeanFlow head and codec decoder run right after sampling, so a frame's audio leaves in the step that produced it. The codec runs as Triton kernels replayed from CUDA graphs when triton is installed (PyTorch's Linux CUDA wheels install it), and in plain PyTorch otherwise. Set HARU_CODEC_BACKEND=torch (or triton) in the environment of vllm serve to override that choice.

Metadata

Release files for haru-speech-vllm 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for haru-speech-vllm 0.1.1
File Size Uploaded Provenance Actions
haru_speech_vllm-0.1.1.tar.gz 19.5 kB With provenance Details for haru_speech_vllm-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for haru-speech-vllm 0.1.1
File Interpreter ABI Platform Actions
haru_speech_vllm-0.1.1-py3-none-any.whl Python 3 none any Details for haru_speech_vllm-0.1.1-py3-none-any.whl

Total release size: 42.0 kB

Release files / haru_speech_vllm-0.1.1.tar.gz

Download URL haru_speech_vllm-0.1.1.tar.gz
Size 19.5 kB
Tags Source
SHA-256 checksum
How to use checksums
9d0f196b272211f4e0b8b5bdd85fe11d72759516973fba2bfa12064403389bfa
BLAKE2b-256 checksum
How to use checksums
7a1345831bf6ad26e4e5ecdedca1974edca756692cfce23fc8b27c8b4eddbaee
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / haru_speech_vllm-0.1.1-py3-none-any.whl

Download URL haru_speech_vllm-0.1.1-py3-none-any.whl
Size 22.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1319c8a504d5e7596a5ca5fe6a2b14262f8d83d9209d601237e486e74aef4f79
BLAKE2b-256 checksum
How to use checksums
3eb7af8d7723be17dfa0b0c1458987c345dae01ff5e19676dc8da0cf2d92c31d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for haru-speech-vllm 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for haru_speech_vllm-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page