Skip to main content

haru_vllm

Serve Haru with vLLM-Omni: many concurrent full-duplex conversations on one GPU, over the realtime WebSocket API.

This is a plugin. Installing it registers the Haru model, pipeline and duplex policy with vLLM-Omni; nothing in vLLM-Omni is patched.

Install and run

pip install vllm==0.32.0 vllm-omni==0.32.0
pip install -e ../haru -e .             # from this directory

vllm serve higurashi-ai/Haru-4B --omni --deploy-config "$(python -m haru_vllm.deploy)"

The server listens on ws://localhost:8000/v1/realtime?duplex=1. To use it from the web demo:

python -m haru.server --backend vllm --vllm-url http://localhost:8000

python -m haru_vllm.deploy prints the path of the bundled deploy configuration (deploy/haru.yaml). Copy and edit it to change the number of concurrent conversations (duplex_session.max_sessions and max_num_seqs) or the cache size.

Tested with vLLM 0.32.0 and vLLM-Omni 0.32.0, the releases on PyPI. It also runs with vLLM 0.33.0 and the vLLM-Omni main branch at commit 04e03a4. The plugin uses model-runner hooks of vLLM-Omni that may change between versions.

Protocol

The endpoint speaks vLLM-Omni's realtime duplex protocol (OpenAI Realtime events). One conversation:

  1. session.update opens the session. instructions is the character card, ref_audio a data:audio/...;base64, URI of the voice. Audio input is pcm_f32le at 24 kHz.
  2. Append one frame (3200 samples) with input_audio_buffer.append. This first append starts the conversation: the prompt runs and the model's frame 0 comes back. Its audio is not used; send silence.
  3. From then on, every appended frame k is answered by model frame k + 1: one response.output_audio.delta (PCM16, 24 kHz, 3200 samples) and, when the frame has text, a response.output_audio_transcript.delta.

Options in session.extra_body, fixed when the session is created:

Key Default Meaning
guidance_scale 1.5 Audio guidance, 1 to 2
top_k, top_p 25, 1.0 Text sampling (with the session's temperature, default 0.7)
eop_bias 0 Added to the logit of the "start speaking" token
speak_first false The model opens the conversation
dry true Repetition penalty on the text stream
seed random Seed of the audio sampling

How it is built

One autoregressive stage. Each conversation is a single resumable engine request whose prompt grows by one position per frame; the language model runs on vLLM's engine (paged attention cache, Gated DeltaNet state, CUDA graphs). The audio parts run in model hooks of the same step: the user frame is encoded before the forward pass, and the MeanFlow head and codec decoder run right after sampling, so a frame's audio leaves in the step that produced it. The codec runs as Triton kernels replayed from CUDA graphs when triton is installed (PyTorch's Linux CUDA wheels install it), and in plain PyTorch otherwise.

Metadata

Release files for haru-speech-vllm 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for haru-speech-vllm 0.1.0
File Size Uploaded Provenance Actions
haru_speech_vllm-0.1.0.tar.gz 18.8 kB With provenance Details for haru_speech_vllm-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for haru-speech-vllm 0.1.0
File Interpreter ABI Platform Actions
haru_speech_vllm-0.1.0-py3-none-any.whl Python 3 none any Details for haru_speech_vllm-0.1.0-py3-none-any.whl

Total release size: 40.8 kB

Release files / haru_speech_vllm-0.1.0.tar.gz

Download URL haru_speech_vllm-0.1.0.tar.gz
Size 18.8 kB
Tags Source
SHA-256 checksum
How to use checksums
ee3fb64a45df56c13d47b129a7e756f6d1d55574fa4249291c9b0c3805e25b79
BLAKE2b-256 checksum
How to use checksums
06f69fb201451df5f1fa9ef1812db38e1c5848a7261cf9e5c8a879cd2826dbfa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / haru_speech_vllm-0.1.0-py3-none-any.whl

Download URL haru_speech_vllm-0.1.0-py3-none-any.whl
Size 22.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3803260876a34b2961219517619c317d3dbc514707dea8e2ffa0f423a92290a0
BLAKE2b-256 checksum
How to use checksums
2dc7211b5b7334ff7d8bebf35d7c5e17348f890890d10a1787d52e561063326c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for haru-speech-vllm 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for haru_speech_vllm-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page