voxedge

English | 中文

Native TensorRT · RKNN · sherpa-onnx voice pipelines for Jetson, Rockchip, and Raspberry Pi — fully on-device, verified on real hardware, zero cloud.

What is voxedge?

voxedge is an embeddable Python library that drives real-time, on-device voice conversations by calling directly into each platform's native inference runtime — TensorRT on Jetson Orin, RKNN on RK3576/RK3588, sherpa-onnx on CPU. No cloud STT/TTS APIs, no internet at runtime, no intermediate abstraction overhead. The same ConversationEngine contract spans the three backend families; the exact model/runtime configuration still determines which features and concurrency level are available.

voxedge is the open-core engine behind OpenVoiceStream — the deployable FastAPI/WebSocket server, device profiles, and agent gallery. Want a container? Start there. Want to embed real-time edge voice in your own app? You're in the right place.

Key Features

• Native runtimes, full performance — calls directly into TensorRT (Jetson), RKNN (Rockchip), and sherpa-onnx (CPU); no wrapper overhead, no cross-platform abstraction tax
• Fully on-device — no speech API key, no per-call bill, no internet dependency at runtime
• Hardware evidence, narrowly scoped — selected Jetson slot-pool configurations have passed N=2 smoke tests on Orin Nano 8 GB; this is not a blanket N=2 guarantee for every backend, model, device, or artifact build
• Streaming + barge-in — partial + final ASR while the user speaks; sentence-level TTS streaming with first-audio latency low enough for live dialogue and cooperative barge-in
• Swap hardware, not code — same ConversationEngine API across Jetson, Rockchip, and sherpa-onnx CPU; only the backend constructor changes
• Test on any machine — mock backends require only numpy; the whole engine runs end-to-end on a Mac with no CUDA or GPU

Quickstart

Runs on any machine — no GPU needed. Swap the backend constructors for a real device; the engine, transport, and event contract never change.

pip install voxedge

import asyncio
from voxedge.engine import ConversationEngine
from voxedge.transport import InProcessTransport
from voxedge.backends.mock import MockASR, MockTTS, MockVAD

engine = ConversationEngine(
    backends={"asr": MockASR(transcript="hello world"), "tts": MockTTS(), "vad": MockVAD()},
    multi_utterance=True,
)

async def main():
    t = InProcessTransport()
    await t.feed_audio(b"\x01\x02" * 8000)   # speech frames (int16 PCM)
    await t.feed_audio(b"\x00\x00" * 8000)   # silence → VAD endpoints the utterance
    t.end_input()
    await engine.run(t)                       # drives ASR → (LLM) → TTS
    for ev in t.drain_events_nowait():        # asr_final / tts_* / ...
        print(ev["type"], ev.get("text", ""))

asyncio.run(main())

On a real device, swap only the backend constructors — everything else is identical:

# Jetson Orin — pip install voxedge[jetson]
from voxedge.backends.jetson import (
    TRTEdgeLLMASRBackend, TRTEdgeLLMASRConfig,
    TRTEdgeLLMTTSBackend, TRTEdgeLLMTTSConfig,
)

engine = ConversationEngine(backends={
    "asr": TRTEdgeLLMASRBackend(TRTEdgeLLMASRConfig(...)),   # Qwen3-ASR, native TRT
    "tts": TRTEdgeLLMTTSBackend(TRTEdgeLLMTTSConfig(...)),   # Qwen3-TTS, streaming
}, multi_utterance=True)

import voxedge is numpy-only — TensorRT, RKNN, and sherpa-onnx are lazy-imported by their backend adapters and pulled in via extras. The example above imports cleanly on a Mac even though the TRT engine only runs on a Jetson.

Install

pip install voxedge            # pure-Python core (numpy only)
pip install voxedge[sherpa]    # sherpa-onnx CPU ASR/TTS
pip install voxedge[jetson]    # Jetson TensorRT backends (aarch64)
pip install "voxedge[rk]==0.0.13a0" # Rockchip RK3576/RK3588 NPU (aarch64)
pip install voxedge[llm]       # OpenAI-compatible LLM backend (httpx)

The jetson extra installs only Python-side dependencies available from the package index. It does not install TensorRT/CUDA, the C++/pybind workers, plugins, TensorRT engines, or model artifacts; those come from JetPack and the engine/artifact release. Therefore pip install voxedge[jetson] alone does not produce a runnable Jetson deployment. The same platform-runtime caveat applies to RKNN and voxedge[rk].

Kokoro ConvOnly on Rockchip

Kokoro ConvOnly is a first-class RKVoice Stream TTS backend exposed through VoxEdge and used directly by OpenVoiceStream (OVS). Install the released adapter with pip install "voxedge[rk]==0.0.13a0"; the RK extra requires rkvoice-stream>=0.2.0 on aarch64.

The image is unified across RK3576 and RK3588. The platform bundle containing RKNN models and the optional Japanese dictionary is mounted externally as a read-only resource, so changing a model or region does not create a new application image. Configure the bundle path and backend profile in OVS. With language=auto, Latin/CJK text is routed by the backend detector and hiragana, katakana, halfwidth katakana, and related kana text route to ja.

Kokoro ConvOnly supports sentence-level streaming through RKVoice Stream and HTTP finite-request cancellation. Closing a consumer stream propagates to the native iterator and releases the backend lifecycle owner. Cancellation of a single in-flight RKNN operation is intentionally not forced; backends whose native call is monolithic finish the current call before teardown.

Library versus server

voxedge is a Python library: backend interfaces, the conversation engine, and in-process/websocket transport adapters. It does not start FastAPI or expose HTTP routes. OpenVoiceStream (OVS) is the deployable server/product layer that selects profiles, owns artifacts and containers, and exposes network APIs.

OVS provides an OpenAI-compatible speech surface including:

• POST /v1/audio/speech — audio is streamed with HTTP chunked transfer rather than buffered into one response;
• GET /v1/models — models available from that OVS deployment;
• GET /v1/capabilities — runtime/backend capabilities detected by that deployment.

These routes describe OVS, not endpoints implemented by this repository.

Architecture

Four layers, all importable without CUDA.

Backends (voxedge/backends/)

Clean ABCs in backends/base.py — every constructor takes explicit params only, no env coupling:

• ASRBackend / ASRStream — streaming recognition
• TTSBackend — synthesize() (batch) + generate_streaming() (sentence-level chunks, cooperative cancel via cancel_token for barge-in)
• VADBackend / VADSession — voice-activity detection for speech / barge-in segmentation
• LLMBackend / LLMEvent — token-streaming LLM for the conversation loop

Concrete adapters live under backends/{jetson,rk,sherpa}/ and import their heavy runtimes lazily (inside methods), so all modules import on any machine:

Jetson TTS voice, clone, and speed matrix

"DSP fallback" means voxedge post-processes PCM; it is not a native model control and may have different quality/latency characteristics.

Transport (voxedge/transport/)

Transport ABC + two implementations:

• InProcessTransport — zero-IPC asyncio queues; default, used everywhere in tests
• WebSocketTransport — duck-typed ws adapter with no FastAPI dependency; idle-watchdog timeout injected by caller, reads no env

Conversation Engine (voxedge/engine/)

ConversationEngine + per-connection Session coordinator, split into focused collaborators: audio_dispatcher (VAD → speech / barge-in), asr_loop, client_events, tts_sequencer / tts_buffer, session_state, and the LLM↔tool loop — llm_turn over the provider-agnostic turn_driver.run_turn pump, with tool_registry (@tool → JSON schema) and coordinator / concurrency_capability for multi-stream concurrency.

Capabilities (voxedge/capabilities/)

Optional, default-off, stateless add-ons (punctuation, speaker embedding) via sherpa-onnx. Opt in explicitly; byte-level no-op when off.

Design Constraints

• Pure Python core — import voxedge is numpy-only. Heavy adapters live under backends/{jetson,rk,sherpa}/ with deferred runtime imports.
• Explicit runtime config — backend configuration is injected as params; profiles and deployment knobs belong to OpenVoiceStream. The optional artifact downloader deliberately honors HF_ENDPOINT after an explicit endpoint and before the manifest endpoint.

Status

voxedge 0.0.9a0 is the current release; the qualified TensorRT Edge-LLM v0.9.1 OVS runtime pins it as of 2026-08-09.

0.0.9a0 stops the sherpa SenseVoice backend from hardcoding two decisions that belong to the deployment. use_itn was pinned to True; ITN is not merely punctuation — it also decides digit form (9点 vs 九点), English casing, and on the 2024-07-17 export even word choice (开放时间 vs 开饭时间). On the 2025-09-09 export, use_itn=True drops the leading character outright, so the hardcoded value is actively harmful there with no way out. It is now SherpaASRConfig.offline_use_itn, still defaulting to True so existing deployments are byte-identical.

transcribe(audio, language=...) also accepted a per-request language, discarded it, and echoed it back in TranscriptionResult.language — a caller asking for zh got a reply claiming zh from a recognizer running in auto. SenseVoice binds language at recognizer construction, so honoring it per request would mean one resident model copy per language (~237 MB each). Instead the pin is deployment-level (offline_language) and the result now reports the language the recognizer was actually built with, warning once per distinct conflicting request. Note the pin is a weak hint: with it set, audio in another supported language is still transcribed in its own language; mainly punctuation shifts.

0.0.9a0 also unifies what TranscriptionResult.language means. Backends were each wrong in a different direction: sherpa and Paraformer-TRT echoed the caller's request back untouched — and Paraformer has no language switch at all, so the parameter never reached its decoder — while SenseVoice-TRT applied the language and then reported None. The field now always describes what the backend actually decoded with, and is None when a backend selects no language. backends.base.resolve_reported_language() carries the three shapes (config pin / honoured per call / no selection) so the judgement is not re-derived, and re-derived wrongly, in every backend.

0.0.8a0 adds a guard against decoder degeneration on short audio. Qwen3-ASR under greedy decoding (top_k=1) collapses into looping repeats: a 300ms fragment transcribed as "帮我，" repeated 128 times. This is not a streaming bug — the offline /asr path produces byte-identical output. The worker accepts only temperature/top_k/top_p/max_generate_length, with no repetition penalty, so the guard lives in the text layer, at the shared exit both paths go through. It is deliberately conservative: word-based with a 6-repeat threshold for space-separated languages (so the the the survives), character-based for CJK (8 repeats for single characters, 3 for longer units), and in both cases the repeat must cover 80% of the text. Two-fold repeats are left alone.

0.0.7a0 carried two field fixes and one breaking change:

• Arabic digits are normalized to spoken Chinese before lexicon lookup. The matcha-icefall-zh-en token table contains no Arabic digit, and a lookup miss is silently skipped — so digits produced no audio at all rather than being mispronounced.
• The streaming ASR worker slot is now returned on every finalize path. It was only returned on the zero-audio branch, so with max_slots=1 every successful recognition leaked a slot and the second utterance onward failed with PoolSaturatedError.
• Breaking: every ASRStream subclass must now declare OWNS_RESOURCES. Omitting it — or declaring True without implementing close() — raises TypeError at class definition time. The previous no-op close() made "forgot to implement" indistinguishable from "nothing to release", which is how the slot leak above survived unnoticed. The release passed the mock suite and Orin NX target-device qualification; platform runtimes, workers, and model engines remain separate deployment artifacts. In particular, the OpenAI HTTP routes above are OVS behavior, not a voxedge compatibility promise.

Contributing

Issues and PRs welcome. The mock backend suite runs on any machine with no hardware:

pip install voxedge
uv run pytest

Ecosystem

voxedge is one layer in a family of repos:

Acknowledgements

• sherpa-onnx — CPU ASR/TTS runtime
• OpenVoiceStream — the deployable server product built on this engine

License

Apache-2.0. See LICENSE.