omnivoice-triton-server 0.1.0

OpenAI-compatible OmniVoice server with batched multi-GPU Triton inferers

OmniVoice Triton Server

OpenAI-compatible OmniVoice TTS server built for batched GPU inference. The server runs FastAPI workers for request handling and one or more independent GPU inferer processes for model execution.

中文说明: docs/README.zh-CN.md

Origin

This repository combines three code lines into one deployable service:

• omnivoice-server: API server, request routing, socket IPC, batching, metrics, deployment scripts, and tests.
• omnivoice-triton: Triton/hybrid inference backend pieces and CUDA/Triton acceleration code.
• k2-fsa/OmniVoice: selected OmniVoice model/runtime code under src/modeling.

The code is kept in a single tree because scheduling, chunking, graph capture, and model invocation need to be tuned together. This is not a clean upstream mirror; it is a server-oriented integration.

Main Optimizations

• Multi-GPU serving with --gpu-inferer N. Each GPU inferer is a separate process with its own model weights on one CUDA device.
• FastAPI worker count defaults to the effective GPU inferer count when --fastapi-workers is not specified.
• Worker-side preprocessing: validation, semantic text chunking, duration splitting, clone reference audio ingestion, response formatting, and SSE framing run outside the inferer.
• Async local TCP socket IPC between workers and inferers. Clone reference audio is sent through the socket; the server does not create temporary prompt audio files for normal requests.
• Shared-memory metrics snapshots, so /metrics does not block generation.
• Clone audio prompt LRU cache controlled by --max-clone-audio-prompt-cache.
• chunk_mode request control:
  • concurrent: default. Clone chunks share the same clone prompt. Auto/design generate chunk 0 first, then use that result as continuity prompt for the remaining chunks, which can run concurrently.
  • sequential: each chunk uses the previous generated chunk as continuity prompt.
  • none: still chunks text, but estimates a larger chunk size from the model context limit to avoid unnecessary splitting.
• Mixed-language semantic chunking with CJK/Thai/Hangul character counting, non-CJK token counting, punctuation-aware recursive splitting, and balanced packing near the configured word limit.
• Fixed-shape CUDA Graph prewarming with compact batch/width buckets, memory headroom checks, graph hit/miss metrics, and graph-aware microbatch splitting for oversized batches.

CPU inferer code was removed. Scale this server with GPU inferer processes.

Requirements

• Python 3.12 or newer.
• PyTorch, Triton, Transformers, FastAPI, and the packages in requirements.txt.
• NVIDIA GPU for inference. The default runner mode is hybrid.
• OmniVoice model files available either from a Hugging Face model id or a local path passed with --model-id / OMNIVOICE_MODEL_ID.

Quick Start

python -m venv .venv
. .venv/bin/activate
pip install omnivoice-triton-server

export CUDA_VISIBLE_DEVICES=0
omnivoice-triton-server start

Two-GPU example:

CUDA_VISIBLE_DEVICES=0,1 \
omnivoice-triton-server start \
  --port 9194 \
  --model-id /path/to/OmniVoice \
  --gpu-inferer 2 \
  --max-batch-size 16 \
  --max-batch-latency 250 \
  --cuda-stream-count 2 \
  --runner-mode hybrid \
  --num-step 32

Installing the package adds an omnivoice-triton-server Python console command. The module entrypoint is also available:

python -m omnivoice-triton-server start --port 9194

Stop a foreground/background process by port or pid file:

omnivoice-triton-server stop --port 9194
omnivoice-triton-server stop --pid-file logs/20260520-212301/server.pid --no-port

Stop a systemd deployment:

omnivoice-triton-server stop --systemd --service-name omnivoice-server

CUDA_VISIBLE_DEVICES is a deployment choice. The benchmark below used CUDA_VISIBLE_DEVICES=6,7 on one 8-GPU test server because those two devices were selected for that run; use the device ids that are correct on your machine. scripts/start_server.sh is only a POSIX shell convenience wrapper around the same module entrypoint.

Systemd install example:

scripts/install_systemd_service.sh \
  --cuda-visible-devices 0,1 \
  --python /path/to/python \
  --service-name omnivoice-server \
  -- \
  --port 9194 \
  --model-id /path/to/OmniVoice \
  --gpu-inferer 2 \
  --max-batch-size 16

Arguments after -- are passed directly to omnivoice-triton-server start. The script writes /etc/omnivoice/<service>.sh and /etc/systemd/system/<service>.service, then reloads, enables, and restarts the unit unless --no-enable or --no-start is used.

Important Arguments

• --model-id: local model path or Hugging Face model id.
• --gpu-inferer: number of GPU inferer processes to launch. The launcher clamps this to the number of visible CUDA devices.
• --fastapi-workers: API worker count. Defaults to effective GPU inferer count when omitted.
• --max-batch-size, --max-batch-latency: scheduler batching controls.
• --cuda-stream-count: backend worker streams per inferer.
• --cuda-graph-min-width, --cuda-graph-max-width: graph width controls.
• --cuda-graph-auto-width-tokens-per-word, --cuda-graph-auto-max-width: context-limit estimation for chunk_mode=none.
• --num-step: global generation step count. Default: 32.
• --max-clone-audio-prompt-cache: clone prompt LRU size. Default: 32.
• --max-continuity-audio-tokens, --max-continuity-text-words: chunk continuity prompt limits.
• --text-chunk-words and --text-chunk-*: chunking and packing controls.
• --log-dir, --log-run-id, --log-file, --pid-file: runtime log layout.

All settings can also be set with OMNIVOICE_* environment variables. Python defaults live in src/config.py; shell scripts do not define service defaults.

API

Speech endpoint:

curl -X POST http://127.0.0.1:9194/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "tts-1",
    "input": "Hello from OmniVoice.",
    "voice": "auto",
    "response_format": "wav",
    "speed": 1.0,
    "chunk_mode": "concurrent"
  }' \
  --output speech.wav

Voice design endpoint:

curl -X POST http://127.0.0.1:9194/v1/audio/design \
  -F 'text=Hello from a designed voice.' \
  -F 'instruct=female, young adult, moderate pitch' \
  -F 'chunk_mode=concurrent' \
  -F 'response_format=wav' \
  --output design.wav

Voice clone endpoint:

curl -X POST http://127.0.0.1:9194/v1/audio/clone \
  -F 'text=Hello from a cloned voice.' \
  -F 'ref_audio=@ref.wav;type=audio/wav' \
  -F 'ref_text=Text spoken in the reference audio.' \
  -F 'chunk_mode=concurrent' \
  -F 'response_format=wav' \
  --output clone.wav

Supported response formats are wav and raw pcm.

Benchmark

These numbers are for capacity planning on one local test host. They are not a hardware-independent promise.

Test configuration:

• Hardware used by this service: 2 x NVIDIA GeForce RTX 3080, 20 GiB each.
• Host inventory: 8 visible RTX 3080 GPUs; this run used CUDA_VISIBLE_DEVICES=6,7.
• Launch: --gpu-inferer 2 --fastapi-workers 2 --runner-mode hybrid --dtype fp16 --max-batch-size 16 --max-batch-latency 250 --cuda-stream-count 2 --num-step 32.
• Load generator: 1000 requests scheduled at 100 req/s. Latency therefore includes queueing after the offered load exceeds service capacity.
• Audio quality smoke: auto, design, and clone outputs were checked by ASR on both short and long texts.

Throughput And Latency

Workload	Wall time	Completed req/s	Generated audio	Audio realtime	RTF	Mean latency	p50	p95	p99
Short speech/design	61.553 s	16.246	786.100 s	12.771x	0.0783	27.2485 s	26.6065 s	50.8381 s	54.6932 s
Mixed short/medium/long speech/design/clone	247.159 s	4.046	2,648.752 s	10.717x	0.0933	120.7916 s	119.3613 s	228.5990 s	236.4849 s

Audio realtime is generated audio duration divided by wall time. RTF is wall time divided by generated audio duration.

Scheduler Efficiency

Workload	Client requests	Backend tasks	Tasks/request	Backend batches	Tasks/backend batch	Backend task/s
Short speech/design	1,000	1,000	1.000	68	14.706	16.246
Mixed short/medium/long speech/design/clone	1,000	1,733	1.733	67	25.866	7.011

The mixed run has more backend tasks than client requests because long inputs are split into chunks. The useful batching signal is Tasks/backend batch: higher means the scheduler kept the GPU inferers fed with larger model batches.

Mixed Workload Breakdown

Kind	Requests	Mean latency	p95	Max
speech	900	122.8774 s	228.6261 s	236.6119 s
design	50	129.3640 s	229.6973 s	230.9176 s
clone	50	74.6742 s	143.6043 s	145.6617 s

CUDA Graph Behavior

Final mixed run graph state, aggregated across two GPU inferers:

• Graph entries per inferer: 14.
• Captured shapes per inferer: (2,8,128), (2,8,160), (2,8,256), (2,8,512), (8,8,64), (8,8,128), (8,8,160), (8,8,256), (16,8,128), (16,8,160), (16,8,256), (32,8,64), (32,8,128), (32,8,160).
• During the mixed 1000-request run: 11,520 graph hits and 32 graph misses after subtracting the pre-run counters.
• Max backend batch seen by each inferer: 49 and 46 tasks.

The graph miss count is the number to watch when changing chunking, graph width buckets, or --max-batch-size; sustained misses usually mean requests are falling outside the prewarmed shape plan.

Development Checks

python -m py_compile \
  src/app.py src/audio.py src/chunking.py src/config.py src/infer_client.py \
  src/inferer.py src/launcher.py src/protocol.py

PYTHONPATH=src python tests/test_chunking.py
python tests/test_api.py
python tests/load_1000_rps100.py --total 1000 --rate 100 --concurrency-limit 512
python tests/load_mixed_1000.py --total 1000 --rate 100 --ref-audio /path/to/ref.wav

Runtime artifacts, logs, generated audio, model weights, and local environment files are ignored by .gitignore.

See docs/DEPLOYMENT.md for operational details.