syne-voice 0.1.0

Voice and avatar conscious companion runtime on top of the Mnemosyne memory substrate. 4-process tree (supervisor, cortex, voice, swarm), FLAME point-cloud avatar, LiveKit voice loop, Darwin-Godel self-editing with Tier-7 filesystem lock. CLI and Python API.

Syne

Voice + avatar conscious companion. The other half of the real-time local voice agent experiment.

Built on top of Mnemosyne. This is an experiment in pushing the latency floor of fully-local, real-time voice agents on a single consumer GPU.

---

What this is

Syne is the mouth, ears, and face of a voice agent. It runs as a 5-process tree (supervisor → cortex / voice / swarms / avatar / emotion) on top of a single Mnemosyne engine, with one LiveKit WebRTC session at a time.

The goal of the experiment:

Can a fully-local voice agent on a single RTX 3060 beat a hosted OpenAI Realtime API on perceived latency, while running 100% offline?

We don't have a final answer. The repo shows where we are.

---

What works today

• LiveKit voice loop with VAD + dual STT (Moonshine partials, Parakeet final)
• LFM 1.2B Tier-1 responder (60+ tok/s on Pascal, fits in 1 GB VRAM)
• Qwen 2B Tier-2 planner with JSON-schema-constrained output
• Tiered degradation:
  • Tier 1: pod-routed (MI300X, ~270ms network)
  • Tier 2: local Ollama when pod unreachable
  • Tier 3: WAL queue when both local + pod fail
• Kokoro + CosyVoice2 + Orpheus TTS pools with barge-in cancellation
• FLAME point-cloud avatar in the terminal (or rendered to a frame file)
• Adaptive backchannel pool that learns from your engagement (mhm-hmm frequency, not random)
• Hedge injection at low classifier confidence
• Tool dispatch with consent: set_timer always prompts, web.fetch prompts once per session, memory.search is safe
• LiveKit QR pairing for phone-as-microphone
• Persistent supervisor with 3-crash-in-30s halt, heartbeat watcher, WAL replay on recover

Perceived latency (local, warm, voice loop): ~500-700 ms. Hard ceiling 1.2s.

---

What we're still figuring out

• Whether specular backchannel (predicting the user's "uh huh" before they say it) actually improves the UX or just feels weird
• Whether memory residues (failed tool calls, abandoned plans) should become first-class memories or stay ephemeral
• Whether the persona drift timer (daily background task that nudges the agent's voice/personality based on long-term memory) is net-positive or just weird at 3am
• How to scale to 4 concurrent voice sessions on one GPU without losing latency (the 1-session benchmark is at the limit of Pascal)
• Whether Qwen flash-attention can be safely enabled on sm_70+ (it segfaults on Pascal sm_61 — see llm/pool.py:qwen_flash_attn)

---

TL;DR

# Build the system
cd syne_rs && cargo build --release
cd ../syne
./bootstrap.sh                         # creates .venv, installs deps
source .venv/bin/activate

# Run the supervisor (Python fallback mode for now)
SYNE_SUPERVISOR_PYTHON=1 python -m syne.supervisor
# → starts cortex, voice, swarms, avatar, emotion as children

# Drive it via the LiveKit dev server (or your own)
# Open the WebRTC client at http://localhost:7880
# Talk. The agent listens, thinks, talks.

For the full setup, see the in-repo .kiro/specs/syne/ (the captured plan log).

---

Architecture (60-second tour)

┌───────────── LiveKit WebRTC session ─────────────┐
│                                                  │
│   You speak ──► 20ms int16 frames @ 16kHz       │
│                                                  │
│   Mic ──► Silero VAD ──► TurnDetector ──► Dual-STT
│                                       (Moonshine + Parakeet)
│                                                  │
│   ──► Prosody extractor (YIN pitch, RMS)       │
│   ──► Ambient monitor (YAMNet)                  │
│   ──► AudioCapture (sha, mnemosyne ref)         │
│                                                  │
│   On turn end:                                   │
│   Tier-0 classifier (~50ms)                     │
│     ├ Tier 1: LFM 1.2B stream (~200ms)         │
│     └ Tier 2: Qwen 2B planner (when tools)     │
│   ──► OutputVerifier                             │
│   ──► TTSRouter (Kokoro / CosyVoice2 / Orpheus) │
│   ──► Speaker                                    │
│                                                  │
└──────────────────────────────────────────────────┘
                ▲                       ▲
                │ HTTP/MCP              │ in-process
                │                       │
   ┌────────────┴───────────┐  ┌────────┴────────┐
   │   Mnemosyne (sister)   │  │  Rust supervisor │
   │   memory + retrieval   │  │  tree (5 procs)  │
   └────────────────────────┘  └──────────────────┘

---

Repository layout

syne/
├── syne/                      # The companion package (Python)
│   ├── voice/                 # LiveKit loop, STT, TTS, prosody, sentence buffer
│   ├── cortex/                # BrainServer, MemoryReflex, opt-out, retention
│   ├── avatar/                # FLAME point-cloud renderer
│   ├── swarms/                # Multi-agent (Claude Code, Kiro, OpenClaw, Hermes)
│   ├── safety/                # BudgetLedger, WAL, rollback, tripwires
│   ├── self_edits/            # Tier-7 self-edit gates (Darwin Gödel Machine)
│   ├── scripts/               # Operational scripts + systemd units
│   ├── cli.py                 # `syne` CLI
│   ├── supervisor.py          # Python fallback for the Rust supervisor
│   ├── bootstrap.sh           # One-shot setup
│   ├── pyproject.toml
│   └── ...
├── syne_rs/                   # Rust workspace (6 crates)
│   ├── crates/
│   │   ├── supervisor/        # PID spawn + heartbeat watcher
│   │   ├── router/            # Sub-millisecond LLM-call router
│   │   ├── swarm_core/        # Per-swarm state machine
│   │   ├── audio_dsp/         # AEC, noise suppression, AGC
│   │   ├── wal/               # Write-ahead log (SQLite + Rust)
│   │   └── pyo3_bindings/     # PyO3 for Python interop
│   ├── Cargo.toml
│   └── ...
├── .kiro/specs/syne/          # The phase log (P0..P9.4)
├── tests/                     # 1736 tests, 99.3% pass
├── packages/syne-py/          # Thin PyPI wrapper around ../syne/
├── .github/workflows/         # release.yml (binary build)
├── LICENSE                    # MSAL v1.0
├── CHANGELOG.md
└── README.md                  # ← you are here

---

The "real-time voice agents locally" experiment

Stack we picked and why:

Choice	Why
LiveKit over raw WebRTC	Mature Python client, server in Go/Rust, free for self-host, scales to mobile later
Parakeet TDT 0.6B as primary STT	Sub-100ms partials, onnxruntime, runs on Pascal sm_61 (GTX 1050 Ti)
Moonshine as fallback STT	4x faster than Whisper, similar quality, smaller
LFM 2.5 1.2B Q4_K_M as Tier-1	65 tok/s on Pascal, fits 1 GB VRAM, good enough for short replies
Qwen 2B Q4_K_M as Tier-2 planner	Structured JSON output via GBNF grammar
Kokoro 82M ONNX as TTS-1	0.2s TTFB, 24 voices, runs on CPU
CosyVoice2 0.5B Q8 as TTS-2	Higher quality, slower, 600 MB VRAM
Ollama as fallback LLM server	Bundles CUDA 12 libs (Arch ships CUDA 13.2 which is Pascal-broken)
Postgres + Qdrant as the substrate	Battle-tested, can be self-hosted
SQLite WAL as Tier-3 queue	No deps, durable, replayable

What we tried that didn't work:

• Whisper large-v3 — too slow (1.2s partials)
• Bark TTS — robotic, slow
• LFM 7B — too big for Pascal, OOMs
• Qwen 7B planner — same
• PyTorch on Pascal sm_61 — segfaults under flash-attention
• Building llama-cpp-python from PyPI — links against CUDA 13 (incompatible)
• Self-hosting Whisper.cpp — too slow

The full Q&A log is in syne/.kiro/specs/syne/.

---

For the impatient

git clone https://github.com/synet-systems/syne.git
cd syne
./bootstrap.sh
source .venv/bin/activate
SYNE_SUPERVISOR_PYTHON=1 python -m syne.supervisor

You should see:

2026-06-14 12:40:36  syne.supervisor spawning syne-cortex
2026-06-14 12:40:36  syne.supervisor spawning syne-voice
2026-06-14 12:40:36  syne.supervisor spawning syne-swarms
2026-06-14 12:40:36  syne.supervisor spawning syne-avatar
2026-06-14 12:40:36  syne.supervisor spawning syne-emotion
2026-06-14 12:40:37  [voice] syne-voice starting
2026-06-14 12:40:37  [swarms] syne-swarms starting
2026-06-14 12:40:37  [avatar] syne-avatar starting
2026-06-14 12:40:37  [emotion] syne-emotion: backend=stub ready
2026-06-14 12:40:37  [emotion] listening on /run/user/1000/syne/emotion.sock

(One of the 5 will likely fail to fully start unless you've built llama-cpp-python against the bundled CUDA 12 libs — see syne/llm/llama_cuda_runtime.py for the recipe. The rest degrades gracefully.)

---

Contributing

This is an experiment. PRs welcome, especially if you have:

• Latency measurements on different GPUs
• A working LiveKit deployment you can test against
• Backchannel-pool training data (recordings of you using a voice agent)
• Better sentence buffer / prosody algorithms
• A working flash-attn kernel for Pascal (please)

Issues with the voice-loop label get the fastest response.

License

This monorepo: MSAL v1.0 — see LICENSE. The published syne PyPI package: MIT — see LICENSE-MIT. Dual-license explanation: NOTICE.