voiceio

Voice dictation that learns you. (website) Local, open source, private — and it fine-tunes itself on your own voice, on your own machine.

Off-the-shelf speech models stumble on exactly the words you say most: your colleagues' names, your project, your jargon. voiceio keeps (with your permission) the audio of what you dictate, has a bigger model transcribe it overnight, and trains a personal model from it — nothing uploaded, no GPU required. On the author's dictation, one overnight run on a laptop CPU cut word errors from 12.1% to 5.0% on held-out clips and recognized more of the author's vocabulary, at the same speed. Every model is scored on clips it never trained on, and it only replaces the current one when it wins.

https://github.com/user-attachments/assets/9cf5d1ac-b4bb-4cf8-b775-7a66dc16b376

Press a hotkey, speak, and the text streams into whatever app has focus. Press again to commit.

• Learns you. voiceio learn fine-tunes a model on your own voice; a vocabulary steers it toward new names right away; corrections fix the rest. See Make it yours.
• Local. Speech is decoded on-device (faster-whisper by default). Audio never leaves the machine and there is no telemetry. Two optional, off-by-default features send text (never audio) to an LLM endpoint you configure: final-transcript polish ([postcorrect]) and an optional local Ollama cleanup ([llm]).
• Yours. History, retained audio, corrections, vocabulary and fine-tuned models are plain local files. See Your data.
• Linux-first. Wayland and X11, GNOME, KDE, Hyprland, sway and i3. Backends are picked by probing what works. See Linux support.

Upgrading from 0.x? See UPGRADING.md — 1.0 renames a few commands and moves desktop dependencies to the [desktop] extra.

Quickstart

# 1. System packages: a C toolchain (evdev has no wheel), PortAudio, IBus + GObject bindings
sudo apt install pipx build-essential python3-dev portaudio19-dev ibus gir1.2-ibus-1.0 python3-gi   # Debian/Ubuntu
sudo dnf install pipx gcc gcc-c++ make python3-devel portaudio-devel ibus ibus-libs python3-gobject   # Fedora
sudo pacman -S python-pipx base-devel portaudio ibus python-gobject                                   # Arch

# 2. voiceio with the desktop extra (microphone, hotkeys, tray)
pipx install 'python-voiceio[desktop]'

# 3. Guided setup: model download, hotkey, autostart
voiceio setup

Then press Ctrl+Alt+V, speak, and press it again. voiceio doctor tells you what is working and voiceio doctor --fix repairs what it can.

NixOS users: see NixOS. Agents installing voiceio non-interactively: see INSTALL.md.

git clone https://github.com/Hugo0/voiceio && cd voiceio
uv sync --extra desktop --extra dev
uv run voiceio setup      # also links voiceio into ~/.local/bin

Commands

Note: the CLI is being restructured for v1.0. This list is the target layout. Check it against voiceio --help before release.

Your data

Everything voiceio keeps is in three directories (they follow XDG_CONFIG_HOME, XDG_STATE_HOME and XDG_DATA_HOME). Files are created 0600, directories 0700. Nothing is uploaded. Recordings, decode traces and window titles are off unless you turn them on — voiceio setup asks whether to keep recordings so voiceio can learn your voice. Logs record sizes and timings, never your words.

Set [data] capture_context = false to stop recording window titles. voiceio uninstall offers to delete the config and state directories; delete ~/.local/share/voiceio yourself.

Make it yours

Most dictation errors are proper nouns and jargon: PostgreSQL heard as post grass, a colleague's name spelled three ways. There are three tools for that, from cheapest to most powerful.

1. Vocabulary. Put your names and terms in ~/.config/voiceio/vocabulary.txt (or use voiceio vocab). They are passed to Whisper as hotwords on every decode, ranked by how often you use them. One honest limit: Whisper's hotword channel is capped at 223 tokens, and a proper noun costs 5 to 7 tokens, so only about 35 terms steer each decode. The rest are kept and ranked, but they do nothing until they rank high enough. This ceiling comes from the model, not from voiceio. CONTRIBUTING.md explains why we still use Whisper.

2. Corrections. Find/replace rules for what still comes out wrong (voiceio corrections add WRONG RIGHT, or say "correct that" right after a mistake and review with voiceio corrections flagged). voiceio vocab suggest lists unusual words from your history — offline — so you can add the real names to your vocabulary and turn the mishearings into corrections.

3. Fine-tuning with voiceio learn. This is how you get past the 35-term ceiling: teach the model your vocabulary directly, from your own retained audio.

voiceio learn import DIR --source phone   # optional: add your own recordings from elsewhere
voiceio learn label      # a stronger teacher model (large-v3-turbo) transcribes your clips
voiceio learn review     # fix the teacher where it matters: your names and terms
voiceio learn review --test   # verify the held-out clips, so eval scores against the truth: it asks only
                              #   where the teacher and your live decode disagree, quickest clips first
voiceio learn augment    # synthesize extra examples of rare terms with local TTS
voiceio learn build      # freeze a versioned train/dev/test dataset (the test split never trains)
voiceio learn eval       # score the current model on your held-out clips: WER and term recall
                         #   --streaming replays them through live dictation and times stop→text
voiceio learn train      # LoRA fine-tune (needs the `train` extra; decoder-only on CPU, --full on a GPU)
voiceio learn promote <run>   # dictate with the fine-tune; `voiceio learn rollback` undoes it
voiceio learn maintain   # later: label what's new, and once 50+ clips piled up, retrain and compare
voiceio learn schedule on   # …or let it run `maintain` on a schedule, and notify you when a model wins
                            #   --at 02:00 --every week: when (default daily at 03:30)
                            #   --auto-promote: switch to a winner by itself; `learn rollback` undoes it
voiceio learn stop          # stop a run in progress (it goes again next time)

Scheduled runs wait for a quiet machine: they start only when the 1-minute load per CPU is at most [learn] max_load (0.25) and you're on mains power, and give up after wait_hours (3) until the next run. When one starts you get a notification with Stop for now and Weekly instead.

To keep a second machine (say, the server your phone dictates through) on the same model, set [learn] on_promote = "scripts/push_model.sh HOST": after every switch it copies the model there and flips a current link the server points at.

Labeling, training and offline eval run in an idle-priority cgroup, so they never slow dictation down. Everything stays on your machine unless you choose train --remote.

voiceio learn status shows where you are, each model's scores, and suggests promote only when a fine-tune wins on everything. promote itself refuses a model that hasn't, and reports how many clips got better or worse with a confidence interval — a few dozen test clips can move an average by luck.

Is it actually helping? Once some clips are labeled, measure instead of guessing:

voiceio learn eval --pipeline   # what each text stage (numbers, commands, corrections, cloud
                                # postcorrect…) did to your past dictations, and its commonest edits
voiceio corrections audit       # rules your own audio contradicts, and ones that look risky
voiceio learn eval --streaming  # accuracy *and* stop→text, replaying clips through live dictation

On the author's data these found a correction that kept turning "composable" into "comparable", a number rule writing "no one" as "no 1", and a cloud pass that never fixed a single vocabulary term.

Choose a model

voiceio models list           # known models, your fine-tunes, and which is active
voiceio models bench          # score candidates on your held-out clips (after `voiceio learn build`)
voiceio models use medium     # switch

Pick on your own audio, not leaderboards: for dictation full of names, how well a model can be biased matters more than its benchmark WER.

Licenses. voiceio ships no model weights. Each model is downloaded from its own source on first use, under its own license. The defaults are permissive (MIT). A fine-tune you make with voiceio learn inherits the license of the base model it was trained from.

Dictate from your phone

python -m voiceio.server (the server extra) serves the same engine over HTTP and WebSocket, for a phone or any other client: POST /transcribe with an audio file, or stream PCM over /ws/transcribe for live partials. It binds 127.0.0.1 by default; put it behind your VPN or a reverse proxy and set VOICE_TOKEN (bearer token) and VOICE_ALLOWED_ORIGINS when it is reachable from elsewhere. VOICE_MODEL, VOICE_LANES, VOICE_HOST/VOICE_PORT override the rest.

The server reads its own machine's voiceio config, so it gets your vocabulary and corrections once you copy them over:

scripts/sync_personalization.sh myserver           # vocabulary + corrections
scripts/sync_personalization.sh myserver --model   # …and your promoted fine-tune

To fine-tune on what you dictate from the phone too, opt in on the server with [data] retain_audio = true in its config.toml, then bring the clips home: scripts/pull_phone_dictation.sh myserver (an rsync over SSH followed by voiceio learn import). Only your own voice should go in: the model learns whoever is speaking.

Linux support

Input methods. IBus is the only way to get a live, underlined preview that is corrected in place on Wayland. voiceio installs its own IBus engine and only activates it while you dictate. If fcitx5 is your input method, voiceio leaves it alone and types the final text with wtype or ydotool. Terminals that ignore input methods get the clipboard.

Hotkeys without the input group. The evdev backend reads keyboards from /dev/input. voiceio doctor --fix installs a udev rule, /etc/udev/rules.d/60-voiceio-uaccess.rules, that grants the logged-in user access to keyboard devices. It works immediately, without logging out. The old route still works as a fallback: sudo usermod -aG input $USER, then log out and back in. You can also skip evdev and bind voiceio toggle in your compositor.

ydotool needs write access to /dev/uinput; voiceio doctor --fix installs a uaccess rule for that too.

NixOS

The repository is a flake. It has not been built yet; please report fixes.

# flake inputs: voiceio.url = "github:Hugo0/voiceio";
# NixOS: package, uinput, and the keyboard uaccess rule
imports = [ voiceio.nixosModules.default ];
programs.voiceio.enable = true;

# Home Manager: config.toml from Nix + a systemd user service
imports = [ voiceio.homeManagerModules.default ];
services.voiceio = {
  enable = true;
  settings.model.name = "small";
};

For the streaming preview, enable IBus as your input method: i18n.inputMethod = { enable = true; type = "ibus"; };.

Windows and macOS

There is a best-effort pynput path for both (hotkeys and typing, no live preview). It is untested and unmaintained, with no installers or CI. Patches are welcome; bug reports will likely sit.

Troubleshooting

Logs: journalctl --user -u voiceio or ~/.local/state/voiceio/voiceio.log.

Roadmap

• Unbounded vocabulary — move to a decoder whose biasing has no token cap (see model choice).
• One model everywhere — sync your fine-tune to your phone server and laptops, peer to peer.

Contributing

See CONTRIBUTING.md for the architecture, conventions and the reasoning behind the model choice. Please open an issue before a large PR.

License

MIT. Model weights are not part of voiceio and carry their own licenses (see Choose a model).