NixOrb 🌐

Floating AI assistant orb for Arch Linux — KDE Plasma 6 Wayland — runs entirely on your machine

A complete ground-up remake of NixOrb with a focus on local AI, rock-solid Qt6 stability, and clean modular architecture.

Models

Every stage is pluggable. Point it at Ollama, at any model on the Hugging Face Hub, or at NVIDIA Nemotron for streaming speech.

pip install 'nixorb[hf]'        # any Hugging Face ASR / TTS / LLM
pip install 'nixorb[nemotron]'  # NVIDIA Nemotron 3.5 streaming ASR
pip install 'nixorb[quant]'     # 4-bit LLMs (bitsandbytes)

Install NixOrb into its own virtualenv, not a shared interpreter. The default ASR checkpoint needs transformers >= 5.13, and much of the local-AI ecosystem still pins transformers <= 4.5x — LLaMA-Factory, for one. Whichever you install second wins, pip prints a single dependency conflicts warning, and the loser breaks later from inside a model loader. ./install.sh already builds one at ~/.local/share/nixorb/venv; nixorb check tells you if you are in a shared environment and names anything you collide with.

Any Hugging Face model

# Speech recognition — anything with an automatic-speech-recognition tag
asr_backend = "huggingface"
asr_model   = "openai/whisper-small"      # distil-whisper, MMS, Wav2Vec2, …

# Thinking — any causal LM, streamed token by token
llm_backend  = "huggingface"
llm_hf_model = "Qwen/Qwen2.5-3B-Instruct"  # Llama, Phi, Gemma, SmolLM, …

# Voice — any text-to-speech model
tts_backend  = "huggingface"
tts_hf_repo  = "facebook/mms-tts-eng"      # SpeechT5, Bark, VITS, Parler, …

When the local model won't load

llm_backend = "auto" prefers whatever is local and answers through the T1 API only when the local model cannot serve — a GGUF whose architecture your llama.cpp predates, a first run with nothing downloaded, a machine with no GPU and no patience. The decision is made once at startup and logged.

llm_backend  = "auto"          # local first, T1 as the safety net
t1_base_url  = "https://t1.example.org"
t1_api_key   = "…"
t1_model     = ""              # blank lets T1 route through your plan

pip install 'nixorb[t1]'
nixorb models --remote         # what your T1 server serves

Leave t1_base_url blank and nothing is ever contacted — auto is then just the local backend with a clearer error.

Naming models the short way

With nixorb[hypernix] installed, llm_model accepts any name from hypernix's catalogue, so you needn't know the owner:

nixorb models qwen     # 23 matches, name and repo id side by side

llm_model = "qwen3.5-4b"   # resolves to Qwen/Qwen3.5-4B

Speech without an ML stack

torch is the usual reason a machine cannot run the good models. Two engines avoid it entirely:

pip install 'nixorb[kokoro]'   # neural voice, 82M params, onnxruntime
pip install 'nixorb[vosk]'     # offline ASR, 40 MB, fetches its own model

tts_backend = "kokoro"
tts_voice   = "af_heart"      # or am_michael, bf_emma, …

asr_backend = "vosk"
asr_model   = "en-us"         # a language, a model name, or a directory

Kokoro sounds like a person and runs on a CPU. Vosk is less accurate than Whisper and does not punctuate, but it is 40 MB and never fails to load — the floor NixOrb can always reach.

Any Piper voice

Piper ships no voices. Name any of the ~180 in rhasspy/piper-voices and it is fetched on first use into ~/.local/share/piper/voices:

tts_backend = "piper"
tts_voice   = "en_GB-northern_english_male-medium"
# or "de_DE-thorsten-high", "fr_FR-siwis-medium", "ja_JA-hi_fi_captain-medium", …
# or the path of a .onnx you trained or downloaded yourself:
# tts_voice = "/home/me/voices/my-voice.onnx"

tts_download_voices = true   # false to stay offline and use only what is on disk

Names are locale-speaker-quality, and quality is one of x_low, low, medium, high — bigger is better and slower. Try one without restarting the orb:

nixorb tts "Testing this voice."

Gated repos need a token: set hf_token, or export HF_TOKEN.

NVIDIA Nemotron 3.5 ASR (streaming)

nvidia/nemotron-3.5-asr-streaming-0.6b is a 0.6B FastConformer-RNNT covering 40 language-locales with punctuation and capitalisation. It gets its own backend rather than going through the generic pipeline because it is cache-aware: it reuses encoder state across strictly non-overlapping chunks, so partial transcripts arrive while you are still talking instead of after you stop.

asr_backend  = "nemotron"
asr_model    = "nvidia/nemotron-3.5-asr-streaming-0.6b"
asr_language = "auto"       # or a locale: "en-US", "de-DE", "ja-JP", …
asr_streaming = true        # emit partials as you speak

# Right-context in 80ms frames — the latency/accuracy dial.
asr_nemotron_lookahead = 3  # 0→80ms  3→320ms  6→560ms  13→1120ms

With asr_language = "auto" the model detects the spoken language and reports it; NixOrb strips the <en-US> tag from the transcript and logs the detection.

Needs transformers >= 5.13 — that is where Nemotron3_5Asr landed, and it is the floor for the [hf] and [nemotron] extras. On an older install NixOrb names the version it found and the command to fix it, then falls back to offline transcription through the generic pipeline.

Architecture Overview

Quick Start

Install (Arch Linux)

git clone https://github.com/trail-b1az3r/nixorb.git
cd nixorb
chmod +x install.sh
./install.sh

The installer puts everything in a virtualenv at ~/.local/share/nixorb/venv and links the launcher into ~/.local/bin. Installing by hand elsewhere, do the same — see the note under Models for why a shared interpreter breaks this project:

python -m venv ~/.local/share/nixorb/venv
~/.local/share/nixorb/venv/bin/pip install torch torchaudio   # or the CUDA build
~/.local/share/nixorb/venv/bin/pip install -e .
ln -sf ~/.local/share/nixorb/venv/bin/nixorb ~/.local/bin/nixorb

Python 3.12 – 3.14 are supported.

Configure it for your machine

nixorb setup            # looks at what you have, writes settings that run
nixorb setup --dry-run  # show the plan, change nothing

The shipped defaults aim high — streaming Nemotron ASR, a reasoning GGUF, a neural voice. On a machine with that stack it is a good assistant; without it, every stage fails somewhere different and the orb starts, listens, and says nothing. nixorb setup looks first and prefers the boring option: faster-whisper over Nemotron, Ollama over an in-process GGUF, a voice that is already installed over one that must be fetched. It names every choice and what to install to do better.

install.sh runs it for you, and a first start with no config at all makes the same choices by itself — so this is a command you only need when something changes, not to get going. A config you have edited is never overwritten.

Start NixOrb

nixorb start          # Launch the floating orb
nixorb trigger        # Activate the running orb (bind to a KDE shortcut)
nixorb status         # Check system status
nixorb quit           # Shut the running orb down
nixorb ask "What is 2+2?"   # One-shot query
nixorb check          # Check dependencies

nixorb trigger talks to the running instance over a Unix socket in $XDG_RUNTIME_DIR, so it works the same under Wayland, X11, or a bare TTY — which is what makes it usable as a KDE global shortcut.

Talking to it

It is built to be used by voice, not read.

• Speaks while it thinks. Each sentence is synthesised as the model produces it, so the reply starts about a second in rather than after the whole answer is written. (tts_streaming)
• Talk over it. Triggering while it is speaking cuts playback off and starts listening, instead of queueing behind the current answer. (barge_in)
• Keep talking. After answering it listens again for a few seconds, so "and what about tomorrow?" needs no second hotkey press. (follow_up_seconds, 0 to disable)
• Answers made for ears. The default prompt asks for one or two spoken sentences, answer first, no markdown — and shell commands are suppressed from the speech rather than read out character by character.
• Live transcripts. With Nemotron streaming on, partial text arrives while you are still speaking.

Pipeline Flow

Trigger (Hotkey/WakeWord/Click)
  → Record Audio (sounddevice + VAD)
  → Transcribe (faster-whisper INT8, ~2.1GB VRAM)
  → Think (Ollama LLM stream, localhost:11434)
  → Speak (Piper TTS, offline)
  → Execute (<ACTION> sandboxed bash)

VRAM Budget (GTX 1080 8 GB)

Configuration

Edit ~/.config/nixorb/config.toml or use the GUI (right-click orb → Settings):

hotkey = "Ctrl+Alt+Space"
orb_size = 88
llm_model = "llama3.2"
ollama_host = "http://localhost:11434"
tts_backend = "piper"

# Requires the optional extra: pip install 'nixorb[wakeword]'
wake_word_enabled = false

# Ask before running every command the model emits.
require_action_confirmation = true

See config/default.toml for every option with comments.

Orb States

The orb has no caption — its colour is the state, and the tray tooltip spells it out in words.

Keyboard Shortcuts

Project Structure

nixorb/
├── core/           # Event bus, VRAM manager, IPC control socket
├── asr/            # base (mic + VAD) · faster-whisper · HF · Nemotron
├── llm/            # Ollama · Hugging Face · backend factory
├── tts/            # Piper · Hugging Face · streaming Speaker
├── ui/             # Qt6 orb, tray, settings, hotkey, confirm dialog
├── action/         # Confirmed command execution + clipboard
├── memory/         # ChromaDB vector memory
├── plugins/        # Plugin loader and tool dispatch
├── utils/          # Paths, logging, web search
├── vision/         # Screen capture
├── hf.py           # Shared Hugging Face device/dtype/auth plumbing
├── main.py         # Entry point and the conversation loop
├── cli.py          # CLI interface
└── settings.py     # Configuration

Dependencies

• System: python 3.12+, qt6-base, qt6-declarative, qt6-wayland, portaudio, wl-clipboard, grim
• LLM: ollama, or pip install 'nixorb[hf]' for in-process models
• AUR: piper-tts (speech). Note its binary is piper-tts — Arch's piper package is the unrelated gaming-mouse tool. espeak-ng is the fallback if Piper is missing.
• Python: See requirements.txt / pyproject.toml

Troubleshooting

NixOrb writes everything it does to ~/.local/share/nixorb/logs/nixorb.log. Start with nixorb status — it reports whether Ollama is actually reachable and whether the configured model is installed.

Testing

pip install -e ".[dev]"
pytest tests/ -v

License

MIT © NixOrb Contributors