NixOrb 🌐

Floating AI assistant orb for Arch Linux — KDE Plasma 6 Wayland — runs entirely on your machine

A complete ground-up remake of NixOrb with a focus on local AI, rock-solid Qt6 stability, and clean modular architecture.

Models

Every stage is pluggable. Point it at Ollama, at any model on the Hugging Face Hub, or at NVIDIA Nemotron for streaming speech.

pip install 'nixorb[hf]'        # any Hugging Face ASR / TTS / LLM
pip install 'nixorb[nemotron]'  # NVIDIA Nemotron 3.5 streaming ASR
pip install 'nixorb[quant]'     # 4-bit LLMs (bitsandbytes)

Install NixOrb into its own virtualenv, not a shared interpreter. The default ASR checkpoint needs transformers >= 5.13, and much of the local-AI ecosystem still pins transformers <= 4.5x — LLaMA-Factory, for one. Whichever you install second wins, pip prints a single dependency conflicts warning, and the loser breaks later from inside a model loader. ./install.sh already builds one at ~/.local/share/nixorb/venv; nixorb check tells you if you are in a shared environment and names anything you collide with.

Any Hugging Face model

# Speech recognition — anything with an automatic-speech-recognition tag
asr_backend = "huggingface"
asr_model   = "openai/whisper-small"      # distil-whisper, MMS, Wav2Vec2, …

# Thinking — any causal LM, streamed token by token
llm_backend  = "huggingface"
llm_hf_model = "Qwen/Qwen2.5-3B-Instruct"  # Llama, Phi, Gemma, SmolLM, …

# Voice — any text-to-speech model
tts_backend  = "huggingface"
tts_hf_repo  = "facebook/mms-tts-eng"      # SpeechT5, Bark, VITS, Parler, …

Gated repos need a token: set hf_token, or export HF_TOKEN.

NVIDIA Nemotron 3.5 ASR (streaming)

nvidia/nemotron-3.5-asr-streaming-0.6b is a 0.6B FastConformer-RNNT covering 40 language-locales with punctuation and capitalisation. It gets its own backend rather than going through the generic pipeline because it is cache-aware: it reuses encoder state across strictly non-overlapping chunks, so partial transcripts arrive while you are still talking instead of after you stop.

asr_backend  = "nemotron"
asr_model    = "nvidia/nemotron-3.5-asr-streaming-0.6b"
asr_language = "auto"       # or a locale: "en-US", "de-DE", "ja-JP", …
asr_streaming = true        # emit partials as you speak

# Right-context in 80ms frames — the latency/accuracy dial.
asr_nemotron_lookahead = 3  # 0→80ms  3→320ms  6→560ms  13→1120ms

With asr_language = "auto" the model detects the spoken language and reports it; NixOrb strips the <en-US> tag from the transcript and logs the detection.

Needs transformers >= 5.13 — that is where Nemotron3_5Asr landed, and it is the floor for the [hf] and [nemotron] extras. On an older install NixOrb names the version it found and the command to fix it, then falls back to offline transcription through the generic pipeline.

Architecture Overview

Quick Start

Install (Arch Linux)

git clone https://github.com/minerofthesoal/nixorb.git
cd nixorb
chmod +x install.sh
./install.sh

The installer puts everything in a virtualenv at ~/.local/share/nixorb/venv and links the launcher into ~/.local/bin. Installing by hand elsewhere, do the same — see the note under Models for why a shared interpreter breaks this project:

python -m venv ~/.local/share/nixorb/venv
~/.local/share/nixorb/venv/bin/pip install torch torchaudio   # or the CUDA build
~/.local/share/nixorb/venv/bin/pip install -e .
ln -sf ~/.local/share/nixorb/venv/bin/nixorb ~/.local/bin/nixorb

Python 3.12 – 3.14 are supported.

Start NixOrb

nixorb start          # Launch the floating orb
nixorb trigger        # Activate the running orb (bind to a KDE shortcut)
nixorb status         # Check system status
nixorb quit           # Shut the running orb down
nixorb ask "What is 2+2?"   # One-shot query
nixorb check          # Check dependencies

nixorb trigger talks to the running instance over a Unix socket in $XDG_RUNTIME_DIR, so it works the same under Wayland, X11, or a bare TTY — which is what makes it usable as a KDE global shortcut.

Talking to it

It is built to be used by voice, not read.

• Speaks while it thinks. Each sentence is synthesised as the model produces it, so the reply starts about a second in rather than after the whole answer is written. (tts_streaming)
• Talk over it. Triggering while it is speaking cuts playback off and starts listening, instead of queueing behind the current answer. (barge_in)
• Keep talking. After answering it listens again for a few seconds, so "and what about tomorrow?" needs no second hotkey press. (follow_up_seconds, 0 to disable)
• Answers made for ears. The default prompt asks for one or two spoken sentences, answer first, no markdown — and shell commands are suppressed from the speech rather than read out character by character.
• Live transcripts. With Nemotron streaming on, partial text arrives while you are still speaking.

Pipeline Flow

Trigger (Hotkey/WakeWord/Click)
  → Record Audio (sounddevice + VAD)
  → Transcribe (faster-whisper INT8, ~2.1GB VRAM)
  → Think (Ollama LLM stream, localhost:11434)
  → Speak (Piper TTS, offline)
  → Execute (<ACTION> sandboxed bash)

VRAM Budget (GTX 1080 8 GB)

Configuration

Edit ~/.config/nixorb/config.toml or use the GUI (right-click orb → Settings):

hotkey = "Ctrl+Alt+Space"
orb_size = 88
llm_model = "llama3.2"
ollama_host = "http://localhost:11434"
tts_backend = "piper"

# Requires the optional extra: pip install 'nixorb[wakeword]'
wake_word_enabled = false

# Ask before running every command the model emits.
require_action_confirmation = true

See config/default.toml for every option with comments.

Orb States

The orb has no caption — its colour is the state, and the tray tooltip spells it out in words.

Keyboard Shortcuts

Project Structure

nixorb/
├── core/           # Event bus, VRAM manager, IPC control socket
├── asr/            # base (mic + VAD) · faster-whisper · HF · Nemotron
├── llm/            # Ollama · Hugging Face · backend factory
├── tts/            # Piper · Hugging Face · streaming Speaker
├── ui/             # Qt6 orb, tray, settings, hotkey, confirm dialog
├── action/         # Confirmed command execution + clipboard
├── memory/         # ChromaDB vector memory
├── plugins/        # Plugin loader and tool dispatch
├── utils/          # Paths, logging, web search
├── vision/         # Screen capture
├── hf.py           # Shared Hugging Face device/dtype/auth plumbing
├── main.py         # Entry point and the conversation loop
├── cli.py          # CLI interface
└── settings.py     # Configuration

Dependencies

• System: python 3.12+, qt6-base, qt6-declarative, qt6-wayland, portaudio, wl-clipboard, grim
• LLM: ollama, or pip install 'nixorb[hf]' for in-process models
• AUR: piper-tts (speech). Note its binary is piper-tts — Arch's piper package is the unrelated gaming-mouse tool. espeak-ng is the fallback if Piper is missing.
• Python: See requirements.txt / pyproject.toml

Troubleshooting

NixOrb writes everything it does to ~/.local/share/nixorb/logs/nixorb.log. Start with nixorb status — it reports whether Ollama is actually reachable and whether the configured model is installed.

Testing

pip install -e ".[dev]"
pytest tests/ -v

License

MIT © NixOrb Contributors