personal-jarvis 2.0.0

Your personal AI ecosystem, controlled entirely by voice.

It drives coding agents, runs shell commands, operates your computer, connects anything that speaks MCP, dictates into any app, and remembers everything.
Open source, and it can run fully on your own hardware, with no cloud account anywhere in the chain.

_{Real time, not sped up. One spoken command, and it takes the screen and does it.}

Install it in one command

Windows (PowerShell)

irm https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.ps1 | iex

macOS and Linux

curl -fsSL https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.sh | bash

Python 3.11+ and Git, nothing else. The installer asks nothing in the terminal, and the app walks you through language, wake word, and keys once. Full install notes below.

A typical voice assistant talks back. Personal Jarvis does the thing. At the center of every voice conversation sits a tool model: it decides how much a request actually needs, runs shell commands, takes the mouse and keyboard, and reaches any service that speaks MCP. The short stuff it handles itself. Anything heavier goes to a coding-agent worker running on whichever CLI or key you already have: Claude Code, Codex, the Gemini CLI, Grok Build, or an in-process worker on a plain API key. That worker runs in isolation, gets checked by a critic, and reports back in the language you spoke.

You do not have to talk to it, either. The home screen has a typed chat that goes through the same brain on the same keys, and the Agentic IDE has a second one where the seat is a coding CLI instead of an API provider. Both take files you drop, paste, or pick.

Every tier has a keyless local option, so the whole assistant can run on your own hardware with no cloud account anywhere in the chain. Runs on your own hardware, below, has the detail. If you would rather use a hosted model, you pick the provider per tier: Gemini, Claude, OpenAI, or OpenRouter, one setting for each. It can rewrite its own configuration, and it runs on a headless server just as well as on a desktop with a microphone.

_{The home view, waiting. Every section is one click away in the sidebar, and the bar across the bottom is the live voice channel. The assistant answers to whatever wake word you pick; this install answers to George.}

What you can say

All six work today; none of this is a roadmap item. Two need extra setup: the phone call needs the optional [telephony] extra plus your own Twilio account, a number, and a publicly reachable HTTPS URL for the webhooks, and computer use needs a desktop install with a screen, not the headless one.

Which model answers you

Three models sit behind that table, and you never choose between them: the handover happens mid-sentence, based on what the request needs.

The realtime model carries the conversation itself. It hears you and answers in under a second, built for talking, not for thinking hard. The moment a request needs an actual tool, that turn hands off to a second model, slower and noticeably smarter, the one that reads your wiki, changes a setting, places the call, or takes the screen. It answers in the same voice, so from where you're sitting it never stopped being one conversation. Real work, the kind that takes minutes, goes to a third: a coding agent running in its own isolated copy of the workspace, reviewed by a critic, that comes back with a file instead of just an answer.

In the app this lives on one screen: API Keys has one tab per tier, each with its own provider, and you only need keys for the tiers you use. Every provider says how it bills: a subscription login you already have, or an API key charged per token.

Runs on your own hardware

Hosted providers are a choice here, not a requirement. Every layer that could reach for a cloud account has a local option that needs no API key and no signup, so a complete install can keep your voice, your screen and your files on the machine they started on.

Mixing is normal and expected: a local recognizer with a hosted brain, or a local brain with a hosted voice. Nothing forces the whole chain one way.

You do not have to know which model fits which job. The Local models section reads the server and the graphics memory it has to work with, then fills the four jobs a local model can take over: chat, voice, tools and screen, and the deep coding work. One button does all four. It starts the server if it is stopped, prefers whatever you already have on disk over a fresh download, fetches only what is genuinely missing, and writes the settings.

Then it checks its own work. Four real round trips run at the end: the server answers, the chat pick produces an answer, the voice pick calls a tool, and the tools and screen pick reads an image. Each reports on its own line with how long it took, and a job you never configured says so rather than quietly counting as a pass. So you find out the setup works before you rely on it instead of after. It finishes by offering to start the server with Jarvis, which keeps the first answer of the day from paying the model load time. Nothing downloads without a click that names the download first.

Two capabilities stay outside this promise, and it would be dishonest to imply otherwise. The outbound phone call goes over Twilio, which is a hosted service by definition, and the coding-agent workers run on whichever agent CLI or API key you point them at. Everything in the table above is genuinely local.

Demo

The clip at the top is the short version. Watch the full demo on YouTube, where one voice command takes the screen and changes a Windows setting live, or browse the channel for the rest.

What it does differently

The router itself stays small. It works out what you said, picks a tool or a worker, and gets out of the way; there is no single giant prompt trying to be everything. Anything non-trivial runs as a mission in an isolated worktree and gets a critic's review before you ever hear the result. You are not left listening to silence while that happens, either: the moment the router picks an action, Jarvis says one line about that specific action, not a generic "working on it".

Providers are interchangeable, and that matters most on the day one of them fails. If the configured provider is unreachable or out of quota, Jarvis crosses to a different provider family instead of leaving you stuck. Workers run on a subscription login or a pay-per-token key, whichever you have. Speech and voice providers can be switched by voice mid- conversation; the brain provider cannot, on purpose, that one stays yours to change from the app or the CLI.

It also remembers: a Knowledge Wiki of plain Markdown files, plus an awareness layer, build up a picture of you across sessions. And it can change its own settings through a guarded, audited pipeline, the full mechanics are under Self-modification below.

How it works

Higher layers can only reach lower ones through protocols; everything else talks over a typed, immutable EventBus. That's the strict seam that makes harnesses, providers, and plugins swappable in the first place.

L7  UI/UX           Desktop app (FastAPI + React + pywebview), tray, Orb overlay
L6  Orchestrator    State machine, Router, BrainManager, Mission-Manager + workers, Controller
L5  Harness adapter python-script, computer-use  (coding agents are L6 mission workers)
L4  Brain           Gemini · Claude · OpenAI · Grok · OpenRouter  +  sub-second Ack-Brain
L3  Intent / Risk   Classifier, four-tier risk policy, approval, rate-limit tracking
L2  Speech          Wake → VAD → STT → TTS  (cloud or local, your choice)
L1  Audio I/O       Device routing, chime feedback
L0  OS / Hardware   Mic, speakers, global hotkeys, optional GPU

Install

One command on Windows, macOS, or Linux. You need Python 3.11 or newer and Git; the installer checks for both and stops with a download link if one is missing. It asks nothing in the terminal. It launches the app, and the app walks you through a one-time setup for language, wake word, and API keys.

What it costs: nothing to us. Personal Jarvis is open-source software (Apache 2.0) you run on your own machine. There is no subscription for it, no paid tier, no marketplace cut, and no referral link behind any provider named on this page. What you do need is access to a model, and that is billed by whoever provides it, straight to you. An AI subscription you already pay for works, and so does a pay-per-token API key. The same goes for the optional pieces: a phone call runs on your own Twilio account at Twilio's prices.

Windows (PowerShell)

irm https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.ps1 | iex

macOS and Linux

curl -fsSL https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.sh | bash

This is open source, so read the installer before you run it. It creates a venv, installs dependencies, prefetches the voice models, and launches the app. Your keys land in your operating system's credential manager, never in the repo. Re-running the same one-liner updates in place.

Removing it again is one command too, under Uninstall.

pipx install personal-jarvis && jarvis serve

pip install personal-jarvis          # cloud-first base: API + WebSocket + browser UI
pip install "personal-jarvis[full]"  # everything: desktop app, telephony, channels, local voice

git clone https://github.com/PersonalJarvis/PersonalJarvis
cd PersonalJarvis
python -m venv .venv && source .venv/bin/activate   # Windows: .\.venv\Scripts\Activate.ps1
pip install -e .[full]
jarvis serve

Run it

jarvis          # full desktop: window + voice + Orb overlay
jarvis serve    # headless server: API + WebSocket + browser UI, no local audio needed

What's inside

Missions

Anything non-trivial, say "research X and write me a report", spawns a worker in an isolated git worktree: a private sandbox copy of the workspace, with crash containment. A critic reviews the result, for up to three rounds, before you ever hear it, and deliverables land in Artifacts as downloadable files.

Automations

Work that repeats without being asked for: a morning brief, a weekly review, a digest of the unread mail, a watch on the topics you follow. Ten ready-made ones sit in a catalogue, where you add one and adjust its schedule. Or describe your own in plain language and it becomes a job with a schedule, a run history, and an honest error when a run fails instead of silent nothing. One-off moments ("do this on Friday at nine") live in the same place.

Skills

A skill is a written-down procedure Jarvis can follow. It is one Markdown file, SKILL.md, with a short YAML header and a body of steps, sitting in a folder on your disk. There is no pip install and no restart: save the file and it is loaded. Changing what a skill does means editing the steps, which is the point of keeping them in Markdown rather than in code.

A skill fires from a spoken phrase, a hotkey, or a cron schedule. Thirty-one ship with the app, mostly one per connected service, so "put that in Notion" or "start a deep work block" already has a procedure behind it. You can write your own, describe one in plain language and have Jarvis draft it, or import someone else's from the Marketplace.

A matched skill is a takeover, not a suggestion, so what may fire on its own is deliberately narrow. A skill that only gives the assistant instructions can match and run. One that reaches an integration keeps the model free to refuse. One that starts a process or a worker never fires from a guess: you name it, or the model asks for it explicitly. A skill Jarvis writes for you lands as a draft, and a draft is never switched on for you.

The decision about whether a skill matches what you just said, and every check that can veto it, lives in one module that the assistant, the in-app match tester and an offline evaluation all call. So the panel showing you why a skill did or did not fire cannot disagree with what actually happened; it is running the same code.

Agentic IDE

Pick a folder, choose how many terminals to open and which agent runs in each one, and you get a grid of real terminals inside the app. Nine coding agents are registered: Claude Code, Codex, Cursor CLI, OpenCode, Kimi Code, GLM Coding Plan, Grok Build, Antigravity and DeepSeek Harness. A pane can also just be your own shell, for the times the job is a git rebase rather than a question for an agent. Each entry brings its own detection, install command and sign-in, so picking one is the whole setup, and the registry is a list rather than a code path, so a new CLI is an entry in it.

Every terminal carries a spoken call sign (Mika, Nova, Aria), so the whole workspace is addressable by voice: "what is Mika doing?", "tell Nova to run the tests". A focus mode narrows Jarvis to that workspace for as long as you want, then switches back cleanly.

A terminal is not the only way to read one of these sessions. Switch a pane to its chat and the same run reads as a conversation: the reply, the reasoning where the CLI shares it, and every tool call with the result it returned, as rows you can scroll. The terminal is one click away and one click back, and the pane keeps running either way.

_{A prompt lands in terminal 1, carrying the task, the key files and how that part of the code works today. The counter underneath shows how long that agent has been thinking · watch the full Agentic IDE demo on YouTube}

Knowledge Wiki

An Obsidian-compatible Markdown vault that Jarvis reads and writes. Tell it something once and every future session knows it. Because it is plain files on your disk, you can read, edit, and sync it yourself.

Computer use

Ask for something that has no API, and Jarvis takes the mouse and keyboard: opening apps, clicking, typing, navigating. There is no scripted path per application. It works the way you would.

The loop is perceive, act, verify. Jarvis takes a screenshot, a vision model says what to click next, the click is made through the platform's own input layer, and then it looks again to check that what it intended actually happened. Two details keep that honest. Coordinates are resolved against the exact frame the model saw, not against a stale picture of the screen, so a window that moved between two steps cannot send a click into nothing. And every action goes through a ledger that refuses duplicates, so a model that repeats itself does not click Send twice.

While it drives, a border sits around the screen so you can see it is not you. That border comes from a small Qt companion in the [desktop] extra. Where the companion is absent, on a base or headless install and on aarch64 Linux, it degrades to a logged no-op and the control itself still works.

Channels and telephony

The desktop window, the browser, Telegram, and Discord all reach the same brain and share the same memory. Real outbound phone calls are possible but not out of the box: they need the optional [telephony] extra, your own Twilio account and number, and a publicly reachable HTTPS URL that Twilio can call back for the voice webhook and the media socket.

Safety tiers

Every action is classified as safe, monitor, ask, or block before it runs. Destructive things ask first, whitelisted routines stop nagging you, and the blacklist always outranks the whitelist.

Self-modification

Jarvis can change its own settings by voice, through a guarded pipeline that validates, backs up, applies, verifies, and rolls back on failure, with a full audit trail. Some things are deliberately out of its reach: secrets and keys, the safety tiers, the review gates, and the active brain provider, which only you can change from the app or the CLI. The same restraint applies to the skills it writes for you, which land as drafts. Nothing self-activates.

Dictation

Hold a key, talk, and the text lands in whatever field has focus, in any application: a browser, an editor, a chat box, some internal tool's form. Jarvis writes it through the clipboard, sends the paste chord, and restores your old clipboard afterward; if a paste doesn't land, it tells you instead of silently losing what you said. One key is a hold, a second is a toggle, both can be armed together, and a third re-pastes the last thing you dictated into whatever field just ate it.

_{Speak, release the key, and the cleaned-up sentence lands in whatever field has focus.}

Recognition runs on whatever speech provider you have configured, local model included, in which case your voice never leaves the machine. Cleanup happens in two passes: a plain pattern match strips filler sounds per language with no model involved at all, and an optional second pass, capped by a hard latency budget, handles punctuation, capitalization, false starts, and spoken numbers, falling back to your raw transcript unchanged if it fails. A separate pass translates instead of cleaning, writing what you said in one fixed target language. Words the recognizer keeps getting wrong go into a dictionary you control. Everything, raw transcript, cleaned version, and the original audio, stays on disk: you can see exactly what a cleanup pass changed, revert it, or retry a transcription that came back empty because a provider was briefly down.

Realtime voice

An optional speech-to-speech mode (OpenAI Realtime, Gemini Live) for sub-second conversational latency, with automatic fallback to the classic wake, STT, brain, TTS pipeline when it is unavailable.

Artifacts

Everything a run produced, in one place, shown rather than listed. A report opens as the document it is, a generated page renders inside a sandbox, a picture appears at full size. Files are downloadable from there. This replaced the old Outputs section; there is no second list of runs anywhere.

Spend

What you have actually paid, per provider, per model and per job. The app meters its own calls through one ledger and prices them, and it reads the usage your coding CLIs record on disk, so a subscription seat and a per-token key show up in the same view. You can separate the two, because a Claude Code seat you already pay for and an API call billed by the token are different questions.

Marketplace

Plugins, skills and wallpapers the community published, browsable and installable from inside the app behind one GitHub sign-in. Nothing here routes through a website you have to visit. The index is a public registry on GitHub, which is why browsing kept working after the hosted storefront was taken down.

Wallpaper gallery

504 wallpapers across 23 art styles, from oil painting and pixel art to synthwave and woodblock, each tagged light or dark so the app can match the wallpaper to your theme. Filter by style or by light and dark, preview fullscreen, mark favorites, or add your own image. The app opens on a plain colour by default; picking a picture here is what switches it, and one control puts the plain ground back.

Drive it from the terminal

The jarvis CLI (aliases jarvisctl, jctl) controls a running instance. Same actions as the app, same safety checks, just scriptable. Anything you can click, you or your scripts or another coding agent can type:

jarvis system status          # {"reachable": true} when Jarvis is up
jarvis --json brain status    # which provider is live, as machine-readable JSON
jarvis api <tag> <op>         # EVERY REST endpoint, auto-generated from OpenAPI

It is a thin client over the local REST API on 127.0.0.1:47821, so it inherits every guardrail (risk tiers, atomic config writes, the audit log) instead of going around them. Full guide: docs/jarvis-cli.md.

Configuration

You do not need a config file. Every setting has a built-in default, and the one-time in-app setup covers the rest. For finer control there is one optional, documented file (jarvis.toml.example):

[profile]
language = "auto"          # "de" | "en" | "auto" (bilingual auto-detect)

[trigger.wake_word]
phrase = ""                # YOUR word; nothing is preset for you
engine = "auto"            # resolves the best engine for your phrase

[stt]
provider = "groq-api"      # or openai-api, openrouter-stt, gemini-api, faster-whisper (local)

[tts]
provider = "gemini-flash-tts"
fallback = "grok-voice"    # cross-provider fallback is the norm everywhere

Overrides cascade from jarvis.toml to ENV (JARVIS__SECTION__KEY=…). Secrets never go in this file. API keys live in your operating system's credential manager, or in .env, and you enter them in the app.

Uninstall

One command as well. It removes the install folder, the autostart entry, and the keychain entries. Add --dry-run to preview, --yes to skip the confirmation:

# Windows (PowerShell)
& "$env:USERPROFILE\.personal-jarvis\install\uninstall.ps1"

# macOS · Linux
bash ~/.personal-jarvis/install/uninstall.sh

Both of those run the uninstaller that is already on your disk. If it is missing or refuses to start, the app can uninstall itself instead: see install/README.md.

Privacy

Your keys stay yours. They are stored in the operating system's credential manager, never in the repo, and never in a file you could commit by accident.

The always-on part is local. Wake-word listening runs entirely on your machine, and audio only goes to a cloud speech provider after you have addressed Jarvis, and only if you chose a cloud provider in the first place. Speech recognition can run fully offline with the [local-voice] extra. Brain and voice output use whichever provider you configure.

Memory is plain files. The Knowledge Wiki is Markdown on your disk, not a hosted database.

Extend it

Every pluggable part is a Python entry point. Write a class against the protocols in jarvis/core/protocols.py, register one line in pyproject.toml, reinstall. No fork, no core edits.

Three rules keep it stable: implement the protocol, stream everything (AsyncIterator, where non-streaming yields one element), and pass the contract suite (pytest tests/contract/). The deep engineering map, with anti-patterns, recurring bug classes, and phase status, lives in docs/LLM-CONTEXT.md, and is built to be pasted into an LLM chat whole.

PersonalJarvis/
├── jarvis/                  # The application
│   ├── core/                #   L6  EventBus, protocols, config + atomic writer
│   ├── orchestrator/        #   L6  State machine and turn control
│   ├── brain/               #   L4  Providers, the router, the Ack-Brain, persona
│   ├── missions/            #   L6  Worker and critic loop, worktree isolation
│   ├── agentic_ide/         #   L6  Terminal grid, call signs, prompt delivery
│   ├── workspace/           #   L6  Which coding agents exist, how they install
│   ├── clis/                #   L6  Detecting and driving external agent CLIs
│   ├── agent_chat/          #   L6  The typed chat and its coding-CLI seats
│   ├── skills/              #   L6  The Markdown skill system: load, match, run
│   ├── local_models/        #   L4  The local server, its roles, one-click setup
│   ├── costs/               #       The spend ledger and what every call cost
│   ├── speech/              #   L2  Wake → VAD → STT → TTS
│   ├── dictation/           #   L2  Cleanup, clipboard insert, polish, history
│   ├── realtime/            #   L2  Speech-to-speech providers
│   ├── memory/              #   L6  Knowledge Wiki, awareness, long-term recall
│   ├── documents/           #       Reading PDFs, office files and media
│   ├── artifacts/           #   L7  What a run produced, rendered not listed
│   ├── marketplace/         #   L7  Community plugins, skills and wallpapers
│   ├── cu/                  #   L5  Computer use: see the screen, drive it
│   ├── safety/              #   L3  The four risk tiers and the approval path
│   ├── channels/            #   L7  Telegram, Discord, and the shared brain behind them
│   ├── telephony/           #   L7  Outbound and inbound calls (optional extra)
│   ├── plugins/             #       Every pluggable backend, wired by entry point
│   ├── cli_ctl/             #       The jarvis / jarvisctl / jctl client
│   └── ui/web/              #   L7  FastAPI server + the React desktop app
├── ui/                      # Orb overlay; loaded by jarvis at runtime
├── board-backend/           # Standalone federation service (signed Board aggregates)
├── conductor/               # YAML-first agentic-workflow canvas, mounted in the app
├── wiki/                    # Seed knowledge vault, created on first run
├── install/                 # One-line installers + release verification (cosign / TUF)
├── tests/                   # Unit, integration, contract, and end-to-end suites
├── docs/                    # Architecture docs, ADRs, the philosophy, design specs
├── assets/                  # Brand art, banner, screenshots, demo recordings
├── .github/                 # CI workflows + issue / pull-request templates
├── scoop-bucket/            # Windows install manifest (Scoop)
├── homebrew-tap/            # macOS install formula (Homebrew)
└── README · LICENSE · NOTICE · CONTRIBUTING · SECURITY · TRADEMARK · CHANGELOG

Documentation

Community

Development happens in the open. The roadmap and the bug hunts land on Discord before they land anywhere else, and questions are welcome there.

Discord · @Ruben_Luetke · Instagram · GitHub

Contributing

Pull requests are welcome, and there is no bar you have to clear before opening one. CONTRIBUTING.md opens with your first PR in five lines; the checklist scales with what you touched, so a docs fix owes nothing that a new speech provider owes. One rule holds for everyone: everything written into the repo is English, because a maintainer has to be able to read every line that lands. Talk to us in any language you like on Discord.

Security issues go through SECURITY.md privately, never a public issue. Everyone here follows the Code of Conduct.

Where a human beats the agents

Most commits here are written with AI assistance, and that is exactly why some work is wide open. These are the places a person contributes something no agent can:

• Real hardware. AMD and Intel GPUs, Apple Silicon and Intel Macs, the distro your laptop actually runs, the microphone you actually own. The maintainer has one Windows box. Label area:hardware.
• Providers we have no key for. Every wake, STT, TTS, brain, and channel plugin is swappable; adding one is self-contained work that pytest tests/contract/ grades for you.
• Your language. The assistant speaks whatever you configure, and native speakers catch what a translation pass never does.
• Telling us it broke. A precise bug report on a setup we cannot reproduce is worth more than a patch.

Contributors

Thanks to everyone who has shipped something here:

The wall is generated from the commit history by scripts/ci/update_contributors.py and refreshes itself weekly. Land a pull request and your face is on it, with no form to fill in and nobody to ask. AI-assisted pull requests are welcome; what gets reviewed is the diff, not how you wrote it.

New to the repo? The issue chooser is the front door, good first issues are scoped down to a file and a test command, and setup questions get answered the same day on Discord.

License

Apache 2.0. Free to use, modify, and distribute, including commercially, with an explicit patent grant from every contributor; see LICENSE and NOTICE.

Personal Jarvis was MIT-licensed through version 1.6.0, and every one of those releases stays MIT, permanently. The switch applies from 2.0 on. docs/licensing.md says why, and what it changes for users, forks, and contributors.

Third-party names and logos belong to their owners, see TRADEMARK.md.

_{Created by Ruben Lütke · @Ruben_Luetke · © 2026 · Apache 2.0
Discord · X · Instagram}