Project Away

The bridge between you and your AI agents, controlled by your hands, face and voice.

Project Away sees your gestures, notices whether you are at the screen and paying attention, and listens for your voice. You use that to run your PC hands-free (point to move the cursor, pinch to click, talk to type), and, new in v0.1.20, to work with the AI agent you already use (Claude, ChatGPT, Cursor or any other): it can ask you a quick question you answer with a thumbs up, tell you when something is done, look at your screen or camera when you allow it, tune Project Away to how you like it, and even use your computer while you watch. Free and local-first.

Nothing leaves your machine unless you connect an agent and press Allow on your own screen (or point the optional LLM cleanup at an endpoint yourself).

User guide (no technical knowledge needed)

1. Installing

1. Download ProjectAway-…-windows-x64-setup.exe from the Releases page.
2. Windows will warn that the app is unrecognized (it isn't code-signed yet). Click More info → Run anyway, then Install.
3. The installer then downloads the speech and tracking models, about 1.6 GB. This is the slow part — read this first:
   • On a slow or unstable connection it can take hours, and the progress bar may stall at the same number for a long time. That's normal; it isn't frozen for good.
   • If it fails, stalls for ages, or the window closes: just open Project Away again. Each time it starts, it checks for missing models and carries on downloading from where it stopped — nothing already downloaded is lost. You may need to do this a few times.
   • You can also use Start menu → Project Away: re-download models to resume.
   • Leave the PC awake and plugged in, and avoid other big downloads while it runs.
4. When it's done, a hand icon appears in the system tray (bottom-right, by the clock — check the ^ arrow). Gestures and voice are on straight away.
5. Have an NVIDIA graphics card? Open Settings → Performance → Install GPU pack (about 570 MB, once). Voice typing gets much faster and more accurate.

1b. Your first five minutes (guided first run)

The first time Project Away starts, a friendly guide opens by itself. It has big buttons, a row of dots that shows how far you are, and a Skip this step button on every page: everything already works with the standard settings, so skipping is always fine.

1. Find your camera: one button checks that it turns on. If it doesn't, you get a plain explanation and a button that fixes the usual cause (open the camera privacy settings, download a missing file, try again).
2. Sit comfortably: a live picture and tips like "You're a bit far away" or "It's a bit dark" until it says Looks great.
3. Try it out (about 2 minutes): point at three circles, pinch to click, scroll, and pause with an open palm. Each one turns green as soon as Project Away sees it. Nothing on your computer is clicked while you practise.
4. Make it comfortable: two optional switches, Only respond when I'm looking at the screen and Switch to head control when my arm is tired (see section 6).
5. Eye setup (optional): follow a few dots with your eyes.

Run it again any time: tray menu → Guided first run..., or Settings → Getting started (or type away tour). People who already used Project Away before v0.1.20 just get a one-time note about it.

2. Setting up your space

• Camera: at or just above your screen, facing you, with your hand clearly visible from about 40–80 cm away.
• Light: light coming from in front of you, not a bright window behind you.
• Background: plain is best. Busy backgrounds and other people's hands in view make tracking jumpier.
• Two webcams (optional): better accuracy. Add the second one in Settings → Cameras, then press Calibrate and slowly wave an open hand where both cameras can see it for about 20 seconds.

3. Gestures: how to project away

Hold up one hand, palm toward the screen, and point.

Auto-scroll, step by step: hold up four fingers with your thumb folded across your palm (or just held against the side of your index finger) for a moment (the cursor ring, if you turned it on, goes orange). Where your hand is at that moment is the "middle": you do not have to move away from it first. Move your hand a little above it to scroll up, below it to scroll down; the further away, the faster. Hold your hand still and it keeps scrolling at that speed, so you can read a long page without moving. Moving left or right scrolls sideways. Close your hand into a fist to stop (it also stops if your hand leaves the camera for more than half a second, if you pause, or if you make another gesture). Keeping your thumb tucked in matters: with the thumb out it is the open palm, which pauses.

Claps: the microphone listens for a sharp hand clap. One clap turns gestures back on; two claps within about half a second pause them (a single clap waits that half second to make sure a second one isn't coming). Claps are ignored while you're in the middle of saying something. If the camera can see both your hands come together it is extra sure it was a clap. Turn it off, or make it pickier, in Settings → Voice & input.

Big cursor: while gestures are on, the Windows mouse pointer is made bigger so you can see it from across the room. It shrinks back to your normal size whenever gestures pause, so the size also tells you whether gestures are on. Choose the size (or 0 for off) in Settings → Voice & input → Cursor size while gestures are active. On Mac and Linux the pointer itself isn't changed; the cursor ring grows instead.

Palm-glide scroll (v0.1.14), step by step: hold one hand up flat, palm toward the screen, fingers together — that hand is your "trackpad". Touch the middle of that palm with the index fingertip of your other hand and hold it there for a moment (about a sixth of a second; the cursor ring, if you turned it on, goes cyan). Now slide the fingertip along the palm: toward the fingers scrolls up, toward the wrist scrolls down. Put your index and middle finger on the palm together and slide sideways to scroll left or right. Flick and lift, and the page keeps going for a moment and slows down, like on a phone; touch the palm again or make any other gesture to stop it. A quick tap on the palm (touch and lift, without sliding) can be given an action in Settings → Bindings → gesture:palm_tap (it does nothing until you do). While your fingertip is on the palm the cursor holds still and nothing clicks, and the flat hand never pauses gestures. Speed, the dead zone, the momentum and the rest are in Settings → Tuning → Palm scroll.

Which hand is which (hand roles, v0.1.14): Away works out by itself which hand is the pointer (moves the cursor, clicks and does every one-hand gesture) and which is the helper (the flat "trackpad" hand). The first hand that points gets the pointer (or simply the only hand in view), and it keeps it — even if the camera mixes up left and right, or the hand disappears for up to a second. Raised your flat "trackpad" hand first? That's fine: as soon as the other hand points, it becomes the pointer. Holding your phone in the other hand? Pick Settings → Voice & input → Hand roles → Right or Left to use only that hand as the pointer; Auto is the default. You can also say "use left hand", "use right hand", or "use either hand" (back to Auto).

Tips for smooth control:

• Small movements go a long way. The cursor speeds up when you move fast and slows for precise work, so slow down near small buttons.
• Pinch crisply and keep your hand still while you do — the cursor locks briefly so the click lands where you aimed.
• Rest your elbow on the desk or armrest. Holding your arm up in the air gets tiring fast.
• Touching your real mouse or keyboard pauses gestures automatically. They come back after a second or two of not touching them.
• F9 turns gesture control on or off at any time. The tray menu does the same.
• Several monitors? The cursor covers all of them. Say "next monitor", "previous monitor" or "monitor two" to jump.
• Your hand stays in view on its own (auto-centering): the camera frame quietly nudges your resting hand back toward the middle, and a quick swing back from the edge of the camera's view doesn't drag the cursor back (like lifting a mouse). Lost the cursor? Say "center" to put it in the middle of the screen you're on.

Teaching your own gestures (Settings → Gesture Studio)

Open Settings and pick the Gesture Studio tab.

1. Press Turn camera on. You see yourself like in a mirror, with your hand drawn on top. (Project Away pauses while the Studio uses the camera and carries on when you turn it off.)
2. Press Teach a new gesture and follow the 3 steps: give it a name, show it when the app says Now! (15 times, in 3 short rounds, with a short "just relax" moment after each round), then try it and choose what it should do: press a key, click, type some text, open an app or a website, or a voice action.
3. Practice mode is on by default: you see what would happen, but nothing is really pressed or opened. Untick it when you are happy.
4. Did something happen that you did not want? Press That was wrong straight away. The last 2 seconds are kept as an example of "no gesture" so it learns not to do that. Then press Learn my gestures now.
5. Where gestures work: the picture is split into squares. Pick Ignore here and click the squares where gestures should be ignored (where your coffee cup is, or where your hand rests on the keyboard). You can also draw a box or say "only above a line". The cursor still moves everywhere unless you tick Also stop the cursor there (under Show details). Areas are saved for each camera, and can differ for each app profile.
6. Show details adds charts: the last 5 seconds, the points of your hand it looks at, which practice example the hand looks most like, and what it mixed up when it checked itself.

Your practice recordings stay on this computer (only hand points, never pictures). Delete all my recordings removes them. Share my gestures... saves your gestures to a file for someone else; recordings are only added if you say so.

4. Voice

Voice is always listening by default. Just talk, and your words are typed wherever the text cursor is.

• Punctuation: say "comma", "period", "question mark", "new line", "new paragraph".
• Mistake? Say "scratch that" to remove the sentence you just said, or "no wait, make it Tuesday" to fix the last word.
• Smart cleanup (optional): Settings > Voice & input > Smart cleanup. Tick it, press Download once (about 0.5-1.1 GB, shown on the button) and your dictation gets proper punctuation, and is casual in chat apps and tidy in email. If it is ever slow, the basic cleanup is typed instead, so you never wait.
• My words: add names and terms to the My words list in the same tab (or say "add Jonie to my dictionary"). Right after a mistake you can say "correct Jonnie to Jonie". Words you fix by hand are learned too.
• Commands: "click", "double click", "right click", "scroll up", "scroll down", "go back", "go forward", "open browser", "pause gestures", "resume gestures", "stop listening", "start listening", "use left hand", "use right hand", "use either hand", "center" (cursor to the middle of this screen), "look here" (cursor to where you're looking, after gaze calibration).
• Shortcuts: Ctrl+Alt+Space pauses/resumes listening, Ctrl+Alt+M mutes the mic, Ctrl+Alt+E lets you edit selected text by voice.
• Tray icon colour: green = listening, amber = not listening, red = mic off, blue = recording, grey = paused.
• Words not being typed? If you're looking away from the webcam, the face check may ignore you. Turn Face gate off in the tray menu.
• Speech is processed on your PC and never saved or uploaded.
• Click things by name (Windows): say "click Save", "double click Photos" or "focus Search". Look at something (or point at it) and say "click that". If two things have the same name, numbers appear next to them: say the number, or "cancel".
• Snap to buttons (Windows): the pointer is gently drawn to buttons and links and holds still over them, so a shaky camera image doesn't make you miss. Both are on by default; turn them off, or turn on Smoother cursor, under Settings > Voice & input > Smart pointer.

Shortcuts: show it once, then a gesture or a word repeats it

Something you do often, like opening your mail and starting a new message? Teach it once:

1. Open Settings > Shortcuts and press the big Record a shortcut button (or say "record shortcut", or pick Record a shortcut in the tray menu).
2. Do the clicks and typing yourself, the normal way, or hands-free: pinch-clicks and dictated text count too. A small red Recording bar shows at the top; press its Stop button when you are done (or say "stop recording").
3. Give it a name, then choose what runs it: say a phrase ("open mail") or make a gesture. That's it.

To stop a shortcut while it runs, press Esc, move the mouse, or say "stop". If a shortcut can't run (the app it was made in isn't open, or another window is in the way), Settings > Shortcuts tells you why and shows a button that fixes it. Shortcuts never record what you type into a password box, and a shortcut that types text stays on your PC. Edit steps lets you remove or reorder steps, change pauses, add text, a key or a wait, and pick the speed.

5. Head and eye tracking (optional, off until you set it up)

• Head pointer: with your hands busy, turn your head to move the cursor and click with a wink, smile or eyebrow raise (turn those on in the face settings). Set hands.cursor.head_assist: pointer.
• Head + hand: head_assist: coarse makes the cursor jump to the part of the screen you turn toward; your hand does the fine aiming.
• Eye tracking ("where am I looking?"): Settings → Performance → Calibrate gaze… (or away gaze calibrate). Look at the dots on each screen until they vanish (about 2 seconds each). Afterwards Project Away knows where your screens are and roughly where on them you look: when you start moving your hand after resting it, the cursor first jumps to where you're looking, then your hand fine-tunes. Webcams are only good for "which screen and which area" (roughly a hand-width on the screen), not for small buttons.
• Optional gaze model: a learned eye-gaze network can be downloaded in Settings → Performance → Models. Its weights are for research use only (see below), so it is never installed automatically.

6. Comfort: only when you're looking, tired arms and breaks

All of these are plain switches in Settings → Getting started (the first tab). The numbers behind them are under Advanced there, and in Settings → Tuning; you never need to touch them.

• Only respond when I'm looking at the screen (off until you turn it on; also a check box in the tray menu): gestures count only while you face the screen and hold your hand up in front of you. Rest your hand on the desk, turn to talk to someone, or reach for your coffee, and nothing gets clicked. The ring at the cursor turns grey while it waits for you, and the tray says what to do ("raise your hand a little higher"). A short glance away is forgiven, so it doesn't flicker on and off. Without face tracking (or before the face model has loaded) it only checks your hand.
• Switch to head control when my arm is tired (off until you turn it on): after a long time with your arm up, or when your hand starts to sink or shake, Project Away tells you to lower your hand and steers the pointer with your head instead. Raise your hand again and it takes back over. Needs face tracking.
• Tell me when my arm looks tired (on): just the gentle note, at most once every 10 minutes.
• Remind me to take a break (on): a short note after 30 minutes of continuous gesture use. Being away from the camera for 2 minutes counts as a break.

7. If something goes wrong

• Models didn't finish downloading: reopen the app (see Installing above). It resumes.
• Cursor doesn't move: check another app isn't using the camera (Zoom, Teams, the Camera app), then restart Project Away from the tray.
• Voice is slow or lagging: install the GPU pack (Settings → Performance, NVIDIA cards only).
• Want to change a gesture or phrase? Tray → Settings… → Bindings.
• A camera is "too slow": Settings → Cameras → Test both cameras reports any camera below 10 frames per second as FAILED. Project Away has already tried every format and size on it; if NVIDIA Broadcast is installed it can use Broadcast's camera instead (and says so). Otherwise use the other camera alone.
• A camera never opens, but the Windows Camera app shows it: Project Away now also tries the Camera app's own way of reading cameras (WinRT). Settings → Cameras → Diagnose shows, for every camera, which way works and how fast. If a camera keeps failing, Project Away stops retrying it after a while and says so; fix it, then press Retry cameras (Settings → Cameras, or the tray menu).
• "The app crashed last time": the details are in crash.log in the logs folder (%LOCALAPPDATA%\ProjectAway\logs\crash.log on Windows, next to away.log). Please attach it to a bug report.

Connect your AI agent

30-second quick start

1. Open Settings → Agent Link and press Connect an AI agent.
2. Pick your AI: Claude Desktop, ChatGPT, Cursor, Claude (web or phone) or Other.
3. Choose what it may do (the safe choices are already ticked) and press Next.
   • Claude Desktop / Cursor on this PC: press Connect. Project Away adds itself to the app's settings (a backup is kept), then restart the app.
   • ChatGPT, Claude.ai or anything else: you get a private link, a QR code and a 6-digit code. Paste the link into the app's "add custom connector" box (ChatGPT: Settings → Apps & Connectors → Advanced → Developer mode → Create; Claude: Settings → Connectors → Add custom connector) or open it on your phone.
4. Your PC asks "Claude wants to connect: Allow / Deny". Answer with a click, a thumbs up, or by saying "allow". You see Connected to Claude ✓. Disconnect all ends every connection at once.

Then just talk to your agent: "What can you see in Project Away?", "The cursor is too fast, fix it", "Ask me with a thumbs up before you send the email".

While an agent uses your computer a bright border and the banner "Claude is using your computer — move your hand or mouse to take over" are on screen. Touching the mouse or keyboard, or making a gesture, pauses it instantly; it must ask again to continue. Ctrl+Alt+Esc, saying "stop", or holding up an open palm for a second stops it. It never types into password fields, never touches Windows security prompts, and asks you before clicking anything like Send, Pay, Buy, Delete, Install (with "Always allow for this agent"). Settings → Agent Link shows everything it did, with a picture of the screen for each step, and Undo last.

Answering your agent with your hands: thumbs up = yes / the first choice, thumbs down = no / the last choice, 1–4 fingers = choice 1–4, open palm = choice 5, or say the answer, or click it.

Talking to your agent with gestures/voice: say "hey agent" (it is called), "hey agent, look" (it gets a camera picture, if allowed) or "hey agent, look at my screen", and "that was wrong" when Project Away did something you did not want (agents can then suggest a fix). Bind them to any gesture in Settings → Bindings (agent:summon, agent:look, agent:screen, agent:wrong, agent:stop).

My preferences: Settings → Agent Link → My preferences is a plain list ("I hate accidental right-clicks", "slower cursor in Photoshop") that you and your agents share; agents read it and cite it when they change something. Changes run in Ask me first mode by default; Just do it and tell me lets them make small adjustments (thresholds and sliders within their normal range) right away and tell you. Every change is in a history with who and why, with Undo and Undo its changes today, and the config is snapshotted before each batch.

Technical details

Transports. One server, three ways in, the same permissions and rules for all of them:

• MCP over stdio for apps on this PC: the app runs away mcp --scopes read,diagnose (the wizard writes it). It forwards to the running Project Away app over 127.0.0.1; the first call shows the Allow prompt (proof of "same user, same PC": a key file only you can read), and the token is kept until it expires (30 days) or you revoke it.
• MCP over Streamable HTTP for remote agents: https://<random>.trycloudflare.com/mcp with Authorization: Bearer <token>, or the pairing link https://<random>.trycloudflare.com/c/<code> used directly as the connector URL (no header needed; ChatGPT and Claude.ai connectors). OAuth is not implemented in v1; the pairing link is the credential and only becomes one after you press Allow.
• REST/JSON for agents without MCP: the same tools under /v1/... (POST /v1/rpc/<tool> for any of them), server-sent events at /v1/events/stream, OpenAPI at /openapi.json. Exchange a pairing code or the 6-digit code for a token with POST /v1/pair {"pin": "123456", "name": "My agent"}.

Claude Desktop by hand (what the wizard writes into claude_desktop_config.json; Windows path shown, macOS uses /Applications/ProjectAway.app/Contents/MacOS/ProjectAway, a pip install uses away):

{"mcpServers": {"project-away": {
  "command": "C:\\Program Files\\ProjectAway\\ProjectAway.exe",
  "args": ["mcp", "--scopes", "read,diagnose,tune"]}}}

Cursor uses the same entry in ~/.cursor/mcp.json; away agent connect claude|cursor writes it from the command line, away agent snippets prints them.

Remote Streamable HTTP client (tested from a cloud sandbox through a real quick tunnel):

from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

async with streamablehttp_client("https://<random>.trycloudflare.com/c/<code>") as (r, w, _):
    async with ClientSession(r, w) as s:
        await s.initialize()  # the person sees "<your client name> wants to connect"
        print(await s.call_tool("get_context", {}))

or without MCP: python scripts/agent_link_client.py --url <link> pair --pin 123456, then ... status, stream, ask "Ready?" Yes No, look out.jpg, set hands.cursor.gain 1.8 --why "...", bundle out.zip.

Tools: get_status, get_context, poll_events, wait_for_event, get_logs, get_config, list_settings, set_config, undo_change, change_history, get_preferences/add_preference/remove_preference, run_diagnose (cameras | sweep | benchmark) + get_job, record_session, get_bundle, download_file, look, get_screen (elements=true adds the Windows UI Automation tree), ask_user, notify_user, request_control/resume_control/stop_control, move, click, double_click, right_click, drag, scroll, type_text, press_keys, open_app, open_url, undo_last_action, disconnect. Resources: away://about (briefing: what Project Away is, your permissions, live context, how to talk to the person; also sent as the MCP instructions), away://preferences, away://context, away://events and away://screen/live (both subscribable: resources/subscribe → notifications/resources/updated), away://user/last-capture. Events: presence (sat down / left), attention, summon, user.look, user.screen, wrong, misfire, detector, control.*, prompt.*, change.*.

Polite prompts. ask_user with urgency low/normal waits until you are present, facing the screen and not typing (normal waits at most half its timeout); high shows at once.

Settings agents can change go through a registry of domains (settings, zones, bindings, custom_gestures, profiles, presets, plus placeholders for features on other branches). Every value is checked with the same range table as the Tuning sliders, and the whole config must still validate before anything is saved.

CLI: away agent status|pair|start|stop|disconnect|agents|revoke ID|connect claude|cursor|snippets|prefs|changes| undo|github-token (aliases away link, away remote), away mcp, away replay FILE, away diagnose bundle.

New config keys (agent_link:): enabled (true), remote_ttl_min (60), local_ttl_days (30), pairing_ttl_s (600), control_ttl_min (30), port (0 = any), rate_per_s (10) / rate_burst (40), action_rate_per_s (4), cloudflared_download (true), cloudflared_path, screen_fps (1), screen_max_width (960), snapshot_max_width (320), record_buffer_s (30), detectors (true), detector_cooldown_s (600), fps_drop_ratio (0.5), misfire_threshold (4), autofix (ask | auto_low_risk), stop_hotkey (ctrl+alt+esc), github_issues (false), github_repo.

Recordings and replay. record_session / the 30 s ring buffer store hand landmarks and app events (never video or audio) as gzip JSON lines (.awayrec, ~25 KB for 5 s). away replay FILE re-runs them through the gesture pipeline headless and deterministically; drop one into tests/replays/ and run the tests once with AWAY_UPDATE_REPLAYS=1 to pin it as a regression test.

Auto-detectors. A camera frame-rate drop, a camera that will not open, a retry loop, a crash, or a spike of "that was wrong" / cancelled actions saves a diagnostics bundle (logs, redacted config, the last camera check, stats, the last 30 s of landmarks) to <data>/diagnostics/ and tells connected agents. Optional, off by default: agent_link.github_issues: true files a GitHub issue (stats only, never the bundle) with a token you store in the OS keyring with away agent github-token.

Security notes (honest)

• The local server listens on 127.0.0.1 only. Remote access exists only while a pairing link or a remote agent is active, through a Cloudflare quick tunnel; Cloudflare can see that traffic (TLS ends at their edge).
• A pairing link / QR / 6-digit code grants nothing by itself: the first client to use it creates a request you must Allow on the PC (permissions listed, plus the matching code). Links are single-use and expire after 10 minutes; a 6-digit code is guessable in principle, so failed attempts lock the server for a minute after 10 tries and requests are rate-limited per client.
• Tokens are 256-bit random, stored only as SHA-256 hashes, compared in constant time; remote access expires after 1 hour, local after 30 days; Disconnect all, the tray menu, or away agent disconnect revoke everything at once.
• A connector that uses the pairing link as its URL keeps that link as its credential until it expires or is revoked: treat the link like a password and do not share it.
• Requests with a browser Origin header are refused (a web page cannot drive it); every request is in <data>/logs/agent-access.log (codes and tokens never written).
• Never possible through Agent Link: running shell commands, reading or writing your files (downloads are limited to bundles, recordings and logs inside Project Away's data folder, with path-traversal and symlink checks), changing allow_shell, hotkeys, updates or the agents' own permissions.
• "Let it use my computer" is the riskiest permission. Its protections (approval every session, expiry, takeover pause, password-field and secure-desktop blocks, risky-click confirmation, blocked Win+R / Ctrl+Alt+Del, no terminals, rate limit, full action log) reduce but cannot remove risk: an allowed agent can still click and type anything you could in ordinary windows. Turn it on only for agents you trust, and watch.
• Password fields are detected with Windows UI Automation (IsPassword). Without UI Automation, typing is refused on Windows; on macOS/Linux only window titles that look like sign-in prompts are blocked.

Architecture

flowchart LR
  CAM[Webcam] --> HL[MediaPipe Hand Landmarker]
  CAM2[Optional 2nd camera] --> PAIR[Threaded capture + timestamp pairing]
  CAM --> PAIR
  PAIR --> HL
  HL --> MV[Multi-view fusion: confidence, triangulation, handover]
  MV --> NORM[Normalize + One Euro filter]
  NORM --> GSM[Gesture state machine + static/TCN models]
  GSM --> STEADY[Steady click + sigmoid curve + dwell/push]
  CAM --> FL[MediaPipe Face Landmarker]
  FL --> ASD[Active speaker + dictation gate]
  MIC[Microphone] --> VAD[Silero VAD]
  VAD --> ASR[faster-whisper / Parakeet]
  ASD --> ASR
  ASR --> CLEAN[Rule cleanup / optional LLM]
  WIFI[Wi-Fi CSI or RSSI] --> CSI[Hampel + Doppler features]
  CSI --> WCLS[Coarse presence/swipe classifier]
  STEADY --> FUS[Fusion + event bus]
  GUARD[Physical mouse/keyboard guard] --> FUS
  WCLS --> FUS
  FUS --> MOUSE[Cursor / click / scroll]
  CLEAN --> KB[Type into active app]
  FUS --> RING[Cursor ring overlay]

Quickstart

Full per-OS instructions: docs/INSTALL.md. Wi-Fi sensing: docs/wifi-setup.md. Experimental iPhone face tracking (free Live Link Face app over Wi-Fi): docs/iphone.md.

pipx install ".[all]"        # Python 3.10–3.12
away setup                   # downloads models, checks camera + mic
away demo                    # synthetic end-to-end check
away run                     # or: away tray ; away autostart on

One-click installers (Releases page) bundle every feature: tracking, dictation, tray, settings window and on-device training (CPU PyTorch). Parakeet/NeMo is the one exception (multi-GB, NVIDIA-oriented; use pipx install "project-away[all,parakeet]"). Installers are ~0.4–0.8 GB; the models (~1.6 GB, or ~0.5 GB with Whisper small) download during install on Windows and on first launch elsewhere.

• Windows: run the setup.exe. It's unsigned, so choose More info → Run anyway, then Install. Models, start-at-sign-in, a desktop shortcut and launching the tray app are all on by default.
• macOS: open the .dmg, drag to Applications, then right-click → Open. The first launch downloads the models with a notification. Allow Camera, Microphone, Accessibility and Input Monitoring.
• Linux: run the AppImage or install the .deb; the first launch downloads the models. Mouse and keyboard control needs X11; Wayland is untested.

Customizing controls

Every control is a trigger → action binding you can change without touching code: from the Settings window (tray → Settings..., or away settings), from the CLI, or in config.yaml (away config path shows where it is). A running Away reloads the file within a second of a save.

Default bindings

Triggers: the gestures above plus your own (record with away record --custom --label wave, then away calibrate), any spoken phrase (matched ignoring case and punctuation), face:brow_raise, face:mouth_open, face:wink_left|wink_right, gesture:dwell, gesture:push, and any key combo (hotkey:ctrl+alt+k). Two fixed hotkeys sit outside the bindings: app.toggle_hotkey (default F9, gestures on/off) and voice.edit_hotkey (default Ctrl+Alt+E, edit the selected text by voice).

Actions: none · mouse:click|double_click|right_click|middle_click|scroll_up|scroll_down (mouse:move, mouse:drag, mouse:scroll, mouse:autoscroll only on point / pinch_hold / two_finger_scroll / four_finger_scroll) · key:ctrl+tab · type:any text · launch:firefox or launch:https://… (no shell) · shell:command (off unless bindings.allow_shell: true) · dictation:toggle|start|stop · mic:toggle|on|off · voice:command_mode|dictation_mode|toggle_mode · pause:toggle|on|off · cursor:next_monitor|prev_monitor|primary_monitor|monitor N (jump to the centre of a monitor) · cursor:center (centre of the current monitor + re-anchor; bind it to a gesture or hotkey, e.g. away bindings set hotkey:ctrl+alt+c cursor:center or away bindings set gesture:palm_tap cursor:center) · cursor:gaze (jump to where you look) · hands:auto|right|left|any (which hand is the pointer until the app restarts).

Multiple monitors

The cursor moves across every monitor (v0.1.6; before that it was stuck on the primary). away monitors lists them, numbered 1..N left to right (then top to bottom), with where the list came from: Windows EnumDisplayMonitors, macOS Quartz/AppKit, Linux screeninfo or xrandr --listmonitors, falling back to the primary monitor only (with the reason) when none works. The same list is in away hardware and the selftest.

• Layouts: side by side, stacked, a monitor left of / above the primary (negative coordinates), mixed sizes. The cursor is always clamped onto a real monitor, never into the gap beside a smaller screen.
• Speed: relative-mode speed is measured against the primary monitor, so slow, precise moves feel exactly as on one screen. Only fast moves get extra reach on a bigger desktop: (desktop width / primary width) ** hands.cursor.monitor_reach. With three equal monitors and the default monitor_reach: 0.5 a quick flick goes ~1.7x further, so crossing all three takes a few flicks rather than three times the arm sweep. 0 = no extra reach, 1 = scales fully with the desktop.
• Jump: say "next monitor", "previous monitor" or "monitor two" (Whisper writing "monitor 2" works too), or bind cursor:next_monitor / cursor:monitor 3 to any gesture or hotkey. A held drag stays held, so "next monitor" while dragging carries a window across.
• Which monitors: hands.cursor.monitors: all (default) | primary | [1, 3].
• Hot-plug: monitors are re-detected every hands.cursor.monitor_refresh_s (3 s; 0 = only at start), no restart needed.
• Windows DPI: the app makes itself per-monitor DPI aware at startup (before pynput/tkinter), so monitor rectangles and cursor positions are both physical pixels on mixed-scaling setups. AWAY_DPI_AWARE=0 turns that off.

hands:
  cursor:
    monitors: all          # all | primary | [1, 3]
    monitor_reach: 0.5     # extra reach for fast moves on a wide desktop
    monitor_refresh_s: 3   # re-detect plugged/unplugged monitors

Per-app profiles

default holds the full map. Any other profile names the apps (process names, case-insensitive, .exe optional) it applies to and overrides only the triggers it lists; everything else falls through to default. active_profile: auto switches profile with the focused app (Linux needs xdotool); away profile use chrome pins one. A profile can also carry its own hand and face tuning: (Settings → Tuning → Save as profile..., see The Tuning tab).

CLI

away bindings list [--profile chrome] [--json]
away bindings set gesture:swipe_left key:alt+left
away bindings set "voice:new tab" key:ctrl+t --profile chrome
away bindings reset [TRIGGER] [--profile NAME | --all]
away bindings export my-controls.yaml        # or "-" for stdout
away bindings import my-controls.yaml [--replace]   # default merges
away profile list | use NAME|auto | new NAME --apps chrome chrome.exe [--copy-from default] | remove NAME
away voice show | mode always_on|push_to_talk|toggle | wake-word away|off | intent dictation|command | sensitivity 0.6
away settings                                # GUI editor (tkinter); prints CLI help when there is no display
away update --check                          # current vs latest release (exit 0 up to date, 10 update available)
away update [--yes]                          # download, verify SHA-256, install (see "Updates")
away hardware [--json]                       # detected CPU/RAM/GPU + the performance profile and why
away gpu install | status | remove           # NVIDIA GPU pack (cuBLAS 12) for Whisper on CUDA
away iphone listen | compare [--seconds N]   # experimental iPhone face source (docs/iphone.md)
away gaze calibrate | test [--seconds N] | clear [--all] | status    # eye tracking + the screen map
away gaze model status | install l2cs|mobilegaze [--accept-license] | remove NAME   # optional gaze network
away tour [--text]                           # guided first run (window); --text: camera check + tips in the terminal

Invalid configs fail with every problem listed and its YAML path, e.g. bindings.profiles.default.gestures.pinchh: unknown gesture 'pinchh' (did you mean 'pinch'?).

Tunables

Every Settings tab scrolls (scroll bar or mouse wheel over the tab); the Save / Reset / Import / Export / Close bar and the status line below it stay visible at any window size (v0.1.10).

The Tuning tab

Settings → Tuning groups every slider by area in collapsible sections: Cursor, Clicks, Scroll, Gestures, Face, Head & eyes, Voice, Other (a section with nothing in it is hidden). Each slider has a plain-language name, a grey line saying what raising or lowering it does, its current value with units (0.32 palm, 800 ms, 2.2×) and a ↺ button that puts just that slider back to its default (greyed out while it is at the default). Each section has Reset section.

• Presets: Precise / Balanced / Fast. Balanced is the shipped defaults. Precise is a slower, steadier pointer with a tighter pinch and more smoothing; Fast is a quicker pointer, less smoothing and an easier pinch. The line under the buttons says which preset your values match ("custom" once you move a slider).

• Sensitivity (master slider): scales cursor speed, acceleration, the hold-still zone and the pinch thresholds together around their defaults (the middle = defaults; right = up to 2× speed, half the dead zone, a 25 % looser pinch; left = the opposite). pinch_off moves with pinch_on, so the click hysteresis is kept. It shows "mixed" once one of those sliders was moved on its own.

• Undo reverts the last change (a stack of 100 steps; one slider drag or one preset is one step).

• Save as profile... keeps your hand and face tuning (whatever differs from the defaults) in a per-app profile, new or existing, under tuning: in that profile. While that app is focused the app uses those values; everywhere else it uses the global ones. Voice, camera and app-wide settings stay global. Click Save to write it to the config file. YAML:

  bindings:
    profiles:
      chrome:
        apps: [chrome]
        tuning: {hands.cursor.gain: 3.0, hands.gestures.pinch_on: 0.36}
  

• Live view: why did a gesture (not) fire? Start it and the tab borrows the camera from the running app (like the Test gesture tab; it is handed back when you stop or close the window), draws your hand skeleton (pointer in blue, the helper hand in grey) and shows live gauges against their thresholds (the red mark): pinch distance vs pinch_on, thumb fold vs autoscroll_thumb_fold, index finger out (fingers count as out above 1.1; how many of four are up) and fingertip to palm vs palm_scroll.depth_threshold (needs both hands). A gauge turns green when its condition is met, so you can see which part of a gesture was missing. Moving a slider moves the red mark at once.

• Auto-tune a gesture (Pinch click, Auto-scroll thumb fold, Palm scroll depth): press Start, do the gesture 5 times when asked (2 s each, with a countdown), then do "something else" 5 times (for the pinch: point and move without pinching; for the thumb: an open palm with the thumb out; for palm scroll: hover the fingertip just above the palm). For each try the most gesture-like moment is taken (10th percentile, so one bad frame does not count) and the threshold that best separates the two sets is suggested (best balanced accuracy, the middle of the widest gap). You see before → after and whether the two sets overlapped; Apply sets it (undoable; for the pinch, pinch_off moves by the same amount) and saves the 10 tries as labelled clips in your recordings folder (away record format with a shared session_id, labels pinch_click/move, four_finger_scroll/palm_clutch, palm_scroll/move) so away calibrate can use them for your personal classifier. Note that four_finger_scroll and palm_scroll become extra classes of the temporal model the next time you calibrate.

New tunables added to away.config.tunable_ranges appear in the tab automatically: an entry in away.app.tunables.TUNABLE_META gives the section, title, help line and unit; without one the section comes from the setting's name (head, gaze, eye, auto_center → Head & eyes; hands.cursor. → Cursor; voice. → Voice...; anything else → Other).

Sliders in the Settings window, or in YAML: hands.cursor.gain (sensitivity), dead_zone, acceleration, One Euro min_cutoff / beta / d_cutoff; hands.gestures.pinch_on / pinch_off (pinch threshold), scroll_gain (scroll speed), swipe_min_dx + swipe_max_time (swipe velocity = distance / time), double_pinch_window, drag_hold, clutch_hold; voice.vad_sensitivity, end_silence_ms, max_utterance_s. Each is range-checked.

v0.1.9 additions:

• Big cursor (hands.cursor.big_cursor_size, 0 = off, 1-5 = 1.5x/2x/3x/4x/6x of 32 px, default 2): on Windows the accessibility cursor size (HKCU\Control Panel\Cursors\CursorBaseSize) is raised while gesture control is active and put back on pause, auto-pause, clutch, F9, double clap, disable and quit (at most one change per 300 ms so auto-pause flapping does not thrash it). Your original size is saved to big_cursor.json in the data folder first; if Away crashes while the cursor is big, the next start restores it.
• Hand choice (hands.max_hands: 2): MediaPipe looks for two hands. Two-hand detection is only run every few frames until a second hand actually appears, so one hand costs about the same CPU as before. max_hands: 1 turns it off (and palm-glide scroll with it). v0.1.9's hands.prefer is replaced by hands.roles (below) in v0.1.14; an old prefer: right|left becomes roles.mode: right|left and prefer: any becomes roles.mode: auto automatically.
• Auto-scroll (hands.gestures.autoscroll_*): autoscroll_enabled, autoscroll_hold (0.2 s before it starts), autoscroll_thumb_fold (0.95, v0.1.16: the thumb counts as folded below this thumb-fold ratio and stays folded until 0.1 above it; it was a fixed 0.75, which a real hand rarely reached, so the pose read as the open palm and paused gestures instead; raise it if auto-scroll is still hard to start, lower it if an open palm starts it), autoscroll_dead_zone (0.15 palm lengths, was 0.25), autoscroll_gain (20 notches/s per palm length beyond the dead zone, was 12 on a power curve; speed is now proportional to the distance from the neutral point), autoscroll_max_speed (30 notches/s), autoscroll_stop_hold (fist, 0.2 s), autoscroll_horizontal. The neutral point is where the hand is while the pose is held (the average over the hold), so scrolling starts from there. Four fingers with the thumb folded always win over the open-palm pause: a hand whose thumb is neither clearly folded nor clearly out never toggles pause. Offsets are measured in palm lengths, so it feels the same near or far from the camera. autoscroll_thumb_fold is in Settings → Tuning.
• Claps (voice.clap_*): clap_enabled, clap_sensitivity (0.5), clap_double_window (0.6 s), clap_require_hands (only when the camera sees both hands close together). Detection uses the voice microphone: a sharp, short (under ~50 ms), broadband transient that is far louder than the moment before it. Two hands close together on camera lower the bar; with no camera frames it is stricter. Ignored while you are mid-utterance and while the mic is muted. The Settings Test gesture tab shows claps as well as gestures.

v0.1.14 additions:

• Hand roles (hands.roles): mode: auto (default; the first hand that points, or the only hand in view, is the pointer) | right | left (that real hand, MediaPipe's label corrected for the mirrored image) | any (the first hand seen, whatever its pose; config only). release_s: 1.0: the pointer keeps its role while it is out of view this long (it is released sooner when it was last seen at the image border). Hands are told apart frame to frame by palm-centre position and size (a minimum-cost assignment), not by MediaPipe's handedness label, which flips; the label is only a weak, smoothed prior (and what right/left read, with hysteresis). A lone hand that has only held a flat palm hands the role to the first hand that points. Only the pointer drives the cursor, clicks and gestures; the helper feeds palm-glide scroll. A helper that becomes the pointer (the other hand left) cannot pause with its open palm until it has shown another pose; when the role moves to the other hand, a running drag ends and the cursor re-anchors without a jump.
• Palm-glide scroll (hands.palm_scroll): enabled: true, engage_s: 0.15 (fingertip on the palm this long before it scrolls), depth_threshold: 0.6 (fingertip at most this far off the palm plane, in palm lengths; 0 = no depth check), margin: 0.25 (palm outline padding, palm lengths), gain: 12 (wheel notches per palm length of glide), dead_zone: 0.05 (palm lengths; also the most a tap may move), invert: false, horizontal: true (index + middle = sideways), momentum: true, friction: 4.0 (1/s; speed × e^(−friction·t) after the lift). It engages when the helper is flat (four fingers out), the pointer's index fingertip falls inside the palm outline (wrist, thumb base, the four knuckles) and is near the palm plane. The glide is measured in the palm's own frame (wrist → middle knuckle = up), so a rotated or tilted palm scrolls the same way. With one camera, depth comes from MediaPipe's relative z and the hands' apparent sizes (both palms are assumed to be the same size); with two calibrated cameras the triangulated 3D points of both hands are used. If the helper's landmarks drop out or jump while the fingertip covers it, the last good palm is used for 0.3 s. The fingertip is smoothed with the One Euro filter. A tap (touch and lift within 0.25 s) fires gesture:palm_tap. While engaged, the pointer's cursor movement, clicks, four-finger auto-scroll and open-palm pause are suppressed.

Pointer accuracy:

• Steady click (hands.cursor.click_freeze: true): the cursor freezes while the pinch is closing (ratio below pinch_on * (1 + freeze_margin)), and the click lands where the cursor was click_rewind_ms (100) before closing began, because closing thumb and index drags the index tip. Drags unfreeze immediately.
• Curve (hands.cursor.curve): sigmoid (default) gives a slow plateau (sigmoid_low) for precise work and a fast plateau (sigmoid_high) with sigmoid_mid / sigmoid_slope in between. power is the original |d|^acceleration curve. speed_up/down/left/right scale each direction, for limited reach or an angled camera. The sigmoid idea comes from Google Project Gameface (Apache-2.0) and is re-implemented.
• Dwell click (hands.dwell): rest the cursor within radius_px for time_s and gesture:dwell fires.
• Face clicks: face.wink_trigger / wink_threshold / wink_open_max / wink_hold, plus brow_threshold, mouth_threshold and the hold times. Thresholds are per user, so tune them in Settings. face.source: webcam|iphone|auto (default webcam) can take the blendshapes from an iPhone's TrueDepth camera instead; see docs/iphone.md (experimental).
• Cursor ring (cursor_overlay: true or away run --cursor-overlay): a small ring by the pointer that fills as the pinch closes, turns solid on pinch, blue while dragging, purple while dwelling, orange with an arrow while auto-scrolling and grey when paused.
• Auto-pause (app.auto_pause: true): touching the real mouse or keyboard pauses gesture output until it has been idle for app.auto_pause_idle_ms. Away's own synthetic input is ignored.

Dictation extras: the vocabulary file is passed to faster-whisper as initial_prompt and as hotwords (when the installed version supports it; 1.2.1 does). The prompt holds only the vocabulary (Vocabulary: a, b, c.), never the focused app: Whisper repeats its prompt on short or noisy audio, and up to v0.1.5 that typed "Dictating into ." into the app. Since v0.1.6 every transcript (push-to-talk, always-on and the edit hotkey) is filtered before it is typed: prompt echoes ("Vocabulary: ...", "Dictating into ...") and bare lists of vocabulary terms are dropped or stripped, segments with a high no_speech_prob and a low avg_logprob are discarded, and stock silence phrases ("Thank you.", "Thanks for watching!", "Please subscribe") are dropped when the utterance is under 2 s. A phrase you bound as a voice command is never filtered. Dropped transcripts show in the log as dropped: <reason>. voice.numbers_as_digits turns "twenty five" into 25 from numbers_min_value (10) up. voice.edit_hotkey works like this: select text, press the hotkey, say an instruction ("uppercase", "replace cat with dog", "clean it up", "make it a bullet list", or anything else when an LLM endpoint is configured), then press it again.

Voice input modes

voice.mode is always_on (default), push_to_talk (hold voice.hold_key) or toggle (press voice.hotkey).

Always-on listens continuously, cuts speech into utterances with an energy VAD (end-pointing after end_silence_ms of silence) and decides per utterance:

• Wake word on (voice.wake_word_enabled: true, wake_word: away): only utterances starting with the wake word count ("away, scroll down"). A bare "away" opens a wake_window for the next utterance.
• Wake word off: text is typed only when the face gate agrees that you are at the screen and your mouth is moving (active-speaker detection), so TV or background talk is ignored. A visible face whose mouth moves counts even when the head is turned (off-axis webcam), and a short gap in face tracking while the CPU is busy is tolerated. Since v0.1.12 "mouth moving" is judged against your own resting mouth level (a webcam at arm's length and off-angle sees small movements), and when your enrolled face is at the screen and facing the camera in at least 60% of an utterance's frames it is typed even with weak mouth motion, unless another face is clearly the speaker. Each voice.utterance log line carries the gate's evidence (face frames, frontal fraction, speaking fraction, mouth score and threshold). After 10 rejections in a row the tray suggests turning the gate off (tray menu Face gate, or Settings). With the face gate off (face.gate_dictation: false), speech is typed without a wake word. With no camera, voice.ambient_fallback: commands_only (default) runs bound phrases but types nothing.
• If speech recognition cannot keep up (e.g. a big model on the CPU), a short backlog is kept, the oldest utterance is dropped, and you get one notification saying so.
• voice.intent: command never types, it only runs voice commands. Say "command mode" / "dictation mode" to switch.
• "stop listening" / "start listening", the mute hotkey and the tray's Mic on item control it. The tray icon shows the state (green active, red mic off, amber not listening, blue recording, grey paused).
• Audio is kept in memory only and dropped after transcription. Nothing is written to disk.

app.always_on: true (default) makes gestures and voice active as soon as the app starts; false starts paused and muted until you resume from the tray. app.start_minimized: false also opens Settings at launch.

Example config.yaml

app:
  always_on: true
  start_minimized: true
voice:
  mode: always_on            # always_on | push_to_talk | toggle
  wake_word_enabled: false
  wake_word: away
  intent: dictation          # dictation | command
  vad_sensitivity: 0.5
  mute_hotkey: <ctrl>+<alt>+m
hands:
  cursor: {curve: sigmoid, gain: 2.2, dead_zone: 0.004, min_cutoff: 1.0, beta: 0.02, click_freeze: true}
  gestures: {pinch_on: 0.32, pinch_off: 0.45, scroll_gain: 40.0, swipe_min_dx: 0.25, swipe_max_time: 0.45}
bindings:
  active_profile: auto
  allow_shell: false
  custom_gestures: [wave]
  profiles:
    default:
      gestures:
        swipe_left: key:alt+left
        swipe_right: key:alt+right
        wave: launch:https://calendar.google.com
      voice:
        "screenshot": key:cmd+shift+s
      hotkeys:
        ctrl+alt+d: type:Best regards,
    chrome:
      apps: [chrome, chrome.exe, Google Chrome]
      gestures:
        two_finger_scroll: key:ctrl+tab
      voice:
        "new tab": key:ctrl+t
        "close tab": key:ctrl+w
    code:
      apps: [code, Code.exe]
      gestures:
        open_palm: key:ctrl+shift+p   # open palm no longer pauses in VS Code

Multi-camera

A laptop webcam and an external camera can work together. A hand that one camera sees edge-on, or that is occluded, is usually clear in the other.

away cameras list                 # device names (V4L2 / Windows DirectShow / macOS) without opening them
away cameras use "HD User Facing" # single-camera mode: pick the camera by name
away cameras add 2                # index, /dev/video2, or part of the name ("brio"); saved by name
away cameras test                 # can all configured cameras stream at the same time? (~10 s)
away cameras test --save          # ...and save the combination that works (e.g. secondary at 320x240 MJPG)
away cameras diagnose [NAME]      # every camera x every backend: native formats, does each mode open, fps
away calibrate-cameras            # wave an open hand through both views for ~20 s
away calibrate-cameras --method checkerboard --checkerboard 9x6 --square-mm 25   # accurate option
away calibrate-cameras --print-board board.png --checkerboard 9x6                # printable board

The same is available in Settings → Cameras (device pickers, roles, calibrate button, live per-camera confidence). Calibrate runs calibrate-cameras as a separate process (in the installed Windows app that is ProjectAway.exe calibrate-cameras, not python -m away), borrows the cameras from the running app, shows progress in the status line and ends with "calibrated, reprojection error X px" or the error. Its full output is saved as calibration.log next to calibration-status.json in the data folder.

Two cameras at the same time (v0.1.15). The app, Settings' Live confidence and Test both cameras, away calibrate-cameras and away cameras test now all open the cameras the same way: the primary first, then each other camera once the primary has delivered its first frame (at most 8 s later), one after another on Windows, with the same fallbacks (MJPG/YUY2/NV12, the device default, and for a secondary 320x240 and 320x240 @ 15 fps, then the NVIDIA Broadcast camera if hands.virtual_fallback). Live confidence shows each camera's state in the Confidence column (opening…, the confidence, no frames, or a short error) and the mode that worked in the status line. Calibrate and Live confidence never run together: Calibrate stops the live view and waits until its cameras are released, and keeps them borrowed from the app across the hand-over. If the secondary still fails, calibration retries it once at 320x240 MJPG after the primary is open. A camera that only works in a fallback mode is remembered by name (camera-modes.json in the data folder) and opened in that mode right away at the next start. Test both cameras / away cameras test streams every camera for ~10 s and reports backend, mode, measured fps, frames and errors; if they do not all stream at the configured size it tries the secondary at 320x240 MJPG @ 15 and offers to save what works. While the app is still probing a slow camera it now says it holds the cameras and lets go as soon as Settings or calibration asks.

Slow cameras, honest verdicts and the NVIDIA Broadcast fallback (v0.1.16). On the reported laptop the built-in camera "worked" at 1.0 fps (10 frames in 10 s, the same at 320x240, so not USB bandwidth) and the test still said both cameras stream together. Now:

• hands.min_camera_fps (default 10): a camera measured below it is FAILED (too slow: X fps), never "ok", in away cameras test / Test both cameras, in Live confidence and in away.log (each camera's measured rate is logged once it has streamed 5 s). Calibration measures every camera first and refuses with a clear message when one is too slow (instead of "no secondary camera could be calibrated").
• Every mode tried is measured for ~1.5 s (not just one frame), and the fastest wins: a mode at or above the minimum is used at once; otherwise every FOURCC (MJPG, YUY2, NV12, the driver's format) is tried with CAP_PROP_FPS set, on DirectShow and then on Media Foundation (at its own index), at the configured size, 640x480 and 320x240 (at most ~60 s, once). Each attempt is one line in away.log: backend, the FOURCC, size and fps the driver reports back, exposure / auto-exposure (logged only; auto-exposure is never changed) and the measured fps. The best mode is remembered per camera name in camera-modes.json with its measured rate; a camera that stayed too slow is checked in that one mode at the next start instead of being searched again.
• If a camera stays below the minimum in every mode and a free Camera (NVIDIA Broadcast) is listed, it is used instead when hands.virtual_fallback: true (default), but only if Broadcast itself reaches the minimum. The slow camera is closed first and not kept open (Broadcast usually shows that same camera). The test, Live confidence and the log say "using Camera (NVIDIA Broadcast) instead (HD User Facing is too slow ...)"; away cameras test --save / Save writes Broadcast's camera into hands.cameras. With virtual_fallback: false the test only offers it.

WinRT MediaCapture backend, diagnose, retries that back off (v0.1.19). On the reported Acer the built-in camera ("HD User Facing") never opens through OpenCV's DirectShow or Media Foundation (a fast "did not open", even when it is the only camera), while the Windows Camera app shows it smoothly. The Camera app uses WinRT MediaCapture through the Windows Camera Frame Server, so Project Away can now too:

• WinRT backend (away.multicam.winrt_capture, Windows only, pywinrt packages: the [winrt] extra, part of [all] and bundled in the Windows installer). The camera is found by its name among WinRT's source groups; MediaCapture is opened with exclusive control (so the format can be chosen), else shared read-only (the current format, when another app controls it). The native format closest to the configured one is used: 30 fps first, then the smallest size that covers the configured one, up to 1280x720, downscaled to fit the configured size (aspect ratio kept). Frames come from a MediaFrameReader as BGRA8 and are converted to BGR on the capture's own thread / asyncio loop; the rest of the app reads them like an OpenCV camera.
• Order (hands.camera_backend: auto, the default): DirectShow → Media Foundation → WinRT (when both failed or stayed below hands.min_camera_fps) → the slow-camera sweep → the NVIDIA Broadcast fallback. A camera WinRT opened is remembered in camera-modes.json as backend WinRT with its native format and opened that way first next time. dshow, msmf or winrt use that backend alone (virtual cameras such as NVIDIA Broadcast always use DirectShow). Settings → Cameras → Capture backend sets it. away.log names the format, sharing mode and output size WinRT used.
• HD sweep: the slow-camera sweep now also tries 1280x720 (MJPG and the driver's format) last; HD frames are downscaled to the configured size.
• Honest MJPG: the FOURCC is set before the size, read back, set once more after the size when the driver reports another one, and a camera that ignores it is logged as such ("asked for MJPG but the driver reports YUY2") instead of being described as MJPG.
• away cameras diagnose [NAME] [--seconds 1] and Settings → Cameras → Diagnose: for each camera and each backend (DirectShow, Media Foundation, WinRT; elsewhere OpenCV's default) the native formats where the backend can list them (WinRT can; OpenCV cannot list DirectShow / Media Foundation formats, so common modes are tried: the configured size in MJPG, YUY2, NV12 and the driver's format, 1280x720 and 320x240), whether each opens, and the measured fps or the error. The cameras are borrowed from the running app first. The test and diagnose output in Settings is now a scrollable, word-wrapped text box (long lines were cut off).
• Retries back off and stop. v0.1.17 re-probed a dead camera every 45 s forever (306 times in a day). Now the failed cameras are re-probed after hands.camera_retry_s (45 s), then 90, 180, 360, 720 s, capped at hands.camera_retry_max_s (900 s = 15 min), and after hands.camera_retry_limit (8) failures in a row the app stops, says so once (tray notification, a red line in Settings → Cameras) and waits for Retry cameras (tray menu, or Settings → Cameras). camera_retry_limit: 0 never stops. The same back-off applies when no camera works at all. A re-probe never reopens the camera that is tracking (it is an error instead, if the failed camera's name now resolves to it), and a camera already measured too slow in this session is checked in its remembered mode only, not swept again.
• Settings → Cameras: the Device picker shows the current camera (it was blank until a row was selected), the only camera is selected for editing and its Role defaults to primary.

Crash capture (v0.1.16). The tray app, the Settings window, the CLI and the installed exe all write crash.log in the logs folder next to away.log (%LOCALAPPDATA%\ProjectAway\logs\crash.log on Windows; data_dir()/logs elsewhere): a timestamped line when each process starts, a dump of every thread on a fatal error (Python faulthandler), and any uncaught exception (main thread or worker thread, also in away.log). At the next start a crash found there is logged and shown in the tray tooltip and a notification, and in the Settings status line: "the app crashed last time; details in crash.log". The camera hand-over was hardened at the same time: the app now acknowledges a hand-over only once none of its cameras is open or still opening (an open in progress gives up), the camera test waits for that confirmation before it opens anything, reading the hand-over files retries through Windows' brief sharing violations (one could make Settings think the app held no camera), and no worker thread touches Tk any more (Settings and the cursor overlay marshal every update to the Tk thread). In YAML:

hands:
  cameras:                       # empty (default) = single-camera mode, unchanged
    - {device: 0, role: primary, width: 640, height: 480, fps: 30, hfov_deg: 70}
    - {device: "Logitech BRIO", role: secondary, width: 1280, height: 720, fps: 30, hfov_deg: 78}
  multicam:
    fusion: weighted             # weighted | best_view
    pair_tolerance_ms: 40
    dropout_ms: 300
    handover_decay_s: 1.0
    use_3d_pinch: true
    push_distance_m: 0.06
    push_time_s: 0.35

Cameras are chosen by name (v0.1.11). Settings and away cameras use/add save the camera's name (hands.camera_index: "HD User Facing", or device: in hands.cameras) plus its current index as a hint (camera_hint / index_hint). The name is resolved to the camera's current index at every open, including retries, so a hot-plugged camera can never take its place; two cameras with the same name are told apart by their Windows DevicePath or NAME #2. Plain integer indices still work. On Windows the list is DirectShow's own (ICreateDevEnum, the order cv2.CAP_DSHOW uses; WMI/PnP only as a logged fallback) and every camera, single or multi, is opened with DirectShow (Media Foundation only as a logged fallback). Virtual cameras (NVIDIA Broadcast, OBS Virtual Camera, Iriun, DroidCam, Camo, Snap Camera, ManyCam, XSplit...) are marked (virtual) and never picked automatically: an old integer index that now lands on one (NVIDIA Broadcast is often DirectShow camera 0) is moved to the first real camera, with a line in away.log. Choose one by name to use it anyway. The camera list (source, index, name, virtual) is logged at start-up and when it changes.

Pixel format and busy cameras (v0.1.12). Every camera is opened in MJPG (hands.camera_fourcc for a single camera, fourcc: per entry in hands.cameras; "" = let the driver choose, which is usually uncompressed YUY2) and counts as open only once a frame arrives; a camera that sends nothing in MJPG is retried in its default format. A second camera that will not stream while another one is open is retried at 320x240, then 320x240 at 15 fps (two USB 2.0 webcams can exceed the bus); the mode that worked is logged and reused for the rest of the session. Errors now say whether a camera did not open or opened but sent no frames. Media Foundation (v0.1.13): when DirectShow cannot use a camera, it is opened with Media Foundation at the index Media Foundation's own list gives it (matched by device path, else by name); v0.1.12 and older reused the DirectShow index, but Media Foundation numbers cameras differently (it does not list DirectShow-only virtual cameras such as NVIDIA Broadcast), so the fallback hit the wrong camera. DirectShow is tried in MJPG, YUY2 and NV12 at the configured size, without a format, and with nothing forced; every attempt is logged. NVIDIA Broadcast: when a physical camera still cannot be opened and Camera (NVIDIA Broadcast) is listed, the log and a notification say " could not be opened directly; NVIDIA Broadcast's camera shows the same picture." With hands.virtual_fallback: true (default) Project Away then opens the one free Broadcast camera instead and logs it. Choosing Camera (NVIDIA Broadcast) by name works for the single camera and in a multi-camera set.

How it works:

1. Capture. Each camera runs in its own thread and stamps frames on arrival. The primary camera sets the pace. Each secondary contributes the frame nearest in time, if it falls within pair_tolerance_ms. A camera that stays silent for dropout_ms is treated as dropped and the others carry on.
2. Per-view confidence. Each view is scored on the MediaPipe handedness score, the share of landmarks inside the image, hand size in pixels, and sharpness (Laplacian variance of the hand crop, which drops with motion blur).
3. Fusion.
   • Uncalibrated: best_view takes the most confident view. weighted averages the views by confidence after mapping secondaries into the primary image with an alignment learned while both cameras see the hand.
   • Calibrated, with both cameras seeing the hand: the 21 landmarks are triangulated (cv2.triangulatePoints). Reprojecting them into the primary view drives the cursor, and the 3D points give true depth.
4. Depth uses.
   • gesture:push: the hand moves push_distance_m toward the screen within push_time_s. The primary camera is assumed to sit at the screen. The gesture is bindable and clicks by default.
   • 3D pinch: the pinch ratio comes from 3D distances, so a thumb and index that only overlap from one angle no longer click.
5. Occlusion handover. When one view loses the hand, tracking continues from the other. The jump between estimates is absorbed by an offset that fades over handover_decay_s, so the One Euro filter sees a continuous signal and the cursor does not jump.

Calibration. The hand-motion path pairs MediaPipe landmarks across views and runs cv2.findFundamentalMat (RANSAC, then a least-squares refit on the inliers) → E = K2ᵀ F K1 → cv2.recoverPose. Intrinsics are approximated from hfov_deg, and the translation is scaled so the median palm (wrist to middle knuckle) is palm_length_m (9 cm). Depth is therefore approximately metric; real palms vary by about 10–15%. The checkerboard path runs calibrateCamera per camera, then stereoCalibrate with the real square size. Either way the extrinsics go to <data dir>/camera_extrinsics.json and the reprojection error is printed.

Each view runs its own MediaPipe instance in sequence, so two cameras cost about twice the CPU of one.

A camera that cannot be opened or sends no frames is skipped with a notification and tracking continues on the one that works, so a wrong or busy secondary camera never stops hand tracking.

Two webcams on Windows (v0.1.8): at start-up each camera gets up to 12 s to send its first frame (Windows can take several seconds to open a USB webcam), and on Windows the cameras open one after another. A camera only counts as failed when it reports an error or stays silent past that deadline, and the notification names just that camera and why ("open error: ..." or "no frames within 12 s"). After falling back to one camera, the missing one is re-checked in the background every 45 s (backing off to 10 min if it keeps failing) without interrupting tracking; when it works again fusion switches back on and you get one "cameras back" notification. If no camera works at all, nothing claims to be tracking: the app says so once and keeps retrying. The Cameras tab's device list now always shows the cameras the app is holding and the configured ones, even when Windows' device query fails.

Sharing the camera with Settings (v0.1.6): webcams are exclusive on Windows, so while the app tracks your hand, Settings' Test gesture, Live confidence and Calibrate (and away calibrate-cameras) ask the running app to hand the cameras over: it pauses tracking, releases them, and takes them back when the test ends or the window closes (a small heartbeat file in the data folder; a crashed Settings window releases its claim within 5 s). The Cameras tab lists the default camera in single-camera mode and marks the one the app is using. If a camera still cannot be opened, you are told it is in use by Project Away or another app instead of a bare "permission denied".

Auto-centering, head tracking and eye tracking (v0.1.19)

All of it reuses what already runs: the hand landmarks and the Face Landmarker (whose facial transformation matrix and iris points are now read too). No new model is needed; the optional gaze network is an opt-in download. Everything is pure logic with unit tests (tests/test_autocenter.py, test_headpose.py, test_gaze.py, test_gaze_models.py, test_headgaze.py).

Auto-centering (hands.cursor.autocenter.*, on by default):

• Soft recenter (relative mode): strokes away from the target get up to soft_strength (12 %) more gain at the frame edge, strokes back the same less, so over many out-and-back strokes the resting hand drifts toward the middle of the camera frame. The cursor never jumps.
• Comfort zone: an EMA (comfort_tau_s) of where the hand rests (slower than rest_speed). Absolute mode centres its active region there, applied only when the hand re-anchors so a resting hand never drifts the cursor; target: comfort makes it the soft-recenter target in relative mode too.
• Edge re-anchor: in the edge_margin band of the frame, a swing back faster than edge_return_speed is a "lift": the cursor stays put and re-anchors when the hand slows down or is back in the inner frame.
• Center: voice "center", or cursor:center on any gesture/hotkey.

Head tracking (hands.cursor.head_assist: off|coarse|pointer, hands.cursor.head.*, off by default): yaw/pitch/roll from the facial transformation matrix (One Euro filtered; the sign is aligned with the on-screen nose movement so a mirrored camera cannot invert it). pointer: head-turn deltas move the cursor while no hand points (Gameface-style; click with the existing wink/smile/brow/mouth triggers). coarse: the head picks one of coarse_cols x coarse_rows zones per monitor (hysteresis coarse_hysteresis, dwell coarse_dwell_s) and the cursor jumps there; the hand refines. monitor_switch: true (needs the screen map): a head turn held switch_dwell_s and switch_margin degrees closer to another monitor moves the cursor there; glances do not. The speech gate also uses the head pose: a face turned toward the screens (the calibrated pose range, or the user's usual pose within gate_tolerance degrees) counts as facing them even when an off-axis webcam sees it turned.

Eye tracking and the screen map (gaze.*; nothing acts until away gaze calibrate ran for the current camera and monitor layout): features are the iris centres (Face Landmarker points 468-477) relative to the eye corners, eyelid opening, head yaw/pitch/roll and face position/size, plus the optional network's yaw/pitch. Calibration shows points_per_monitor (9) dots on every selected monitor and fits a ridge regression with quadratic terms (ridge_alpha) to virtual-desktop pixels, stored per camera name in gaze-calibration.json in the config folder, with its leave-one-dot-out error. The screen map is the calibration: it knows which monitor and roughly where. Expect about 2-4 degrees with a webcam (a few hundred pixels at arm's length), so it is for "which screen / which area", never for small targets.

• Look-to-jump (MAGIC pointing, look_to_jump): when the hand starts moving after resting jump_idle_s (0.6 s) and the gaze point is more than jump_min_px away, the cursor warps there first, then the hand refines. jump_across_monitors_only: true limits it to looking at another monitor.
• Drift correction (drift_correction): every click nudges the model's offset by drift_rate toward the click (you look where you click), bounded by drift_max_px and ignored beyond drift_max_err_px; the offset is saved with the calibration when the app exits.
• The app reloads the calibration file by itself (Settings runs away gaze calibrate in its own process); a changed monitor layout disables it until you recalibrate.

Optional learned gaze model (gaze.model: auto|iris|l2cs|mobilegaze; auto = l2cs when downloaded and onnxruntime offers a GPU provider, else mobilegaze when downloaded, else iris): an L2CS-Net-style network (90-bin yaw/pitch classifier, 448x448 face crop) from github.com/yakhyo/gaze-estimation, run with onnxruntime on the Face Landmarker's face box at most gaze.model_rate_hz (10) times a second, CUDA first when available. Its yaw/pitch become two extra features of the same calibration, so recalibrate after installing one; the running app picks a newly downloaded model up without a restart.

License notice: the code is MIT, but both networks were trained on the Gaze360 dataset, whose license is research use only and excludes models trained on it from commercial applications. They are therefore an explicit opt-in (away gaze model install NAME --accept-license, or the confirm dialog in Settings → Performance → Models), downloaded from the author's GitHub release with a pinned SHA-256 (resumable, through the same downloader as Whisper), and never bundled in the installers. Setup only mentions them.

hands:
  cursor:
    autocenter: {enabled: true, soft_recenter: true, soft_strength: 0.12, target: frame,
                 comfort_zone: true, comfort_tau_s: 20, edge_reanchor: true, edge_margin: 0.08}
    head_assist: "off"          # off | coarse | pointer
    head: {monitor_switch: false, switch_dwell_s: 0.6, switch_margin: 4, gate_tolerance: 25}
gaze:
  enabled: true                 # still needs `away gaze calibrate`
  model: auto                   # auto | iris | l2cs | mobilegaze
  look_to_jump: true
  jump_idle_s: 0.6
  jump_min_px: 250
  jump_across_monitors_only: false
  drift_correction: true

Smart pointer (v0.1.20)

Three helpers that make a webcam pointer feel like a real mouse. Each is one plain switch in Settings > Voice & input > Smart pointer; the numbers behind them are sliders in Settings > Tuning > Advanced. Nothing needs setting up.

hands:
  cursor:
    smoothing: one_euro            # one_euro | predictive  ("Smoother cursor")
    magnet:                        # "Snap to buttons"
      enabled: true
      radius_px: 40                # how far from a target the pull starts (5..200)
      strength: 0.35               # how far toward the target the pointer is pulled (0..1)
      sticky: 0.5                  # how much slower small moves are over a target (0..0.95)
    predictive:                    # only with smoothing: predictive
      lookahead_ms: 33             # 0..100
      process_noise: 5.0           # 0.1..50: higher follows quick turns sooner
      measurement_noise: 0.004     # 0.0005..0.02: expected landmark jitter (hand units)
      speed_gate: 0.3              # 0.01..2: hand speed (frame widths/s) for the full look-ahead
voice:
  click_by_name: true              # "Click things by saying their name"

How it stays out of the cursor's way: a background thread refreshes the element list 4 times a second with one UI Automation call (FindAllBuildCache with a cache request for name, type and rectangle) and publishes an immutable snapshot with a spatial grid; the cursor loop only reads the latest snapshot (about 0.08 ms per frame with 2,000 elements, measured on the build machine). Elements older than 2 s are dropped. A voice command refreshes once more on the voice thread before matching. UI Automation comes from comtypes (MIT), part of the Windows installer and of pip install "project-away[pointer]" (also in [all]). Without it, Settings says so in plain words with an Install button.

Predictive smoothing, measured with the included synthetic benchmark (python -m away.pointer.bench: minimum-jerk reaches with rests, 1 frame of latency, 0.002 landmark jitter, 5 seeds; numbers in hand units, i.e. fractions of the camera frame):

The trade-off is visible: the predictive filter is far closer to where your hand is, at the cost of a small overshoot when a fast move stops abruptly. That is why it ships off until it has been tried on real webcams.

Attention gate, arm fatigue and the guided first run (v0.1.20)

New code lives in away/comfort/ (gate, fatigue, hub, words, Settings tab) and away/app/first_run*.py; the shared files only gained small hooks (Pipeline.comfort, two config fields, a tray menu pair, one Settings tab call, away tour).

Attention gate (attention.*, AttentionGate). Each hand frame is checked for: facing (the head pose from the face landmarker, compared with the calibrated screen pose box or the learned usual pose, within facing_tolerance degrees), gaze on a screen (only with require_gaze and a gaze calibration), hand raised (wrist at least min_hand_height above the picture bottom, palm at least min_hand_size), held for onset_s. Once engaged the hand may sink 0.05 lower and the head turn 5° further (hysteresis), and any failing check is forgiven for grace_s. A frame that does not count is fed to the gesture state machine as no hand, so a drag ends cleanly and the head pointer can take over. gate_voice also drops spoken commands while you look away (dictation is still the face gate's job). The state (off | idle | arming | engaged | grace, a hint, the checks) is Pipeline.attention_state(), published as attention.state on change, shown in the tray title and as a grey / yellow ring tint (attention.indicator).

Arm fatigue (comfort.*, FatigueMonitor). Score = time with the wrist above raised_height without a rest_s rest ÷ raised_limit_s, + ½ × hand sinking (5 s average vs. the first 10 s, full at height_drop),

• ½ × tremor (RMS second difference of the palm centre while held fairly still, vs. the first 10 s, full at jitter_ratio). Tired at score ≥ 1 and at least min_raised_s up: with no other sign after 4 min, with one strong sign after 2 min, with both after 30 s. fatigue_switch then sets hands.cursor.head_assist: pointer on the running app only (never written to the file) and restores your own value once the hand has rested and been back up for return_s. Break reminders: break_every_min of continuous use; break_reset_s away resets it. These thresholds are a reasoned starting point, not a validated fatigue model.

Guided first run (away tour, FirstRun, TutorialCoach). Steps: welcome, camera (probe_camera, with the full away cameras diagnose report a click away), framing (face size/centre, brightness, hand in the picture), a pluggable zones step, the tutorial (the real gesture state machine on a borrowed camera; tasks are ticked from its events, nothing is executed), the comfort switches, optional gaze calibration (away gaze calibrate), done. Progress is saved in first_run.json in the config folder. Other features add steps with away.app.first_run.register_step(Step(...), after="framing"); the module away.studio.first_run is imported when present, so Gesture Studio can register zones (replacing the placeholder, which is only shown when away.studio is installed) and a custom_gesture offer after gaze. Errors go through away.comfort.ui.friendly_error: a plain title, a sentence, and one fix button (retry, camera_privacy, download_models, install_help, pause_app).

attention:
  enabled: false          # "Only respond when I'm looking at the screen"
  require_facing: true
  require_gaze: false     # only used with a gaze calibration
  min_hand_height: 0.25   # 0 = picture bottom, 1 = top
  min_hand_size: 0.03
  onset_s: 0.25
  grace_s: 0.8
  facing_tolerance: 25.0  # degrees
  gate_voice: false
  indicator: true
comfort:
  fatigue_switch: false   # "Switch to head control when my arm is tired"
  fatigue_tips: true
  break_reminders: true
  raised_height: 0.30
  raised_limit_s: 240.0
  min_raised_s: 30.0
  height_drop: 0.08
  jitter_ratio: 1.8
  rest_s: 5.0
  return_s: 1.0
  tip_cooldown_min: 10.0
  break_every_min: 30.0
  break_reset_s: 120.0

Smart cleanup and My words

v0.1.20. Everything runs on your PC; nothing is uploaded.

Smart cleanup (voice.cleanup). After Whisper, every dictation goes through three steps:

1. Rules (always on unless mode: off): fillers ("um", "uh"), stutters ("the the"), spoken edits ("scratch that"; "no wait, make it Tuesday" replaces the last weekday, month or time; "the total is 50 no wait 60"), spoken punctuation, numbers as digits, a question mark for sentences that start like a question.
2. The focused app decides the tone. The process name and window title are mapped to chat (Slack, Discord, Teams, WhatsApp...), email (Outlook, Thunderbird, a Gmail tab...), docs (Word, Notion, Google Docs...), code editors (VS Code, JetBrains, Visual Studio, vim...) and terminals. Chat is casual (a one-line message loses its period), email and documents are formal, code editors and terminals get the text verbatim (no casing, no period, and the model is never used there). When the text before the cursor can be read (Windows UI Automation, read-only, no keystrokes), a dictation that continues a sentence is not capitalised.
3. Optional local model (mode: llm, the Smart cleanup switch): a small instruct model polishes the rule output. It only gets timeout_ms (default 1500 ms); slower, failing or missing engines mean the rule output is typed. A guard throws the model's answer away when it adds words you did not say (beyond a few function words and your dictionary), drops much of the text, changes it too much or adds a preamble, so the model cannot answer questions in your text or invent content.

The old voice.llm_cleanup: true still works and means mode: llm with the openai backend.

Models (Settings > Voice & input > Smart cleanup > Download, or Settings > Performance > Models). Downloaded from Hugging Face pinned to a commit, SHA-256 checked, resumable, never bundled:

Both licenses were checked on the model cards and in the GGUF repositories' LICENSE files. Qwen2.5-3B is deliberately not offered: it uses the Qwen Research license (non-commercial).

Engine. In the installed Windows and Linux apps (v0.1.20) the Download button fetches the engine together with the model: the official prebuilt CPU wheel of llama-cpp-python 0.3.36 for your computer, pinned by size and SHA-256 in away/voice/engine.py (Windows x64 7.8 MB, Linux x86-64 27.1 MB, Linux ARM64 24.9 MB, from the llama-cpp-python GitHub release). Only its llama_cpp package is unpacked, into <data dir>/engines/, and it is first loaded in a separate process (away cleanup-engine check), so a broken native library can never crash the app; if that check fails the engine is removed again and Settings explains why (on Windows the usual cause is a missing Microsoft Visual C++ Redistributable). The wheels talk to llama.cpp through ctypes (they work with any Python version) and ship no msvcp140.dll. The small pure-Python packages it imports (diskcache, jinja2) are bundled in the installers. away cleanup-engine status|install|check|remove does the same from a terminal. No ready-made engine is offered for Macs (the release's Apple Silicon wheel is damaged as published: its dylibs fail their CRC check, which the CI caught) or Windows on ARM: use Ollama (ollama pull qwen2.5:1.5b, then Advanced > Engine > Ollama), which is also the way to use a GPU today. From pip, pip install "project-away[cleanup]" installs llama-cpp-python itself (a pip copy always wins over a downloaded one); for an NVIDIA GPU install a CUDA wheel, e.g. pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124.

Measured latency (scripts/bench_cleanup.py, 8 sample sentences, CPU-only llama-cpp-python 0.3.36 on 2 shared ARM Neoverse cores, the build sandbox, which other jobs were loading at the time): light model median 1.4 s (0.8-3.0 s) on a lightly loaded run and 2.4 s (1.2-5.6 s) at load average 7.6, model text accepted 8 of 8 times (7 of 8 before "stand-up"/"standup" were treated as one word); standard model median 3.6 s (2.1-10.4 s), 8 of 8. On such a CPU the default 1500 ms budget means many sentences fall back to the rule output. Not measured on a GPU (expected to be far faster on an RTX 2060, but unverified); run the script on your PC to see your numbers.

My words (voice.dictionary). A word list stored locally as JSON. Words go to Whisper as vocabulary (initial_prompt and hotwords) and are re-cased after it ("jonie" -> "Jonie", except names that are also everyday words such as Will or May); fixes (Jonnie -> Jonie) are replaced after Whisper. Ways to add them: the My words list (Add / Remove / Import / Export as .txt or .json), the voice commands "add X to my dictionary", "remove X from my dictionary", "correct X to Y" (only when X was dictated in the last few minutes, so prose like "correct the plan to match the budget" is still typed) and "spell X as J O N I E", and your own corrections: on Windows the focused field is re-read for watch_seconds after a dictation, and a small word substitution you make in that span is learned after it was seen learn_after times.

Gesture Studio (v0.1.20)

Settings → Gesture Studio replaces the old "Test gesture" tab (its "built-in gesture" line still shows what the state machine saw and what it is bound to). It borrows the camera from the running app through the same camera lease as the Tuning tab's live view (away.app.camera_share), so the camera is never opened twice.

• Zones (away/studio/zones.py): one map applies to every gesture: a grid (default 6 x 4, draggable boundaries) of works here / ignore here cells, free-form boxes on top (later boxes win), and an optional activation line (gestures count only above it). Coordinates are normalised in the mirrored image (what the preview shows); landmarks from a camera with mirror: false are flipped before the test. The point tested is the palm centre (zones.anchor). A hysteresis band (zones.hysteresis, 0.03 image widths) stops a hand on a boundary from flickering. The pipeline enforces it at runtime (Pipeline.on_hand): events that start something (click, double click, drag start, scroll, auto-scroll start, open palm, swipes, push, dwell, palm tap, custom gestures, Studio gestures) are dropped in ignored areas and published as gesture.zoned; ongoing and ending events (moves, drag end, auto-scroll, hand lost) always pass, so a drag never sticks. The cursor is only stopped with zones.block_cursor: true (default off: a cursor that freezes at a zone edge feels like a dead mouse, and keeping clicks out of an area is what zones are for). Maps are saved per camera (the primary camera's name) and per bindings profile, falling back camera+profile → camera+default → any camera+profile → any camera+default.
• What the recogniser sees (away/studio/engine.py): per-class bars including No gesture, the top candidate (shown as Not sure below the gesture's threshold), the per-gesture state (idle → arming → firing → cooldown; recognized = seen but switched off or in an ignored area), a 5-second timeline with fired actions, an occlusion importance map (each landmark replaced by its training mean, the drop in the top class's probability; 21 batched forward passes, at most 3 times a second), the nearest practice example ("looks most like practice example #7 of Wave") and a practice-mode (dry-run) switch that suppresses actions (on by default). Watch: Built-in gestures shows the bundled temporal model's classes instead (its idle = No gesture).
• Custom gestures from a blank model (away/studio/trainer.py): a gesture is a static pose or a motion, one or two hands. Gestures are trained in groups by (kind, hands), each with a nothing class: static poses on static_features (the static pose model's input; two hands add the second hand + its offset), motions on sequence_features windows resampled to 24 frames (the temporal TCN's input). With the [train] extra, poses use the StaticPoseMLP shape and motions the GestureTCN (exported to ONNX), trained with away.hands.train._fit; without torch (the installed app) a small numpy MLP of the same input is trained, so teaching works out of the box. Training takes seconds. The held-out score is group-disjoint by session_id (PR #6): every practice round is a session, the wizard records 3 rounds of 5 plus a relax round after each, and a group k-fold (up to 5 folds) never trains and checks on the same round. With fewer than 3 rounds there is no score and the result says so. Warnings: fewer than 3 rounds, fewer than 8 examples, two gestures that look alike (class means closer than half their spread), and pairs mixed up 30 %+ of the time. The confusion chart is under Show details. Improve the built-in gestures records examples of a built-in gesture and runs the existing away.hands.train.calibrate fine-tune (needs [train]).
• Negatives: the automatic relax round after each practice round, plus That was wrong (the last 2 s).
• Binding: Studio gestures are ordinary custom gestures (bindings.custom_gestures, gesture:<name>), bound through the existing bindings with plain choices (key, mouse, text, app, URL, dictation/voice/mic/pause; shell only with bindings.allow_shell: true). Optional follow-up steps (then:, an automation chain of actions and wait:<s> pauses) run after the bound action, off the camera thread, only when the action ran (not while paused). Per gesture: on/off, how sure it must be, hold time (hold-to-arm) and rest time (cooldown).
• Packs (away/studio/pack.py): a zip with manifest.json (format away-gesture-pack v1, gestures, their settings and actions) + the trained model; practice recordings only when the user ticks it (never by default). Import rejects unsafe paths, foreign or newer packs and packs over 200 MB, and skips shell actions unless shell is allowed. An imported pack's model replaces the current one (teach again to combine).
• Privacy: recordings live in <data folder>/studio/clips (hand landmarks only, never images) and never leave the computer unless exported; Delete all my recordings removes them.
• Plain words: every visible string lives in away/studio/words.py and a test checks it for jargon ("confidence", "model", "takes", "blocked", ...). Problems show a friendly message with a fix button (open camera settings, try again, record a relax round, practise more, show me how).

New config keys:

zones:
  enabled: true        # false keeps the maps but never enforces them
  block_cursor: false  # true: the cursor also stops in ignored areas (default: only activations are ignored)
  hysteresis: 0.03     # image widths the hand must pass a boundary by before its side changes (0..0.2)
  anchor: palm         # palm | index_tip | wrist: the point of the hand that is tested
  maps: {}             # camera name or "*" -> profile name -> {grid: {cols, rows, xs, ys, cells}, rects, line}
studio:
  gestures:            # per Studio gesture
    wave: {enabled: true, threshold: 0.8, hold_s: 0.0, cooldown_s: 1.0, then: ["wait:0.3", "key:enter"]}

Hardware auto-tuning and GPU

away hardware shows the detected CPU, RAM, disk and GPU (NVIDIA via NVML / nvidia-smi, others via WMI or lspci, Apple Silicon) and the profile picked from them: Whisper model, CPU or CUDA, compute type, threads, beam size, camera resolution/fps and face-tracking frame skipping, each with the reason. Explicit settings (performance.mode: auto|cpu|gpu, voice.device, voice.asr_model, performance.*) win.

NVIDIA GPUs: the one-click installers ship CPU-only; setup (or Settings > Performance, or away gpu install) downloads the GPU pack, the cuBLAS 12 libraries from NVIDIA's official PyPI wheel (~570 MB, SHA-256 verified). That is the only CUDA library CTranslate2 4.8 loads. pip installs: pip install nvidia-cublas-cu12==12.8.4.1. If CUDA fails (at startup or later), dictation falls back to the CPU with the CPU-sized model from the profile (not large-v3-turbo, which cannot keep up on most CPUs) and says why; the tray offers Install GPU pack (one notification, then a menu item). A voice.asr_model: large-v3-turbo written by v0.1.4's setup is migrated to auto (configs now carry config_version: 2; a model you pick from now on is kept). Details: docs/INSTALL.md.

Updates

The installed app checks GitHub Releases when the tray starts and every 24 hours. When a new version is out you get a notification and an Update to vX.Y.Z tray item; clicking it downloads the installer for your system, checks its SHA-256 (from the GitHub API digest, or the release's SHA256SUMS.txt; an unverifiable file is refused), installs it and restarts the app. Windows runs the Inno Setup installer silently; macOS swaps the .app from the .dmg after the app quits; an AppImage is replaced in place; a .deb is installed with pkexec apt install only after you confirm. pip/pipx installs are never touched: run pipx upgrade project-away.

updates:
  check: true          # false = never contact GitHub
  auto_install: false  # true = install silently when gestures are paused or idle (never for .deb)
  channel: stable      # beta = also offer pre-releases

The only network request is an anonymous GET to the public GitHub Releases API; nothing about you is sent. Same options under Settings → Voice & input; from a terminal: away update --check / away update --yes. The tray also has Check for updates, which always tells you the result; a check that fails (offline at sign-in) is retried after 30 minutes instead of the next day.

Windows updates (v0.1.10): the installer is downloaded to %LOCALAPPDATA%\ProjectAway\updates, verified, and started by a plain batch file next to it (away-update.cmd, run hidden by cmd.exe; no PowerShell). It logs every step to %LOCALAPPDATA%\ProjectAway\logs\update-helper.log, waits until Project Away (and the Settings window) has closed, runs the installer silently and records its exit code. Before quitting, the app checks that the helper is really running (its first log line, or the process still alive after 3 seconds); if not, it starts the installer directly instead. The app quits by itself (forced after a few seconds if something hangs), the installer writes logs\update-install.log, and the app restarts. On the next start you get "Project Away was updated to vX", "The update to vX is still installing" (with the helper's last step), or "The update to vX did not install" with the reason, the helper's last log line and the remedy: right-click the tray icon → Run update installer, which opens the kept installer normally (visible, like a double-click). Every step is logged to away.log.

Upgrading from v0.1.9 or older: those copies still use the old PowerShell helper, which did not run the installer on some Windows 11 PCs. If the tray update to v0.1.10 does not take, download ProjectAway-0.1.10-windows-x64-setup.exe from the releases page and run it once by hand (quit Project Away first). Updates after that use the new helper.

The v0.1.10 → v0.1.11 update is the first one installed by the batch helper on a real PC. If it does not take, use Run update installer in the tray, or run ProjectAway-0.1.11-windows-x64-setup.exe by hand.

v0.1.12 helper: the installer's exit code decides the outcome, not just the version number: a non-zero code (e.g. Inno Setup 5, aborted on a locked file) is reported as an incomplete update with the Run update installer item, even when the version already matches. The helper force-closes ProjectAway.exe and its child processes, retries a failed installer once after 5 s, clears the PyInstaller variables it inherited, and restarts Project Away itself after the installer finishes unless the installer's own relaunch already did (it watches for the app for up to 10 s first, so there is never a second copy).

Shortcuts: teach by demonstration (v0.1.20)

A shortcut (macro in code and config) is a recorded list of steps that any gesture, voice phrase or hotkey runs through the normal bindings with the action macro:<id> (for example voice: {open mail: macro:open_mail}). macro_control:record | stop_recording | toggle_recording | stop drive recording and stop a running shortcut; the default voice phrases are "record shortcut" / "record macro", "stop recording" and "stop".

Recording (Settings > Shortcuts, the tray, voice, or away shortcut record) captures your own mouse and keyboard with pynput, with timing:

• clicks (single/double, any button, ctrl/shift/alt/cmd held), drags with their path, wheel scrolls, typed text (one step per run of characters) and key presses (enter, ctrl+c, a lone Windows key);
• each click is stored at its screen position and relative to the window it landed in, plus the app (process), the window title and, on Windows, the UI Automation name of the control under the pointer (e.g. "Send");
• Project Away's own input is left out: every synthetic event Away sends (gesture clicks, voice commands, replays) is marked, and on Windows every event the OS flags as injected is dropped too; clicks on Away's own windows (Settings, the Recording bar) and the tray click that stops a recording are dropped;
• every keystroke is skipped while a password box has focus (UI Automation IsPassword, or a classic Edit control with ES_PASSWORD). Where that can't be checked (macOS, Linux, or Windows without UI Automation) typed characters are not recorded at all unless macros.record_text_without_password_check: true; keys like enter, tab and ctrl+c still are;
• pauses longer than macros.max_delay_s are shortened.

A recording started from voice or the tray is saved as "Shortcut N"; name it and pick a gesture or phrase in Settings.

Replay runs on a worker thread while gesture output is paused:

• a click is aimed at the recorded control found again by name (Windows UI Automation), else at the same spot relative to the app's window (so moving the window is fine), else at the screen position (macOS/Linux);
• it is refused when the app is not open, another app covers the spot, the window got too small, or (for text and keys) another app has focus, unless the shortcut (Edit steps > "Allow it to click and type in any app") or macros.allow_other_app allows it; text is never typed into a password box;
• stop any time: macros.abort_hotkey (Esc), real mouse movement (macros.abort_on_mouse_move), voice "stop", tray "Stop shortcut". Keys and buttons held by a step are always released;
• speed per shortcut: 1x, 2x or instant (waits you add yourself always wait their full time).

Storage and privacy. One JSON file per shortcut in <data dir>/macros (local, never the roaming config folder). A shortcut that types text is marked sensitive: owner-only file permissions on macOS/Linux, "(private)" in the list, and export refuses it unless you confirm ("Export anyway" / --include-private). The outcome of the last replay is kept in _last_result.json there, so Settings can show what went wrong with a fix button.

away shortcut list                                  # alias: away macro
away shortcut record [--name "Open mail"] [--seconds 10]
away shortcut bind open_mail --say "open mail" | --gesture palm_tap [--replace]
away shortcut run open_mail [--speed 1x|2x|instant]
away shortcut show | rename | delete | unbind ID
away shortcut export FILE [ID ...] [--include-private]
away shortcut import FILE [--overwrite]

Config (macros: in config.yaml, all optional):

Verified vs untested

Changes in v0.1.20

v0.1.20 turns Project Away into the bridge between you and your AI agents, and makes everyday hands-free use easier for people who are not technical. Everything below is new in this release; the sections above have the details, and "Verified vs untested" says honestly what has and has not been tried on real hardware.

Agent Link: connect any AI agent. Claude, ChatGPT, Cursor or anything else can now talk to Project Away, through MCP (on this PC or over a private link) or a plain REST API. Connecting is a 3-step wizard in Settings → Agent Link, and nothing connects until you press Allow on your own screen (with a click, a thumbs up or by saying "allow"). You choose what each agent may do in plain words: see your gestures, see your camera or screen only when you allow it, change your settings, or use your computer. Your agent can ask you quick questions that you answer with a thumbs up or down, a finger count or your voice, and tell you when something is done. When you let it use your computer, a bright border shows it is working; touching the mouse or keyboard, or making a gesture, pauses it at once, and Ctrl+Alt+Esc, saying "stop" or an open palm stops it. It never types into password boxes and asks before anything like Send, Pay or Delete. Agents can also tune Project Away to how you like it (a slower cursor, fewer accidental right-clicks): they ask first by default, every change is listed with who made it and why, and Undo puts it back. Agents can also manage your custom gestures and shortcuts and see when your arm looks tired. Disconnect all ends every connection at once. See "Connect your AI agent".

Gesture Studio: teach your own gestures. Settings → Gesture Studio replaces the old "Test gesture" tab. Show a new gesture 15 times, then choose what it does (a key, a click, some text, an app, a website or a voice action). Practice mode lets you try it safely first, That was wrong teaches it what not to react to, and you can mark parts of the camera picture where gestures are ignored (your coffee cup, your keyboard). The guided first run now includes it. Recordings are hand points only, never pictures, and stay on your PC.

Shortcuts: show it once, repeat it with a word or a gesture. Press Record a shortcut, do the clicks and typing yourself (pinch-clicks and dictated text count too, so it works hands-free), then give it a phrase or a gesture. Esc, a mouse move or "stop" halts a running shortcut, password boxes are never recorded, and Edit steps lets you tidy it up.

Smart pointer (Windows). The pointer is gently drawn to buttons and links and holds still over them, so a shaky webcam picture doesn't make you miss. Say "click Save" or look at something and say "click that"; if two things share a name, numbers appear and you say the number. An optional Smoother cursor predicts your hand's movement (off by default until it has been tried with real webcams).

Attention & comfort. A friendly guided first run (find your camera, sit comfortably, a 2-minute practice where nothing is really clicked) opens on the first start and from Settings → Getting started, now the first tab. Optional switches: Only respond when I'm looking at the screen (resting your hand or turning away clicks nothing), Switch to head control when my arm is tired, gentle tired-arm notes and break reminders. The Settings tabs are reordered for new users: Getting started, Bindings, Gesture Studio, Shortcuts, Voice & input, Agent Link, Cameras, Tuning, Performance.

Smart cleanup and My words. Dictation now drops "um"s and repeated words, understands "no wait, make it Tuesday", and adjusts its tone to the app: casual in chat, tidy in email and documents, word for word in code editors and terminals. The optional Smart cleanup switch adds a small local AI model (Qwen2.5, Apache-2.0) for proper punctuation; in the Windows and Linux apps one Download button fetches the model and a pinned, SHA-256 checked CPU engine (llama-cpp-python), tested in a separate process before it is used (on a Mac, use Ollama). If the model is ever slow the basic cleanup is typed instead, so you never wait. For a GPU, use Ollama. My words is your own word list for names and terms ("add Jonie to my dictionary", "correct Jonnie to Jonie"); fixes you make by hand are learned too. Everything runs on your PC.

Changes in v0.1.19

Cameras: WinRT MediaCapture backend, away cameras diagnose, retries that back off. Built-in cameras that OpenCV's DirectShow and Media Foundation cannot open (the reported Acer "HD User Facing") are tried through WinRT MediaCapture, the API the Windows Camera app uses (hands.camera_backend: auto|dshow|msmf|winrt; pywinrt packages in the [winrt] extra, part of [all] and bundled in the Windows installer). away cameras diagnose (and Diagnose in Settings → Cameras) lists every camera x backend with its native formats and the fps each mode really delivers. A failing camera is retried with a back-off and stops after hands.camera_retry_limit failures with one notice; Retry cameras in the tray and in Settings probes again at once. MJPG is reported honestly (read back after setting it). See "Multi-camera" above. The Windows installer drops pywinrt's bundled msvcp140.dll (an old 14.29 copy that made onnxruntime crash when it loaded first) and the backend preloads the system VC++ copy before importing pywinrt.

Auto-centering, head tracking, eye tracking and the screen map, plus an opt-in learned gaze model and a Models list in Settings → Performance. See "Auto-centering, head tracking and eye tracking (v0.1.19)" above. New voice commands: "center", "look here". New CLI: away gaze calibrate|test|clear|status, away gaze model install|status|remove.

Settings → Tuning redesign. Sliders grouped in collapsible sections (Cursor, Clicks, Scroll, Gestures, Face, Head & eyes, Voice, Other) with plain-language names, help lines, units and per-slider reset; Precise / Balanced / Fast presets, a master Sensitivity slider, Undo, Save as profile... (per-app hand and face tuning under tuning: in a bindings profile, applied live when that app is focused), a live view with gauges against the thresholds, and auto-tune for the pinch, auto-scroll thumb fold and palm-scroll depth. See "The Tuning tab" above. The 28 auto-centering, head and gaze sliders (hands.cursor.autocenter.*, hands.cursor.head.*, gaze.*) sit in Head & eyes; presets and the master Sensitivity slider do not touch them. Per-app profiles may tune the auto-centering and head sliders (they are hand-cursor settings); gaze.* stays global, like voice and cameras.

Changes in v0.1.18

Experimental iPhone face source. face.source: webcam|iphone|auto can take ARKit blendshapes from the free Live Link Face app over Wi-Fi (UDP) instead of the webcam, with away iphone listen|compare to check the link and an automatic webcam fallback when the phone goes quiet. Unit-tested with synthetic packets and a UDP loopback; untested with a real iPhone. See docs/iphone.md. The repository now ships an MIT LICENSE file matching the package metadata.

Changes in v0.1.17

Honest temporal gesture evaluation. Training the temporal gesture model now keeps separate train, validation and test splits, grouped by session_id/subject_id so frames from one recording never land on both sides. With fewer than 3 recording groups no recorded holdout score is reported (a warning explains why). The learned gesture model stays opt-in after calibrate: away calibrate prints the metrics but no longer turns on hands.gestures.use_model; review them, test on real gestures, then enable it yourself.

Roadmap

1. Real-hardware validation on Windows/macOS/Linux.
2. Personal calibration data → fine-tune the temporal model on real recordings.
3. Face recognition for the enrolled-user gate (currently largest face).
4. TalkNet/Light-ASD active-speaker backend.
5. Live AX210 CSI capture and a learned swipe/push-pull classifier.
6. Phone-as-second-camera / IMU fusion.
7. Real-hardware tuning of multi-camera fusion; parallel per-camera landmark inference.

Development

pip install -e ".[dev]"
pytest -q && ruff check . && ruff format --check . && mypy src
python scripts/mutation.py && python scripts/mutation.py --summary

MIT licensed; see LICENSE.