Project Away

Hands-free PC control, free and local. Webcam hand gestures drive the cursor, push-to-talk voice typing handles text, face cues gate who is dictating, and optional Wi-Fi CSI adds coarse motion confidence.

Nothing leaves your machine unless you point the optional LLM cleanup at an endpoint yourself.

User guide (no technical knowledge needed)

1. Installing

1. Download ProjectAway-…-windows-x64-setup.exe from the Releases page.
2. Windows will warn that the app is unrecognized (it isn't code-signed yet). Click More info → Run anyway, then Install.
3. The installer then downloads the speech and tracking models, about 1.6 GB. This is the slow part — read this first:
   • On a slow or unstable connection it can take hours, and the progress bar may stall at the same number for a long time. That's normal; it isn't frozen for good.
   • If it fails, stalls for ages, or the window closes: just open Project Away again. Each time it starts, it checks for missing models and carries on downloading from where it stopped — nothing already downloaded is lost. You may need to do this a few times.
   • You can also use Start menu → Project Away: re-download models to resume.
   • Leave the PC awake and plugged in, and avoid other big downloads while it runs.
4. When it's done, a hand icon appears in the system tray (bottom-right, by the clock — check the ^ arrow). Gestures and voice are on straight away.
5. Have an NVIDIA graphics card? Open Settings → Performance → Install GPU pack (about 570 MB, once). Voice typing gets much faster and more accurate.

2. Setting up your space

• Camera: at or just above your screen, facing you, with your hand clearly visible from about 40–80 cm away.
• Light: light coming from in front of you, not a bright window behind you.
• Background: plain is best. Busy backgrounds and other people's hands in view make tracking jumpier.
• Two webcams (optional): better accuracy. Add the second one in Settings → Cameras, then press Calibrate and slowly wave an open hand where both cameras can see it for about 20 seconds.

3. Gestures: how to project away

Hold up one hand, palm toward the screen, and point.

Auto-scroll, step by step: hold up four fingers with your thumb folded across your palm for a moment (the cursor ring, if you turned it on, goes orange). Where your hand is at that moment is the "middle". Move your hand a little above it to scroll up, below it to scroll down; the further away, the faster. Hold your hand still and it keeps scrolling at that speed, so you can read a long page without moving. Moving left or right scrolls sideways. Close your hand into a fist to stop (it also stops if your hand leaves the camera for more than half a second, if you pause, or if you make another gesture). Keeping your thumb tucked in matters: with the thumb out it is the open palm, which pauses.

Claps: the microphone listens for a sharp hand clap. One clap turns gestures back on; two claps within about half a second pause them (a single clap waits that half second to make sure a second one isn't coming). Claps are ignored while you're in the middle of saying something. If the camera can see both your hands come together it is extra sure it was a clap. Turn it off, or make it pickier, in Settings → Voice & input.

Big cursor: while gestures are on, the Windows mouse pointer is made bigger so you can see it from across the room. It shrinks back to your normal size whenever gestures pause, so the size also tells you whether gestures are on. Choose the size (or 0 for off) in Settings → Voice & input → Cursor size while gestures are active. On Mac and Linux the pointer itself isn't changed; the cursor ring grows instead.

Palm-glide scroll (v0.1.14), step by step: hold one hand up flat, palm toward the screen, fingers together — that hand is your "trackpad". Touch the middle of that palm with the index fingertip of your other hand and hold it there for a moment (about a sixth of a second; the cursor ring, if you turned it on, goes cyan). Now slide the fingertip along the palm: toward the fingers scrolls up, toward the wrist scrolls down. Put your index and middle finger on the palm together and slide sideways to scroll left or right. Flick and lift, and the page keeps going for a moment and slows down, like on a phone; touch the palm again or make any other gesture to stop it. A quick tap on the palm (touch and lift, without sliding) can be given an action in Settings → Bindings → gesture:palm_tap (it does nothing until you do). While your fingertip is on the palm the cursor holds still and nothing clicks, and the flat hand never pauses gestures. Speed, the dead zone, the momentum and the rest are in Settings → Tuning → Palm scroll.

Which hand is which (hand roles, v0.1.14): Away works out by itself which hand is the pointer (moves the cursor, clicks and does every one-hand gesture) and which is the helper (the flat "trackpad" hand). The first hand that points gets the pointer (or simply the only hand in view), and it keeps it — even if the camera mixes up left and right, or the hand disappears for up to a second. Raised your flat "trackpad" hand first? That's fine: as soon as the other hand points, it becomes the pointer. Holding your phone in the other hand? Pick Settings → Voice & input → Hand roles → Right or Left to use only that hand as the pointer; Auto is the default. You can also say "use left hand", "use right hand", or "use either hand" (back to Auto).

Tips for smooth control:

• Small movements go a long way. The cursor speeds up when you move fast and slows for precise work, so slow down near small buttons.
• Pinch crisply and keep your hand still while you do — the cursor locks briefly so the click lands where you aimed.
• Rest your elbow on the desk or armrest. Holding your arm up in the air gets tiring fast.
• Touching your real mouse or keyboard pauses gestures automatically. They come back after a second or two of not touching them.
• F9 turns gesture control on or off at any time. The tray menu does the same.
• Several monitors? The cursor covers all of them. Say "next monitor", "previous monitor" or "monitor two" to jump.

4. Voice

Voice is always listening by default. Just talk, and your words are typed wherever the text cursor is.

• Punctuation: say "comma", "period", "question mark", "new line", "new paragraph".
• Mistake? Say "scratch that" to remove the sentence you just said.
• Commands: "click", "double click", "right click", "scroll up", "scroll down", "go back", "go forward", "open browser", "pause gestures", "resume gestures", "stop listening", "start listening", "use left hand", "use right hand", "use either hand".
• Shortcuts: Ctrl+Alt+Space pauses/resumes listening, Ctrl+Alt+M mutes the mic, Ctrl+Alt+E lets you edit selected text by voice.
• Tray icon colour: green = listening, amber = not listening, red = mic off, blue = recording, grey = paused.
• Words not being typed? If you're looking away from the webcam, the face check may ignore you. Turn Face gate off in the tray menu.
• Speech is processed on your PC and never saved or uploaded.

5. If something goes wrong

• Models didn't finish downloading: reopen the app (see Installing above). It resumes.
• Cursor doesn't move: check another app isn't using the camera (Zoom, Teams, the Camera app), then restart Project Away from the tray.
• Voice is slow or lagging: install the GPU pack (Settings → Performance, NVIDIA cards only).
• Want to change a gesture or phrase? Tray → Settings… → Bindings.

Architecture

flowchart LR
  CAM[Webcam] --> HL[MediaPipe Hand Landmarker]
  CAM2[Optional 2nd camera] --> PAIR[Threaded capture + timestamp pairing]
  CAM --> PAIR
  PAIR --> HL
  HL --> MV[Multi-view fusion: confidence, triangulation, handover]
  MV --> NORM[Normalize + One Euro filter]
  NORM --> GSM[Gesture state machine + static/TCN models]
  GSM --> STEADY[Steady click + sigmoid curve + dwell/push]
  CAM --> FL[MediaPipe Face Landmarker]
  FL --> ASD[Active speaker + dictation gate]
  MIC[Microphone] --> VAD[Silero VAD]
  VAD --> ASR[faster-whisper / Parakeet]
  ASD --> ASR
  ASR --> CLEAN[Rule cleanup / optional LLM]
  WIFI[Wi-Fi CSI or RSSI] --> CSI[Hampel + Doppler features]
  CSI --> WCLS[Coarse presence/swipe classifier]
  STEADY --> FUS[Fusion + event bus]
  GUARD[Physical mouse/keyboard guard] --> FUS
  WCLS --> FUS
  FUS --> MOUSE[Cursor / click / scroll]
  CLEAN --> KB[Type into active app]
  FUS --> RING[Cursor ring overlay]

Quickstart

Full per-OS instructions: docs/INSTALL.md. Wi-Fi sensing: docs/wifi-setup.md.

pipx install ".[all]"        # Python 3.10–3.12
away setup                   # downloads models, checks camera + mic
away demo                    # synthetic end-to-end check
away run                     # or: away tray ; away autostart on

One-click installers (Releases page) bundle every feature: tracking, dictation, tray, settings window and on-device training (CPU PyTorch). Parakeet/NeMo is the one exception (multi-GB, NVIDIA-oriented; use pipx install "project-away[all,parakeet]"). Installers are ~0.4–0.8 GB; the models (~1.6 GB, or ~0.5 GB with Whisper small) download during install on Windows and on first launch elsewhere.

• Windows: run the setup.exe. It's unsigned, so choose More info → Run anyway, then Install. Models, start-at-sign-in, a desktop shortcut and launching the tray app are all on by default.
• macOS: open the .dmg, drag to Applications, then right-click → Open. The first launch downloads the models with a notification. Allow Camera, Microphone, Accessibility and Input Monitoring.
• Linux: run the AppImage or install the .deb; the first launch downloads the models. Mouse and keyboard control needs X11; Wayland is untested.

Customizing controls

Every control is a trigger → action binding you can change without touching code: from the Settings window (tray → Settings..., or away settings), from the CLI, or in config.yaml (away config path shows where it is). A running Away reloads the file within a second of a save.

Default bindings

Triggers: the gestures above plus your own (record with away record --custom --label wave, then away calibrate), any spoken phrase (matched ignoring case and punctuation), face:brow_raise, face:mouth_open, face:wink_left|wink_right, gesture:dwell, gesture:push, and any key combo (hotkey:ctrl+alt+k). Two fixed hotkeys sit outside the bindings: app.toggle_hotkey (default F9, gestures on/off) and voice.edit_hotkey (default Ctrl+Alt+E, edit the selected text by voice).

Actions: none · mouse:click|double_click|right_click|middle_click|scroll_up|scroll_down (mouse:move, mouse:drag, mouse:scroll, mouse:autoscroll only on point / pinch_hold / two_finger_scroll / four_finger_scroll) · key:ctrl+tab · type:any text · launch:firefox or launch:https://… (no shell) · shell:command (off unless bindings.allow_shell: true) · dictation:toggle|start|stop · mic:toggle|on|off · voice:command_mode|dictation_mode|toggle_mode · pause:toggle|on|off · cursor:next_monitor|prev_monitor|primary_monitor|monitor N (jump to the centre of a monitor) · hands:auto|right|left|any (which hand is the pointer until the app restarts).

Multiple monitors

The cursor moves across every monitor (v0.1.6; before that it was stuck on the primary). away monitors lists them, numbered 1..N left to right (then top to bottom), with where the list came from: Windows EnumDisplayMonitors, macOS Quartz/AppKit, Linux screeninfo or xrandr --listmonitors, falling back to the primary monitor only (with the reason) when none works. The same list is in away hardware and the selftest.

• Layouts: side by side, stacked, a monitor left of / above the primary (negative coordinates), mixed sizes. The cursor is always clamped onto a real monitor, never into the gap beside a smaller screen.
• Speed: relative-mode speed is measured against the primary monitor, so slow, precise moves feel exactly as on one screen. Only fast moves get extra reach on a bigger desktop: (desktop width / primary width) ** hands.cursor.monitor_reach. With three equal monitors and the default monitor_reach: 0.5 a quick flick goes ~1.7x further, so crossing all three takes a few flicks rather than three times the arm sweep. 0 = no extra reach, 1 = scales fully with the desktop.
• Jump: say "next monitor", "previous monitor" or "monitor two" (Whisper writing "monitor 2" works too), or bind cursor:next_monitor / cursor:monitor 3 to any gesture or hotkey. A held drag stays held, so "next monitor" while dragging carries a window across.
• Which monitors: hands.cursor.monitors: all (default) | primary | [1, 3].
• Hot-plug: monitors are re-detected every hands.cursor.monitor_refresh_s (3 s; 0 = only at start), no restart needed.
• Windows DPI: the app makes itself per-monitor DPI aware at startup (before pynput/tkinter), so monitor rectangles and cursor positions are both physical pixels on mixed-scaling setups. AWAY_DPI_AWARE=0 turns that off.

hands:
  cursor:
    monitors: all          # all | primary | [1, 3]
    monitor_reach: 0.5     # extra reach for fast moves on a wide desktop
    monitor_refresh_s: 3   # re-detect plugged/unplugged monitors

Per-app profiles

default holds the full map. Any other profile names the apps (process names, case-insensitive, .exe optional) it applies to and overrides only the triggers it lists; everything else falls through to default. active_profile: auto switches profile with the focused app (Linux needs xdotool); away profile use chrome pins one.

CLI

away bindings list [--profile chrome] [--json]
away bindings set gesture:swipe_left key:alt+left
away bindings set "voice:new tab" key:ctrl+t --profile chrome
away bindings reset [TRIGGER] [--profile NAME | --all]
away bindings export my-controls.yaml        # or "-" for stdout
away bindings import my-controls.yaml [--replace]   # default merges
away profile list | use NAME|auto | new NAME --apps chrome chrome.exe [--copy-from default] | remove NAME
away voice show | mode always_on|push_to_talk|toggle | wake-word away|off | intent dictation|command | sensitivity 0.6
away settings                                # GUI editor (tkinter); prints CLI help when there is no display
away update --check                          # current vs latest release (exit 0 up to date, 10 update available)
away update [--yes]                          # download, verify SHA-256, install (see "Updates")
away hardware [--json]                       # detected CPU/RAM/GPU + the performance profile and why
away gpu install | status | remove           # NVIDIA GPU pack (cuBLAS 12) for Whisper on CUDA

Invalid configs fail with every problem listed and its YAML path, e.g. bindings.profiles.default.gestures.pinchh: unknown gesture 'pinchh' (did you mean 'pinch'?).

Tunables

Every Settings tab scrolls (scroll bar or mouse wheel over the tab); the Save / Reset / Import / Export / Close bar and the status line below it stay visible at any window size (v0.1.10).

Sliders in the Settings window, or in YAML: hands.cursor.gain (sensitivity), dead_zone, acceleration, One Euro min_cutoff / beta / d_cutoff; hands.gestures.pinch_on / pinch_off (pinch threshold), scroll_gain (scroll speed), swipe_min_dx + swipe_max_time (swipe velocity = distance / time), double_pinch_window, drag_hold, clutch_hold; voice.vad_sensitivity, end_silence_ms, max_utterance_s. Each is range-checked.

v0.1.9 additions:

• Big cursor (hands.cursor.big_cursor_size, 0 = off, 1-5 = 1.5x/2x/3x/4x/6x of 32 px, default 2): on Windows the accessibility cursor size (HKCU\Control Panel\Cursors\CursorBaseSize) is raised while gesture control is active and put back on pause, auto-pause, clutch, F9, double clap, disable and quit (at most one change per 300 ms so auto-pause flapping does not thrash it). Your original size is saved to big_cursor.json in the data folder first; if Away crashes while the cursor is big, the next start restores it.
• Hand choice (hands.max_hands: 2): MediaPipe looks for two hands. Two-hand detection is only run every few frames until a second hand actually appears, so one hand costs about the same CPU as before. max_hands: 1 turns it off (and palm-glide scroll with it). v0.1.9's hands.prefer is replaced by hands.roles (below) in v0.1.14; an old prefer: right|left becomes roles.mode: right|left and prefer: any becomes roles.mode: auto automatically.
• Auto-scroll (hands.gestures.autoscroll_*): autoscroll_enabled, autoscroll_hold (0.2 s before it starts), autoscroll_dead_zone (0.25 palm lengths), autoscroll_gain (12), autoscroll_max_speed (30 notches/s), autoscroll_stop_hold (fist, 0.2 s), autoscroll_horizontal. Offsets are measured in palm lengths, so it feels the same near or far from the camera.
• Claps (voice.clap_*): clap_enabled, clap_sensitivity (0.5), clap_double_window (0.6 s), clap_require_hands (only when the camera sees both hands close together). Detection uses the voice microphone: a sharp, short (under ~50 ms), broadband transient that is far louder than the moment before it. Two hands close together on camera lower the bar; with no camera frames it is stricter. Ignored while you are mid-utterance and while the mic is muted. The Settings Test gesture tab shows claps as well as gestures.

v0.1.14 additions:

• Hand roles (hands.roles): mode: auto (default; the first hand that points, or the only hand in view, is the pointer) | right | left (that real hand, MediaPipe's label corrected for the mirrored image) | any (the first hand seen, whatever its pose; config only). release_s: 1.0: the pointer keeps its role while it is out of view this long (it is released sooner when it was last seen at the image border). Hands are told apart frame to frame by palm-centre position and size (a minimum-cost assignment), not by MediaPipe's handedness label, which flips; the label is only a weak, smoothed prior (and what right/left read, with hysteresis). A lone hand that has only held a flat palm hands the role to the first hand that points. Only the pointer drives the cursor, clicks and gestures; the helper feeds palm-glide scroll. A helper that becomes the pointer (the other hand left) cannot pause with its open palm until it has shown another pose; when the role moves to the other hand, a running drag ends and the cursor re-anchors without a jump.
• Palm-glide scroll (hands.palm_scroll): enabled: true, engage_s: 0.15 (fingertip on the palm this long before it scrolls), depth_threshold: 0.6 (fingertip at most this far off the palm plane, in palm lengths; 0 = no depth check), margin: 0.25 (palm outline padding, palm lengths), gain: 12 (wheel notches per palm length of glide), dead_zone: 0.05 (palm lengths; also the most a tap may move), invert: false, horizontal: true (index + middle = sideways), momentum: true, friction: 4.0 (1/s; speed × e^(−friction·t) after the lift). It engages when the helper is flat (four fingers out), the pointer's index fingertip falls inside the palm outline (wrist, thumb base, the four knuckles) and is near the palm plane. The glide is measured in the palm's own frame (wrist → middle knuckle = up), so a rotated or tilted palm scrolls the same way. With one camera, depth comes from MediaPipe's relative z and the hands' apparent sizes (both palms are assumed to be the same size); with two calibrated cameras the triangulated 3D points of both hands are used. If the helper's landmarks drop out or jump while the fingertip covers it, the last good palm is used for 0.3 s. The fingertip is smoothed with the One Euro filter. A tap (touch and lift within 0.25 s) fires gesture:palm_tap. While engaged, the pointer's cursor movement, clicks, four-finger auto-scroll and open-palm pause are suppressed.

Pointer accuracy:

• Steady click (hands.cursor.click_freeze: true): the cursor freezes while the pinch is closing (ratio below pinch_on * (1 + freeze_margin)), and the click lands where the cursor was click_rewind_ms (100) before closing began, because closing thumb and index drags the index tip. Drags unfreeze immediately.
• Curve (hands.cursor.curve): sigmoid (default) gives a slow plateau (sigmoid_low) for precise work and a fast plateau (sigmoid_high) with sigmoid_mid / sigmoid_slope in between. power is the original |d|^acceleration curve. speed_up/down/left/right scale each direction, for limited reach or an angled camera. The sigmoid idea comes from Google Project Gameface (Apache-2.0) and is re-implemented.
• Dwell click (hands.dwell): rest the cursor within radius_px for time_s and gesture:dwell fires.
• Face clicks: face.wink_trigger / wink_threshold / wink_open_max / wink_hold, plus brow_threshold, mouth_threshold and the hold times. Thresholds are per user, so tune them in Settings.
• Cursor ring (cursor_overlay: true or away run --cursor-overlay): a small ring by the pointer that fills as the pinch closes, turns solid on pinch, blue while dragging, purple while dwelling, orange with an arrow while auto-scrolling and grey when paused.
• Auto-pause (app.auto_pause: true): touching the real mouse or keyboard pauses gesture output until it has been idle for app.auto_pause_idle_ms. Away's own synthetic input is ignored.

Dictation extras: the vocabulary file is passed to faster-whisper as initial_prompt and as hotwords (when the installed version supports it; 1.2.1 does). The prompt holds only the vocabulary (Vocabulary: a, b, c.), never the focused app: Whisper repeats its prompt on short or noisy audio, and up to v0.1.5 that typed "Dictating into ." into the app. Since v0.1.6 every transcript (push-to-talk, always-on and the edit hotkey) is filtered before it is typed: prompt echoes ("Vocabulary: ...", "Dictating into ...") and bare lists of vocabulary terms are dropped or stripped, segments with a high no_speech_prob and a low avg_logprob are discarded, and stock silence phrases ("Thank you.", "Thanks for watching!", "Please subscribe") are dropped when the utterance is under 2 s. A phrase you bound as a voice command is never filtered. Dropped transcripts show in the log as dropped: <reason>. voice.numbers_as_digits turns "twenty five" into 25 from numbers_min_value (10) up. voice.edit_hotkey works like this: select text, press the hotkey, say an instruction ("uppercase", "replace cat with dog", "clean it up", "make it a bullet list", or anything else when an LLM endpoint is configured), then press it again.

Voice input modes

voice.mode is always_on (default), push_to_talk (hold voice.hold_key) or toggle (press voice.hotkey).

Always-on listens continuously, cuts speech into utterances with an energy VAD (end-pointing after end_silence_ms of silence) and decides per utterance:

• Wake word on (voice.wake_word_enabled: true, wake_word: away): only utterances starting with the wake word count ("away, scroll down"). A bare "away" opens a wake_window for the next utterance.
• Wake word off: text is typed only when the face gate agrees that you are at the screen and your mouth is moving (active-speaker detection), so TV or background talk is ignored. A visible face whose mouth moves counts even when the head is turned (off-axis webcam), and a short gap in face tracking while the CPU is busy is tolerated. Since v0.1.12 "mouth moving" is judged against your own resting mouth level (a webcam at arm's length and off-angle sees small movements), and when your enrolled face is at the screen and facing the camera in at least 60% of an utterance's frames it is typed even with weak mouth motion, unless another face is clearly the speaker. Each voice.utterance log line carries the gate's evidence (face frames, frontal fraction, speaking fraction, mouth score and threshold). After 10 rejections in a row the tray suggests turning the gate off (tray menu Face gate, or Settings). With the face gate off (face.gate_dictation: false), speech is typed without a wake word. With no camera, voice.ambient_fallback: commands_only (default) runs bound phrases but types nothing.
• If speech recognition cannot keep up (e.g. a big model on the CPU), a short backlog is kept, the oldest utterance is dropped, and you get one notification saying so.
• voice.intent: command never types, it only runs voice commands. Say "command mode" / "dictation mode" to switch.
• "stop listening" / "start listening", the mute hotkey and the tray's Mic on item control it. The tray icon shows the state (green active, red mic off, amber not listening, blue recording, grey paused).
• Audio is kept in memory only and dropped after transcription. Nothing is written to disk.

app.always_on: true (default) makes gestures and voice active as soon as the app starts; false starts paused and muted until you resume from the tray. app.start_minimized: false also opens Settings at launch.

Example config.yaml

app:
  always_on: true
  start_minimized: true
voice:
  mode: always_on            # always_on | push_to_talk | toggle
  wake_word_enabled: false
  wake_word: away
  intent: dictation          # dictation | command
  vad_sensitivity: 0.5
  mute_hotkey: <ctrl>+<alt>+m
hands:
  cursor: {curve: sigmoid, gain: 2.2, dead_zone: 0.004, min_cutoff: 1.0, beta: 0.02, click_freeze: true}
  gestures: {pinch_on: 0.32, pinch_off: 0.45, scroll_gain: 40.0, swipe_min_dx: 0.25, swipe_max_time: 0.45}
bindings:
  active_profile: auto
  allow_shell: false
  custom_gestures: [wave]
  profiles:
    default:
      gestures:
        swipe_left: key:alt+left
        swipe_right: key:alt+right
        wave: launch:https://calendar.google.com
      voice:
        "screenshot": key:cmd+shift+s
      hotkeys:
        ctrl+alt+d: type:Best regards,
    chrome:
      apps: [chrome, chrome.exe, Google Chrome]
      gestures:
        two_finger_scroll: key:ctrl+tab
      voice:
        "new tab": key:ctrl+t
        "close tab": key:ctrl+w
    code:
      apps: [code, Code.exe]
      gestures:
        open_palm: key:ctrl+shift+p   # open palm no longer pauses in VS Code

Multi-camera

A laptop webcam and an external camera can work together. A hand that one camera sees edge-on, or that is occluded, is usually clear in the other.

away cameras list                 # device names (V4L2 / Windows DirectShow / macOS) without opening them
away cameras use "HD User Facing" # single-camera mode: pick the camera by name
away cameras add 2                # index, /dev/video2, or part of the name ("brio"); saved by name
away cameras test                 # can all configured cameras stream at the same time? (~10 s)
away cameras test --save          # ...and save the combination that works (e.g. secondary at 320x240 MJPG)
away calibrate-cameras            # wave an open hand through both views for ~20 s
away calibrate-cameras --method checkerboard --checkerboard 9x6 --square-mm 25   # accurate option
away calibrate-cameras --print-board board.png --checkerboard 9x6                # printable board

The same is available in Settings → Cameras (device pickers, roles, calibrate button, live per-camera confidence). Calibrate runs calibrate-cameras as a separate process (in the installed Windows app that is ProjectAway.exe calibrate-cameras, not python -m away), borrows the cameras from the running app, shows progress in the status line and ends with "calibrated, reprojection error X px" or the error. Its full output is saved as calibration.log next to calibration-status.json in the data folder.

Two cameras at the same time (v0.1.15). The app, Settings' Live confidence and Test both cameras, away calibrate-cameras and away cameras test now all open the cameras the same way: the primary first, then each other camera once the primary has delivered its first frame (at most 8 s later), one after another on Windows, with the same fallbacks (MJPG/YUY2/NV12, the device default, and for a secondary 320x240 and 320x240 @ 15 fps, then the NVIDIA Broadcast camera if hands.virtual_fallback). Live confidence shows each camera's state in the Confidence column (opening…, the confidence, no frames, or a short error) and the mode that worked in the status line. Calibrate and Live confidence never run together: Calibrate stops the live view and waits until its cameras are released, and keeps them borrowed from the app across the hand-over. If the secondary still fails, calibration retries it once at 320x240 MJPG after the primary is open. A camera that only works in a fallback mode is remembered by name (camera-modes.json in the data folder) and opened in that mode right away at the next start. Test both cameras / away cameras test streams every camera for ~10 s and reports backend, mode, measured fps, frames and errors; if they do not all stream at the configured size it tries the secondary at 320x240 MJPG @ 15 and offers to save what works. While the app is still probing a slow camera it now says it holds the cameras and lets go as soon as Settings or calibration asks. In YAML:

hands:
  cameras:                       # empty (default) = single-camera mode, unchanged
    - {device: 0, role: primary, width: 640, height: 480, fps: 30, hfov_deg: 70}
    - {device: "Logitech BRIO", role: secondary, width: 1280, height: 720, fps: 30, hfov_deg: 78}
  multicam:
    fusion: weighted             # weighted | best_view
    pair_tolerance_ms: 40
    dropout_ms: 300
    handover_decay_s: 1.0
    use_3d_pinch: true
    push_distance_m: 0.06
    push_time_s: 0.35

Cameras are chosen by name (v0.1.11). Settings and away cameras use/add save the camera's name (hands.camera_index: "HD User Facing", or device: in hands.cameras) plus its current index as a hint (camera_hint / index_hint). The name is resolved to the camera's current index at every open, including retries, so a hot-plugged camera can never take its place; two cameras with the same name are told apart by their Windows DevicePath or NAME #2. Plain integer indices still work. On Windows the list is DirectShow's own (ICreateDevEnum, the order cv2.CAP_DSHOW uses; WMI/PnP only as a logged fallback) and every camera, single or multi, is opened with DirectShow (Media Foundation only as a logged fallback). Virtual cameras (NVIDIA Broadcast, OBS Virtual Camera, Iriun, DroidCam, Camo, Snap Camera, ManyCam, XSplit...) are marked (virtual) and never picked automatically: an old integer index that now lands on one (NVIDIA Broadcast is often DirectShow camera 0) is moved to the first real camera, with a line in away.log. Choose one by name to use it anyway. The camera list (source, index, name, virtual) is logged at start-up and when it changes.

Pixel format and busy cameras (v0.1.12). Every camera is opened in MJPG (hands.camera_fourcc for a single camera, fourcc: per entry in hands.cameras; "" = let the driver choose, which is usually uncompressed YUY2) and counts as open only once a frame arrives; a camera that sends nothing in MJPG is retried in its default format. A second camera that will not stream while another one is open is retried at 320x240, then 320x240 at 15 fps (two USB 2.0 webcams can exceed the bus); the mode that worked is logged and reused for the rest of the session. Errors now say whether a camera did not open or opened but sent no frames. Media Foundation (v0.1.13): when DirectShow cannot use a camera, it is opened with Media Foundation at the index Media Foundation's own list gives it (matched by device path, else by name); v0.1.12 and older reused the DirectShow index, but Media Foundation numbers cameras differently (it does not list DirectShow-only virtual cameras such as NVIDIA Broadcast), so the fallback hit the wrong camera. DirectShow is tried in MJPG, YUY2 and NV12 at the configured size, without a format, and with nothing forced; every attempt is logged. NVIDIA Broadcast: when a physical camera still cannot be opened and Camera (NVIDIA Broadcast) is listed, the log and a notification say " could not be opened directly; NVIDIA Broadcast's camera shows the same picture." With hands.virtual_fallback: true (default) Project Away then opens the one free Broadcast camera instead and logs it. Choosing Camera (NVIDIA Broadcast) by name works for the single camera and in a multi-camera set.

How it works:

1. Capture. Each camera runs in its own thread and stamps frames on arrival. The primary camera sets the pace. Each secondary contributes the frame nearest in time, if it falls within pair_tolerance_ms. A camera that stays silent for dropout_ms is treated as dropped and the others carry on.
2. Per-view confidence. Each view is scored on the MediaPipe handedness score, the share of landmarks inside the image, hand size in pixels, and sharpness (Laplacian variance of the hand crop, which drops with motion blur).
3. Fusion.
   • Uncalibrated: best_view takes the most confident view. weighted averages the views by confidence after mapping secondaries into the primary image with an alignment learned while both cameras see the hand.
   • Calibrated, with both cameras seeing the hand: the 21 landmarks are triangulated (cv2.triangulatePoints). Reprojecting them into the primary view drives the cursor, and the 3D points give true depth.
4. Depth uses.
   • gesture:push: the hand moves push_distance_m toward the screen within push_time_s. The primary camera is assumed to sit at the screen. The gesture is bindable and clicks by default.
   • 3D pinch: the pinch ratio comes from 3D distances, so a thumb and index that only overlap from one angle no longer click.
5. Occlusion handover. When one view loses the hand, tracking continues from the other. The jump between estimates is absorbed by an offset that fades over handover_decay_s, so the One Euro filter sees a continuous signal and the cursor does not jump.

Calibration. The hand-motion path pairs MediaPipe landmarks across views and runs cv2.findFundamentalMat (RANSAC, then a least-squares refit on the inliers) → E = K2ᵀ F K1 → cv2.recoverPose. Intrinsics are approximated from hfov_deg, and the translation is scaled so the median palm (wrist to middle knuckle) is palm_length_m (9 cm). Depth is therefore approximately metric; real palms vary by about 10–15%. The checkerboard path runs calibrateCamera per camera, then stereoCalibrate with the real square size. Either way the extrinsics go to <data dir>/camera_extrinsics.json and the reprojection error is printed.

Each view runs its own MediaPipe instance in sequence, so two cameras cost about twice the CPU of one.

A camera that cannot be opened or sends no frames is skipped with a notification and tracking continues on the one that works, so a wrong or busy secondary camera never stops hand tracking.

Two webcams on Windows (v0.1.8): at start-up each camera gets up to 12 s to send its first frame (Windows can take several seconds to open a USB webcam), and on Windows the cameras open one after another. A camera only counts as failed when it reports an error or stays silent past that deadline, and the notification names just that camera and why ("open error: ..." or "no frames within 12 s"). After falling back to one camera, the missing one is re-checked in the background every 45 s (backing off to 10 min if it keeps failing) without interrupting tracking; when it works again fusion switches back on and you get one "cameras back" notification. If no camera works at all, nothing claims to be tracking: the app says so once and keeps retrying. The Cameras tab's device list now always shows the cameras the app is holding and the configured ones, even when Windows' device query fails.

Sharing the camera with Settings (v0.1.6): webcams are exclusive on Windows, so while the app tracks your hand, Settings' Test gesture, Live confidence and Calibrate (and away calibrate-cameras) ask the running app to hand the cameras over: it pauses tracking, releases them, and takes them back when the test ends or the window closes (a small heartbeat file in the data folder; a crashed Settings window releases its claim within 5 s). The Cameras tab lists the default camera in single-camera mode and marks the one the app is using. If a camera still cannot be opened, you are told it is in use by Project Away or another app instead of a bare "permission denied".

Hardware auto-tuning and GPU

away hardware shows the detected CPU, RAM, disk and GPU (NVIDIA via NVML / nvidia-smi, others via WMI or lspci, Apple Silicon) and the profile picked from them: Whisper model, CPU or CUDA, compute type, threads, beam size, camera resolution/fps and face-tracking frame skipping, each with the reason. Explicit settings (performance.mode: auto|cpu|gpu, voice.device, voice.asr_model, performance.*) win.

NVIDIA GPUs: the one-click installers ship CPU-only; setup (or Settings > Performance, or away gpu install) downloads the GPU pack, the cuBLAS 12 libraries from NVIDIA's official PyPI wheel (~570 MB, SHA-256 verified). That is the only CUDA library CTranslate2 4.8 loads. pip installs: pip install nvidia-cublas-cu12==12.8.4.1. If CUDA fails (at startup or later), dictation falls back to the CPU with the CPU-sized model from the profile (not large-v3-turbo, which cannot keep up on most CPUs) and says why; the tray offers Install GPU pack (one notification, then a menu item). A voice.asr_model: large-v3-turbo written by v0.1.4's setup is migrated to auto (configs now carry config_version: 2; a model you pick from now on is kept). Details: docs/INSTALL.md.

Updates

The installed app checks GitHub Releases when the tray starts and every 24 hours. When a new version is out you get a notification and an Update to vX.Y.Z tray item; clicking it downloads the installer for your system, checks its SHA-256 (from the GitHub API digest, or the release's SHA256SUMS.txt; an unverifiable file is refused), installs it and restarts the app. Windows runs the Inno Setup installer silently; macOS swaps the .app from the .dmg after the app quits; an AppImage is replaced in place; a .deb is installed with pkexec apt install only after you confirm. pip/pipx installs are never touched: run pipx upgrade project-away.

updates:
  check: true          # false = never contact GitHub
  auto_install: false  # true = install silently when gestures are paused or idle (never for .deb)
  channel: stable      # beta = also offer pre-releases

The only network request is an anonymous GET to the public GitHub Releases API; nothing about you is sent. Same options under Settings → Voice & input; from a terminal: away update --check / away update --yes. The tray also has Check for updates, which always tells you the result; a check that fails (offline at sign-in) is retried after 30 minutes instead of the next day.

Windows updates (v0.1.10): the installer is downloaded to %LOCALAPPDATA%\ProjectAway\updates, verified, and started by a plain batch file next to it (away-update.cmd, run hidden by cmd.exe; no PowerShell). It logs every step to %LOCALAPPDATA%\ProjectAway\logs\update-helper.log, waits until Project Away (and the Settings window) has closed, runs the installer silently and records its exit code. Before quitting, the app checks that the helper is really running (its first log line, or the process still alive after 3 seconds); if not, it starts the installer directly instead. The app quits by itself (forced after a few seconds if something hangs), the installer writes logs\update-install.log, and the app restarts. On the next start you get "Project Away was updated to vX", "The update to vX is still installing" (with the helper's last step), or "The update to vX did not install" with the reason, the helper's last log line and the remedy: right-click the tray icon → Run update installer, which opens the kept installer normally (visible, like a double-click). Every step is logged to away.log.

Upgrading from v0.1.9 or older: those copies still use the old PowerShell helper, which did not run the installer on some Windows 11 PCs. If the tray update to v0.1.10 does not take, download ProjectAway-0.1.10-windows-x64-setup.exe from the releases page and run it once by hand (quit Project Away first). Updates after that use the new helper.

The v0.1.10 → v0.1.11 update is the first one installed by the batch helper on a real PC. If it does not take, use Run update installer in the tray, or run ProjectAway-0.1.11-windows-x64-setup.exe by hand.

v0.1.12 helper: the installer's exit code decides the outcome, not just the version number: a non-zero code (e.g. Inno Setup 5, aborted on a locked file) is reported as an incomplete update with the Run update installer item, even when the version already matches. The helper force-closes ProjectAway.exe and its child processes, retries a failed installer once after 5 s, clears the PyInstaller variables it inherited, and restarts Project Away itself after the installer finishes unless the installer's own relaunch already did (it watches for the app for up to 10 s first, so there is never a second copy).

Verified vs untested

Roadmap

1. Real-hardware validation on Windows/macOS/Linux.
2. Personal calibration data → fine-tune the temporal model on real recordings.
3. Face recognition for the enrolled-user gate (currently largest face).
4. TalkNet/Light-ASD active-speaker backend.
5. Live AX210 CSI capture and a learned swipe/push-pull classifier.
6. Phone-as-second-camera / IMU fusion.
7. Real-hardware tuning of multi-camera fusion; parallel per-camera landmark inference.

Development

pip install -e ".[dev]"
pytest -q && ruff check . && ruff format --check . && mypy src
python scripts/mutation.py && python scripts/mutation.py --summary

Private project. All rights reserved by the author until a license is chosen.