Project Away

Hands-free PC control, free and local. Webcam hand gestures drive the cursor, push-to-talk voice typing handles text, face cues gate who is dictating, and optional Wi-Fi CSI adds coarse motion confidence.

Nothing leaves your machine unless you point the optional LLM cleanup at an endpoint yourself.

Architecture

flowchart LR
  CAM[Webcam] --> HL[MediaPipe Hand Landmarker]
  CAM2[Optional 2nd camera] --> PAIR[Threaded capture + timestamp pairing]
  CAM --> PAIR
  PAIR --> HL
  HL --> MV[Multi-view fusion: confidence, triangulation, handover]
  MV --> NORM[Normalize + One Euro filter]
  NORM --> GSM[Gesture state machine + static/TCN models]
  GSM --> STEADY[Steady click + sigmoid curve + dwell/push]
  CAM --> FL[MediaPipe Face Landmarker]
  FL --> ASD[Active speaker + dictation gate]
  MIC[Microphone] --> VAD[Silero VAD]
  VAD --> ASR[faster-whisper / Parakeet]
  ASD --> ASR
  ASR --> CLEAN[Rule cleanup / optional LLM]
  WIFI[Wi-Fi CSI or RSSI] --> CSI[Hampel + Doppler features]
  CSI --> WCLS[Coarse presence/swipe classifier]
  STEADY --> FUS[Fusion + event bus]
  GUARD[Physical mouse/keyboard guard] --> FUS
  WCLS --> FUS
  FUS --> MOUSE[Cursor / click / scroll]
  CLEAN --> KB[Type into active app]
  FUS --> RING[Cursor ring overlay]

Quickstart

Full per-OS instructions: docs/INSTALL.md. Wi-Fi sensing: docs/wifi-setup.md.

pipx install ".[all]"        # Python 3.10–3.12
away setup                   # downloads models, checks camera + mic
away demo                    # synthetic end-to-end check
away run                     # or: away tray ; away autostart on

One-click installers (Releases page) bundle every feature: tracking, dictation, tray, settings window and on-device training (CPU PyTorch). Parakeet/NeMo is the one exception (multi-GB, NVIDIA-oriented; use pipx install "project-away[all,parakeet]"). Installers are ~0.4–0.8 GB; the models (~1.6 GB, or ~0.5 GB with Whisper small) download during install on Windows and on first launch elsewhere.

• Windows: run the setup.exe. It's unsigned, so choose More info → Run anyway, then Install. Models, start-at-sign-in, a desktop shortcut and launching the tray app are all on by default.
• macOS: open the .dmg, drag to Applications, then right-click → Open. The first launch downloads the models with a notification. Allow Camera, Microphone, Accessibility and Input Monitoring.
• Linux: run the AppImage or install the .deb; the first launch downloads the models. Mouse and keyboard control needs X11; Wayland is untested.

Customizing controls

Every control is a trigger → action binding you can change without touching code: from the Settings window (tray → Settings..., or away settings), from the CLI, or in config.yaml (away config path shows where it is). A running Away reloads the file within a second of a save.

Default bindings

Triggers: the gestures above plus your own (record with away record --custom --label wave, then away calibrate), any spoken phrase (matched ignoring case and punctuation), face:brow_raise, face:mouth_open, face:wink_left|wink_right, gesture:dwell, gesture:push, and any key combo (hotkey:ctrl+alt+k). Two fixed hotkeys sit outside the bindings: app.toggle_hotkey (default F9, gestures on/off) and voice.edit_hotkey (default Ctrl+Alt+E, edit the selected text by voice).

Actions: none · mouse:click|double_click|right_click|middle_click|scroll_up|scroll_down (mouse:move, mouse:drag, mouse:scroll only on point / pinch_hold / two_finger_scroll) · key:ctrl+tab · type:any text · launch:firefox or launch:https://… (no shell) · shell:command (off unless bindings.allow_shell: true) · dictation:toggle|start|stop · mic:toggle|on|off · voice:command_mode|dictation_mode|toggle_mode · pause:toggle|on|off.

Per-app profiles

default holds the full map. Any other profile names the apps (process names, case-insensitive, .exe optional) it applies to and overrides only the triggers it lists; everything else falls through to default. active_profile: auto switches profile with the focused app (Linux needs xdotool); away profile use chrome pins one.

CLI

away bindings list [--profile chrome] [--json]
away bindings set gesture:swipe_left key:alt+left
away bindings set "voice:new tab" key:ctrl+t --profile chrome
away bindings reset [TRIGGER] [--profile NAME | --all]
away bindings export my-controls.yaml        # or "-" for stdout
away bindings import my-controls.yaml [--replace]   # default merges
away profile list | use NAME|auto | new NAME --apps chrome chrome.exe [--copy-from default] | remove NAME
away voice show | mode always_on|push_to_talk|toggle | wake-word away|off | intent dictation|command | sensitivity 0.6
away settings                                # GUI editor (tkinter); prints CLI help when there is no display

Invalid configs fail with every problem listed and its YAML path, e.g. bindings.profiles.default.gestures.pinchh: unknown gesture 'pinchh' (did you mean 'pinch'?).

Tunables

Sliders in the Settings window, or in YAML: hands.cursor.gain (sensitivity), dead_zone, acceleration, One Euro min_cutoff / beta / d_cutoff; hands.gestures.pinch_on / pinch_off (pinch threshold), scroll_gain (scroll speed), swipe_min_dx + swipe_max_time (swipe velocity = distance / time), double_pinch_window, drag_hold, clutch_hold; voice.vad_sensitivity, end_silence_ms, max_utterance_s. Each is range-checked.

Pointer accuracy:

• Steady click (hands.cursor.click_freeze: true): the cursor freezes while the pinch is closing (ratio below pinch_on * (1 + freeze_margin)), and the click lands where the cursor was click_rewind_ms (100) before closing began, because closing thumb and index drags the index tip. Drags unfreeze immediately.
• Curve (hands.cursor.curve): sigmoid (default) gives a slow plateau (sigmoid_low) for precise work and a fast plateau (sigmoid_high) with sigmoid_mid / sigmoid_slope in between. power is the original |d|^acceleration curve. speed_up/down/left/right scale each direction, for limited reach or an angled camera. The sigmoid idea comes from Google Project Gameface (Apache-2.0) and is re-implemented.
• Dwell click (hands.dwell): rest the cursor within radius_px for time_s and gesture:dwell fires.
• Face clicks: face.wink_trigger / wink_threshold / wink_open_max / wink_hold, plus brow_threshold, mouth_threshold and the hold times. Thresholds are per user, so tune them in Settings.
• Cursor ring (cursor_overlay: true or away run --cursor-overlay): a small ring by the pointer that fills as the pinch closes, turns solid on pinch, blue while dragging, purple while dwelling and grey when paused.
• Auto-pause (app.auto_pause: true): touching the real mouse or keyboard pauses gesture output until it has been idle for app.auto_pause_idle_ms. Away's own synthetic input is ignored.

Dictation extras: the vocabulary file is passed to faster-whisper as initial_prompt and as hotwords (when the installed version supports it; 1.2.1 does). voice.numbers_as_digits turns "twenty five" into 25 from numbers_min_value (10) up. voice.edit_hotkey works like this: select text, press the hotkey, say an instruction ("uppercase", "replace cat with dog", "clean it up", "make it a bullet list", or anything else when an LLM endpoint is configured), then press it again.

Voice input modes

voice.mode is always_on (default), push_to_talk (hold voice.hold_key) or toggle (press voice.hotkey).

Always-on listens continuously, cuts speech into utterances with an energy VAD (end-pointing after end_silence_ms of silence) and decides per utterance:

• Wake word on (voice.wake_word_enabled: true, wake_word: away): only utterances starting with the wake word count ("away, scroll down"). A bare "away" opens a wake_window for the next utterance.
• Wake word off: text is typed only when the face gate agrees that you are at the screen and your mouth is moving (active-speaker detection), so TV or background talk is ignored. With no camera, voice.ambient_fallback: commands_only (default) runs bound phrases but types nothing.
• voice.intent: command never types, it only runs voice commands. Say "command mode" / "dictation mode" to switch.
• "stop listening" / "start listening", the mute hotkey and the tray's Mic on item control it. The tray icon shows the state (green active, red mic off, amber not listening, blue recording, grey paused).
• Audio is kept in memory only and dropped after transcription. Nothing is written to disk.

app.always_on: true (default) makes gestures and voice active as soon as the app starts; false starts paused and muted until you resume from the tray. app.start_minimized: false also opens Settings at launch.

Example config.yaml

app:
  always_on: true
  start_minimized: true
voice:
  mode: always_on            # always_on | push_to_talk | toggle
  wake_word_enabled: false
  wake_word: away
  intent: dictation          # dictation | command
  vad_sensitivity: 0.5
  mute_hotkey: <ctrl>+<alt>+m
hands:
  cursor: {curve: sigmoid, gain: 2.2, dead_zone: 0.004, min_cutoff: 1.0, beta: 0.02, click_freeze: true}
  gestures: {pinch_on: 0.32, pinch_off: 0.45, scroll_gain: 40.0, swipe_min_dx: 0.25, swipe_max_time: 0.45}
bindings:
  active_profile: auto
  allow_shell: false
  custom_gestures: [wave]
  profiles:
    default:
      gestures:
        swipe_left: key:alt+left
        swipe_right: key:alt+right
        wave: launch:https://calendar.google.com
      voice:
        "screenshot": key:cmd+shift+s
      hotkeys:
        ctrl+alt+d: type:Best regards,
    chrome:
      apps: [chrome, chrome.exe, Google Chrome]
      gestures:
        two_finger_scroll: key:ctrl+tab
      voice:
        "new tab": key:ctrl+t
        "close tab": key:ctrl+w
    code:
      apps: [code, Code.exe]
      gestures:
        open_palm: key:ctrl+shift+p   # open palm no longer pauses in VS Code

Multi-camera

A laptop webcam and an external camera can work together. A hand that one camera sees edge-on, or that is occluded, is usually clear in the other.

away cameras list                 # V4L2 names on Linux; probes indices on Windows/macOS
away cameras add 2                # index, /dev/video2, or part of the name ("brio")
away calibrate-cameras            # wave an open hand through both views for ~20 s
away calibrate-cameras --method checkerboard --checkerboard 9x6 --square-mm 25   # accurate option
away calibrate-cameras --print-board board.png --checkerboard 9x6                # printable board

The same is available in Settings → Cameras (device pickers, roles, calibrate button, live per-camera confidence). In YAML:

hands:
  cameras:                       # empty (default) = single-camera mode, unchanged
    - {device: 0, role: primary, width: 640, height: 480, fps: 30, hfov_deg: 70}
    - {device: "Logitech BRIO", role: secondary, width: 1280, height: 720, fps: 30, hfov_deg: 78}
  multicam:
    fusion: weighted             # weighted | best_view
    pair_tolerance_ms: 40
    dropout_ms: 300
    handover_decay_s: 1.0
    use_3d_pinch: true
    push_distance_m: 0.06
    push_time_s: 0.35

How it works:

1. Capture. Each camera runs in its own thread and stamps frames on arrival. The primary camera sets the pace. Each secondary contributes the frame nearest in time, if it falls within pair_tolerance_ms. A camera that stays silent for dropout_ms is treated as dropped and the others carry on.
2. Per-view confidence. Each view is scored on the MediaPipe handedness score, the share of landmarks inside the image, hand size in pixels, and sharpness (Laplacian variance of the hand crop, which drops with motion blur).
3. Fusion.
   • Uncalibrated: best_view takes the most confident view. weighted averages the views by confidence after mapping secondaries into the primary image with an alignment learned while both cameras see the hand.
   • Calibrated, with both cameras seeing the hand: the 21 landmarks are triangulated (cv2.triangulatePoints). Reprojecting them into the primary view drives the cursor, and the 3D points give true depth.
4. Depth uses.
   • gesture:push: the hand moves push_distance_m toward the screen within push_time_s. The primary camera is assumed to sit at the screen. The gesture is bindable and clicks by default.
   • 3D pinch: the pinch ratio comes from 3D distances, so a thumb and index that only overlap from one angle no longer click.
5. Occlusion handover. When one view loses the hand, tracking continues from the other. The jump between estimates is absorbed by an offset that fades over handover_decay_s, so the One Euro filter sees a continuous signal and the cursor does not jump.

Calibration. The hand-motion path pairs MediaPipe landmarks across views and runs cv2.findFundamentalMat (RANSAC, then a least-squares refit on the inliers) → E = K2ᵀ F K1 → cv2.recoverPose. Intrinsics are approximated from hfov_deg, and the translation is scaled so the median palm (wrist to middle knuckle) is palm_length_m (9 cm). Depth is therefore approximately metric; real palms vary by about 10–15%. The checkerboard path runs calibrateCamera per camera, then stereoCalibrate with the real square size. Either way the extrinsics go to <data dir>/camera_extrinsics.json and the reprojection error is printed.

Each view runs its own MediaPipe instance in sequence, so two cameras cost about twice the CPU of one.

Verified vs untested

Roadmap

1. Real-hardware validation on Windows/macOS/Linux.
2. Personal calibration data → fine-tune the temporal model on real recordings.
3. Face recognition for the enrolled-user gate (currently largest face).
4. TalkNet/Light-ASD active-speaker backend.
5. Live AX210 CSI capture and a learned swipe/push-pull classifier.
6. Phone-as-second-camera / IMU fusion.
7. Real-hardware tuning of multi-camera fusion; parallel per-camera landmark inference.

Development

pip install -e ".[dev]"
pytest -q && ruff check . && ruff format --check . && mypy src
python scripts/mutation.py && python scripts/mutation.py --summary

Private project. All rights reserved by the author until a license is chosen.