fasthamer

Realtime HaMeR 3D hand mesh recovery on the Apple Neural Engine. Give it an image, get MANO parameters, 3D joints, camera parameters, and 2D joints per hand — a MediaPipe-Hands-style API, but with a full 3D hand mesh behind it.

• Fast: the whole network (ViT-H backbone + MANO head) runs as one CoreML program on the ANE, the MANO mesh is a sub-millisecond numpy step — ~30 FPS end-to-end with two hands at 640px on a fanless M4 MacBook Air, torch-free at runtime.
• Simple: fasthamer.load() → result = hands(rgb) → done.
• Complete outputs: MANO global_orient / hand_pose / betas (rotation matrices and axis-angle), 778-vertex mesh, 21 3D joints, camera intrinsics + per-hand translation, projected 2D joints.
• Batteries included: hand detection via fasthands (CoreML/ANE, ~1 ms) and a tiny C mesh rasterizer for overlays (~10x faster than pyrender, no GPU/OpenGL).

Requires macOS on Apple Silicon.

Install

You need your own MANO model. fasthamer computes the hand mesh with MANO, which is free for non-commercial research but license-gated, so fasthamer does not ship it. Create an account at https://mano.is.tue.mpg.de and accept the MANO license there; the setup below then either uses a MANO_RIGHT.pkl / mano_v1_2.zip you already have, or downloads mano_v1_2.zip with your MANO account email + password (sent only to the MPI server, never stored).

pip install fasthamer
fasthamer-setup                           # or: fasthamer-setup --mano ~/Downloads/mano_v1_2.zip
fasthamer-webcam --mirror                 # live demo: 3D mesh overlay from your webcam

fasthamer-setup runs once. It (1) imports your MANO model into ~/.cache/fasthamer/mano/ — prompting for a path or for your MANO account if it can't find one — and (2) downloads the prebuilt CoreML model bundle (~470 MB, no MANO data inside) into ~/.cache/fasthamer. If you skip this step, the same prompts run on your first fasthamer.load().

The first fasthamer.load() also compiles the model for your device (one-time, typically 10-30 s) and caches the compiled .mlmodelc; every load after that takes a couple of seconds.

Non-interactive environments (CI, scripts): point FASTHAMER_MANO_PATH at your MANO_RIGHT.pkl / mano_v1_2 folder / mano_v1_2.zip, or set MANO_USERNAME and MANO_PASSWORD to let fasthamer-setup download it. python -m fasthamer.mano_download downloads the zip on its own.

Quickstart

import cv2
import fasthamer

hands = fasthamer.load()                       # mode="image" by default
rgb = cv2.cvtColor(cv2.imread("photo.jpg"), cv2.COLOR_BGR2RGB)
result = hands(rgb)

for hand in result:
    hand.is_right          # bool (left hands: see note below)
    hand.keypoints_2d      # (21, 2)  pixels in the input image
    hand.keypoints_3d      # (21, 3)  meters, hand-centered
    hand.keypoints_camera  # (21, 3)  meters, camera frame (= keypoints_3d + cam_t)
    hand.vertices          # (778, 3) MANO mesh, hand-centered
    hand.cam_t             # (3,)     hand -> camera translation
    hand.global_orient     # (3, 3)   MANO global orientation (rotation matrix)
    hand.hand_pose         # (15, 3, 3) MANO pose  (also .hand_pose_aa / .global_orient_aa)
    hand.betas             # (10,)    MANO shape

result.focal, result.cx, result.cy   # pinhole camera: u = f*X/Z + cx
overlay = hands.render(rgb, result)               # mesh overlay (C rasterizer)
overlay = fasthamer.draw_landmarks(overlay, result)  # 2D skeleton

Video mode

mode="video" enables temporal hand tracking in the detector (faster and smoother than per-frame detection) — feed frames sequentially:

hands = fasthamer.load(mode="video")
while True:
    ok, frame = cap.read()
    result = hands(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))

Webcam demo

fasthamer-webcam (installed with the package) opens your webcam and overlays the reconstructed meshes in realtime:

fasthamer-webcam --mirror              # selfie view
fasthamer-webcam --skeleton            # 2D joints instead of the mesh
fasthamer-webcam --camera 0            # pick a camera (auto-probed by default)
fasthamer-webcam --stabilize           # lock Right/Left per hand across frames
fasthamer-webcam --camera clip.mp4     # run on a video file instead of a camera
fasthamer-webcam --record out.mp4      # also save the annotated output
fasthamer-webcam --no-display --max-frames 120   # headless benchmark, prints FPS

Keys: q / ESC quit, m toggle mesh, s toggle skeleton. It uses the same low-latency loop as the threaded example below (freshest-frame capture thread

• worker/display split), so the FPS shown is the real end-to-end rate — it is capped by your camera's frame rate, and only drops below that when the model is the bottleneck (hold your hands in view when benchmarking; with no hands only the ~1 ms detector runs). See fasthamer-webcam --help for the full option list (--max-hands, --alpha, --force-handedness, --fasthands-detector, --compute-units, --width/--height, ...).

Examples:

• examples/image_demo.py — single image in, mesh + skeleton overlay out.
• examples/webcam_demo.py — minimal synchronous webcam loop (easiest to read).
• examples/webcam_threaded_demo.py — low-latency webcam demo: a background camera thread always serves the freshest frame, and a worker/display split keeps the window responsive while the mesh stays aligned to the frame it was computed on. Use this one if the simple demo feels laggy (cv2.VideoCapture otherwise hands you buffered, stale frames). Supports --no-display --max-frames N for benchmarking.

Conventions

• Input images are RGB uint8 (H×W×3).
• 3D outputs use a camera-style frame: x right, y down, z forward, meters. hand.vertices / hand.keypoints_3d are hand-centered; add hand.cam_t (or use the *_camera properties) for camera-frame coordinates. Projection uses the pinhole camera in the result (result.project(pts)).
• Joint order is MANO/OpenPose-style: wrist, then 4 joints each for thumb, index, middle, ring, pinky (fasthamer.HAND_EDGES has the skeleton).
• Left hands: HaMeR mirrors left-hand crops, so MANO parameters always describe a right MANO hand. vertices/keypoints_3d are already un-mirrored. To re-pose a left hand yourself, apply the params to a right-hand MANO model and negate x.
• hands.faces gives the (1538, 3) triangle faces (flip winding for left hands).

Options

fasthamer.load(
    mode="image",             # or "video"
    max_hands=2,
    detector="fasthands",     # detection stack: "fasthands" or "mediapipe"
    fasthands_detector=None,  # model inside fasthands: "whim" or "mediapipe"
    model_dir=None,           # local model bundle dir (skips the download)
    mano_path=None,           # your MANO_RIGHT.pkl / mano_v1_2 folder / zip
                              #   (default: the copy cached by fasthamer-setup)
    rescale_factor=2.0,       # hand-box padding before cropping
    swap_handedness=False,    # if handedness looks inverted (mirrored inputs)
    stabilize_handedness=False,  # video: lock R/L per hand across frames
    handedness_iou=0.3,       # IoU to treat a detection as the same hand
    handedness_ttl=10,        # drop a track after N unseen frames
    handedness_switch_frames=5,  # switch only after N consecutive disagreements
    force_handedness=None,    # "right" / "left" to pin it outright
    compute_units="CPU_AND_NE",
)

Choosing a hand detector

Two independent choices, and confusingly both offer something called "mediapipe" — they are not the same thing:

• detector picks the detection stack: "fasthands" (default) runs MediaPipe Hands ported to CoreML on the Neural Engine, ~1 ms/frame; "mediapipe" runs Google's official MediaPipe Tasks API on TFLite/XNNPACK/GPU (needs pip install "fasthamer[mediapipe]", and exposes det_conf / track_conf).
• fasthands_detector picks the detector model inside fasthands (requires fasthands >= 0.4.0): "whim" — the WHIM-fine-tuned full-hand-box detector, steadier than the palm detector; "mediapipe" — the original palm detector. Left as None, fasthands' own default (whim) applies.

hands = fasthamer.load(mode="video", fasthands_detector="mediapipe")

The two detector models frame hands differently — on the bundled test image the boxes are [408,507,583,921] (whim) vs [414,518,584,906] (palm) — and that box drives the crop fed to HaMeR, so the choice affects the reconstructed geometry, not just detection recall. Worth trying both on unusual footage (egocentric/wrist cameras, heavy foreshortening).

Handedness stability (video)

Detectors decide Right/Left from a single frame, so on hard footage — foreshortened, self-occluding hands, e.g. a wrist/egocentric camera — the label can flicker frame to frame. This is not cosmetic: HaMeR is a right-hand model, so "left" hands are mirror-flipped before the network and un-mirrored after. A flickered label produces mirror-wrong geometry on that frame (we measure ~200 mm of vertex jump on an otherwise identical frame), which is why relabeling the output afterwards is not enough.

stabilize_handedness=True fixes it upstream of the crop: each hand gets a persistent IoU-matched track, and a spatially continuous hand keeps the label it first appeared with. Tracks expire after handedness_ttl unseen frames so a genuinely new hand can re-acquire.

The lock is hysteretic, not permanent: a tracked hand only switches its label once the detector disagrees for handedness_switch_frames consecutive frames (default 5). A single agreeing frame resets the streak, so alternating flicker never accumulates into a switch — but a sustained correction still wins, so a genuinely misidentified hand recovers instead of staying wrong for the life of the track. Set handedness_switch_frames=0 for a permanent lock.

hands = fasthamer.load(mode="video", stabilize_handedness=True)
...
hands.reset()      # clear tracks when the video sequence restarts

For rigs where handedness is known and fixed (single-hand egocentric mounts), force_handedness="right" pins every detection outright — a stronger constraint that takes precedence over the tracker. Both are orthogonal to swap_handedness, which is a global flip applied by the detector.

Environment variables: FASTHAMER_MANO_PATH points at your MANO model (MANO_RIGHT.pkl, mano_v1_2/, or mano_v1_2.zip); MANO_USERNAME / MANO_PASSWORD let setup download it non-interactively; FASTHAMER_MODEL_DIR points at a local model bundle; FASTHAMER_ASSETS_URL overrides where the bundle is downloaded from.

How it works

MediaPipe-style palm detection (fasthands, ANE) finds hand boxes; each box is cropped HaMeR-style and run through a single CoreML mlprogram containing the ViT-H backbone (192×144 input, interpolated position embeddings) and the MANO transformer head — 6-bit palettized weights — which outputs the MANO parameters and camera. The MANO mesh itself (778 vertices, 21 joints) is then computed in numpy from your MANO_RIGHT.pkl (fasthamer.mano, ~0.5 ms per hand, no chumpy/scipy/torch); it matches HaMeR's smplx MANO layer to 1e-7 m. End-to-end outputs match the reference HaMeR torch pipeline to a few mm (fp16 + palettization noise). The overlay renderer is a small C rasterizer with supersampled anti-aliasing, compiled once on first use.

License

Code: MIT. The CoreML model bundle is converted from the MIT-licensed HaMeR checkpoint and contains no MANO data. The MANO model is licensed by the Max Planck Institute for non-commercial scientific research and is obtained by you from your own account at https://mano.is.tue.mpg.de under its license; fasthamer only reads the copy you provide. Cite HaMeR and MANO in academic work.