daft-physical-ai 0.1.1

Physical-AI data processing on Daft, starting with hand tracking.

daft-physical-ai

Physical-AI data processing on Daft, starting with hand tracking. The methods run as Daft UDFs, so they slot into any Daft pipeline and execute lazily, batched, and distributed.

Available on PyPI:

pip install daft-physical-ai[mediapipe]

API

The package operates on a Daft image column and returns a hand-pose column. A LeRobot dataset is a natural source: Daft's native reader daft.datasets.lerobot (added in Daft #7090) decodes each camera into an image column with load_video_frames.

import daft
from daft.datasets import lerobot
from daft_physical_ai.hands import track_hands

# one row per frame; the camera key is decoded into an image column.
# egodex-test is a tiny EgoDex sample (3 episodes / 632 frames) in LeRobot v3 format.
df = lerobot.read("pepijn223/egodex-test", load_video_frames="observation.image")

# pick a method (each returns the same schema):
# mediapipe -> CPU, 2D only, permissive license, no weights to supply
# wilor     -> GPU, 3D MANO keypoints (MANO weights user-supplied)
df = df.with_column("hands", track_hands(df["observation.image"], method="mediapipe"))

df.write_parquet("annotated/")

Install the method you need as an extra: pip install daft-physical-ai[mediapipe] (CPU, 2D), pip install daft-physical-ai[wilor] (GPU, 3D), or pip install daft-physical-ai[all] for both. WiLoR additionally needs a CUDA torch build and chumpy from git (pip install 'chumpy @ git+https://github.com/mattloper/chumpy', omitted from the extra because PyPI metadata can't carry direct references), plus a user-supplied MANO_RIGHT.pkl (research-gated).

Output schema

One unified output schema regardless of method: each frame yields a list of 0-2 detected hands. A single hand value (MediaPipe):

{
    "handedness": "right",        # "left", "right", or "unknown"
    "confidence": 0.979,
    "kp2d": [[1412.1, 1111.1],    # 21 image-space [x, y] keypoints
             [1357.9, 1075.9],
             ...],
    "kp3d": None,                 # 21 [x, y, z] keypoints, or null for 2D-only methods
}

The Daft type is list[struct{ handedness: string, confidence: float32, kp2d: list[list[float32]], kp3d: list[list[float32]] }], defined as HANDS_DTYPE in daft_physical_ai/hands/schema.py.

Example

A complete walkthrough - read a dataset, run track_hands (MediaPipe), draw the keypoints, and score against EgoDex ground truth:

Available in three equivalent forms:

• examples/demo.md - read it start to finish; code and outputs inline.
• examples/demo.ipynb - runnable notebook (outputs included).
• examples/demo.py - plain script.

Generate your own (other methods, a Modal GPU runtime, with/without eval) with the daft-physical-ai hands command - run it with no flags for an interactive walkthrough, or pass flags:

# No flags - interactive walkthrough that asks a few questions
daft-physical-ai hands

# --no-input skips all prompts; flags supply the answers, the rest use defaults
daft-physical-ai hands --method mediapipe --output-dir my-demo --no-input
daft-physical-ai hands --method wilor --runtime modal --mano-path ./MANO_RIGHT.pkl --no-input

Hand tracking is the first capability; each new one will be its own subcommand (daft-physical-ai <command> lists what's available).

The bare daft-physical-ai command comes with the PyPI package (pip install daft-physical-ai). From a clone of this repo, uv sync installs it too (uv run daft-physical-ai).

To run a generated demo you also need its inference stack. In a clone, uv sync already brings a Daft with the LeRobot reader; install the extras into the venv, then run from the activated venv - not uv run, which re-syncs the env and would drop them:

source .venv/bin/activate
uv pip install -U av mediapipe scipy opencv-python matplotlib jupyterlab
jupyter lab hand-tracking-demo/demo.ipynb

Development

uv sync                      # set up env + install deps
uv run pre-commit install    # install lint/format hooks
uv run pytest tests/ -v      # run the test suite

Versioning

Versions are derived from git tags via hatch-vcs. Tag releases as v0.1.0, v0.2.0, etc.

Publishing

Publishing a GitHub release triggers .github/workflows/publish-package.yml, which builds a wheel and sdist with uv build and uploads both to PyPI via trusted publishing. Configure the trusted publisher on PyPI for this repository before the first release.