gesto

Train and run gesture recognition models from Gesto Labeller datasets.

Point it at a project folder, pick a mode and a region, and it handles the rest — loading, training, versioned model storage, and live detection that matches how the data was captured.

pip install gesto

Two modes

Static needs far less data (every captured frame is a training sample) and predicts instantly with no warm-up. Reach for sequence only when two gestures share the same shape and differ by movement.

Five regions

The region must match how the project was captured — gesto checks the feature dimension and tells you if it doesn't.

Command line

# what's in this dataset?
gesto inspect ./gesto_projects/signs

# train
gesto train static hands_one ./gesto_projects/signs
gesto train sequence pose ./gesto_projects/jogging --seq-len 30

# detect (newest model by default)
gesto detect static hands_one
gesto detect sequence pose --source clip.mp4
gesto detect static hands_one --version 2

# what have I trained?
gesto list

Python

import gesto

run = gesto.train("./gesto_projects/signs", region="hands_one", mode="static")
gesto.detect("static", "hands_one")

Or drive a model yourself:

from gesto.detect import Predictor

predictor = Predictor.load("static", "hands_one")
vector = predictor.features(holistic_result)   # extract + normalize
probs = predictor.predict(vector)

Where models go

Everything lands under one artifacts/ folder, split by mode then region. Training never overwrites an earlier run — it versions:

artifacts/
    static/
        hands_one/          model.keras, labels.json
        hands_one_2/        the next run
        pose/
    sequence/
        pose/
        pose_2/

gesto detect static pose picks the newest version; --version 1 picks a specific one.

Matching your capture

Predictions are only correct when detection feeds the model the same kind of vector it trained on. gesto mirrors Gesto Labeller exactly:

• the same engine — MediaPipe Holistic, same confidence settings
• the same landmark order — including one-hand mode preferring the right hand and falling back to the left
• the same normalization — translation/scale-invariant, verified identical
• the same mirroring — webcam frames are flipped, video files are not

If you captured with Gesto's Normalise unchecked, pass --raw when training so detection knows to skip it.

Getting good results

• Balance your classes. Similar sample counts per class; gesto applies class weights but balanced data is better.
• Enough samples. ~20–30 static frames per class, or ~15–30 sequences. Small datasets automatically get a lighter model, since an oversized network on little data overfits and collapses to predicting one class.
• Consistent clip length for sequence mode — set "Max frames" in Gesto Labeller so every capture is the same length.

Run gesto inspect <project> to check all of this before training.

Installation

pip install gesto

gesto uses MediaPipe's legacy solutions API, which was removed in MediaPipe 0.10.31. It also needs versions of TensorFlow, NumPy and protobuf that agree with that MediaPipe — newer TensorFlow (2.21+) and OpenCV (5.0) pull protobuf and NumPy in an incompatible direction. The package therefore pins a coherent, tested set:

Install into a fresh virtual environment so these don't clash with other projects:

python -m venv gesto_env
# Windows:  gesto_env\Scripts\activate
# macOS/Linux:  source gesto_env/bin/activate
pip install gesto

If you already hit dependency conflicts (e.g. you had TensorFlow 2.21 or OpenCV 5.0 installed), the cleanest fix is a fresh venv as above. To repair an existing environment, pin the set explicitly:

pip install "mediapipe==0.10.21" "tensorflow==2.17.1" "numpy==1.26.4" "protobuf==4.25.9" "opencv-python==4.11.0.86"

Roadmap

The legacy MediaPipe solutions API won't be maintained forever. A future release will move to MediaPipe's newer Tasks API (HandLandmarker, PoseLandmarker), which lifts the version ceiling. That API produces slightly different hand-landmark geometry, so models would need retraining — hence it's a deliberate, separate step rather than a drop-in change.

License

MIT