gesto

Train and run gesture recognition models from Gesto Labeller datasets.

Website · Docs · GitHub

pip install gesto

Point it at a project folder, pick a mode and a region, and it handles the rest — loading, training, versioned model storage, and live detection that matches how the data was captured.

Two modes

Static needs far less data (every captured frame is a training sample) and predicts instantly with no warm-up. Reach for sequence only when two gestures share the same shape and differ by movement.

Five regions

The region must match how the project was captured — gesto checks the feature dimension and tells you if it doesn't.

Command line

Two equivalent styles. Use whichever you like.

General — mode and region as arguments:

gesto train static hands_one ./gesto_projects/signs
gesto detect sequence pose --source clip.mp4
gesto image hands_one photo.jpg

Per-combination — one command per mode+region (there's one for each):

gesto train-static-legs ./gesto_projects/stances --epochs 250
gesto detect-sequence-pose --source clip.mp4
gesto image-static-hands-one photo.jpg

Detecting on camera vs video

--source takes a webcam index or a file path:

gesto detect static hands_one                       # default webcam (index 0)
gesto detect static hands_one --source 1            # second camera
gesto detect sequence pose --source walk.mp4        # a video file
gesto detect sequence pose --source C:\clips\run.avi

Classifying a single image (static models)

gesto image hands_one photo.jpg                     # opens a window with the result
gesto image hands_one photo.jpg --no-show           # just print the prediction
gesto image-static-pose posture.png --version 2     # a specific model version

Drawing landmarks

Landmarks are drawn on the frame by default. Turn them off with --no-draw:

gesto detect static hands_one                        # skeleton drawn (default)
gesto detect static hands_one --no-draw              # clean video, no skeleton
gesto image hands_one photo.jpg --no-draw

Training options

Epochs and other hyperparameters are adjustable on any train command:

gesto train sequence pose ./proj --epochs 400 --batch-size 32 --seq-len 30
gesto train-static-hands-one ./proj --epochs 150 --large   # force full model

Python

import gesto

run = gesto.train("./gesto_projects/signs", region="hands_one", mode="static")
gesto.detect("static", "hands_one")                       # camera
gesto.detect("sequence", "pose", source="clip.mp4")       # video

# classify a single image with a static model
from gesto.detect import predict_image
label, confidence, probs = predict_image("static", "hands_one", "photo.jpg",
                                         show=False, draw_landmarks=False)

Or drive a model yourself:

from gesto.detect import Predictor

predictor = Predictor.load("static", "hands_one")
vector = predictor.features(holistic_result)   # extract + normalize
probs = predictor.predict(vector)

Where models go

Everything lands under one artifacts/ folder, split by mode then region. Training never overwrites an earlier run — it versions:

artifacts/
    static/
        hands_one/          model.keras, labels.json
        hands_one_2/        the next run
        pose/
    sequence/
        pose/
        pose_2/

gesto detect static pose picks the newest version; --version 1 picks a specific one.

Matching your capture

Predictions are only correct when detection feeds the model the same kind of vector it trained on. gesto mirrors Gesto Labeller exactly:

• the same engine — MediaPipe Holistic, same confidence settings
• the same landmark order — including one-hand mode preferring the right hand and falling back to the left
• the same normalization — translation/scale-invariant, verified identical
• the same mirroring — webcam frames are flipped, video files are not

If you captured with Gesto's Normalise unchecked, pass --raw when training so detection knows to skip it.

Getting good results

• Balance your classes. Similar sample counts per class; gesto applies class weights but balanced data is better.
• Enough samples. ~20–30 static frames per class, or ~15–30 sequences. Small datasets automatically get a lighter model, since an oversized network on little data overfits and collapses to predicting one class.
• Consistent clip length for sequence mode — set "Max frames" in Gesto Labeller so every capture is the same length.

Run gesto inspect <project> to check all of this before training.

Installation

pip install gesto

gesto uses MediaPipe's legacy solutions API, which was removed in MediaPipe 0.10.31. It also needs versions of TensorFlow, NumPy and protobuf that agree with that MediaPipe — newer TensorFlow (2.21+) and OpenCV (5.0) pull protobuf and NumPy in an incompatible direction. The package therefore pins a coherent, tested set:

Install into a fresh virtual environment so these don't clash with other projects:

python -m venv gesto_env
# Windows:  gesto_env\Scripts\activate
# macOS/Linux:  source gesto_env/bin/activate
pip install gesto

If you already hit dependency conflicts (e.g. you had TensorFlow 2.21 or OpenCV 5.0 installed), the cleanest fix is a fresh venv as above. To repair an existing environment, pin the set explicitly:

pip install "mediapipe==0.10.21" "tensorflow==2.17.1" "numpy==1.26.4" "protobuf==4.25.9" "opencv-python==4.11.0.86"

Roadmap

The legacy MediaPipe solutions API won't be maintained forever. A future release will move to MediaPipe's newer Tasks API (HandLandmarker, PoseLandmarker), which lifts the version ceiling. That API produces slightly different hand-landmark geometry, so models would need retraining — hence it's a deliberate, separate step rather than a drop-in change.

License

MIT