islkit

Indian Sign Language (ISL) recognition from MediaPipe Holistic landmarks.

islkit turns camera frames into glosses. It covers every step: a landmark feature encoder, a dual-branch temporal convolutional network (TCN), a loader for the INCLUDE dataset, live inference with a confidence-gated decline, and a small HTTP/SSE recognition service. It runs on CPU and is built to work on small ARM boards as well as laptops.

camera → MediaPipe Holistic → RawFrame → encode_clip (T×352) → TCN → gloss | None

Install

pip install islkit            # inference and the recognition service
pip install "islkit[train]"   # + pandas, pyarrow, xgboost for INCLUDE and the baseline

Requires Python 3.12. mediapipe is pinned to 0.10.18, which also pins numpy<2. That is the last MediaPipe release that keeps mp.solutions and still runs on ARMv8.0-A (Cortex-A53) boards.

On macOS, XGBoost needs brew install libomp. Don't import torch and xgboost in the same process: each bundles its own libomp, and the process aborts with OMP: Error #15. import islkit keeps torch lazy for this reason.

Design

Features, not raw landmarks. Holistic's flattened output has 1,662 values, and 85% of them are face mesh. The encoder reduces each frame to 183 floats (352 with velocity):

The encoder uses three coordinate frames rather than one normalisation. Body frame: origin at the shoulder midpoint, scaled by shoulder width. Hand-local frame: origin at the wrist, scaled by the wrist-to-middle-MCP distance. Wrist position is recovered separately in the body frame. This keeps handshape separate from location.

Principles the code enforces:

• Hand slots are geometric. Slot 0 is the hand nearer the dominant-side shoulder. MediaPipe's handedness label flips under occlusion and is never read.
• Mask, never zero-fill. A missing hand is not a hand at the origin. A validity bit is carried per part and multiplied through the network.
• The label map is frozen. LabelMap is saved beside the weights. Class order rebuilt from a directory listing silently shifts indices.
• Checkpoints are tied to the encoder. A checkpoint records an encoder fingerprint, and SignRecogniser refuses to serve it through a different encoder.
• Store raw landmarks. Recordings hold raw MediaPipe output, so they can be re-encoded when the normalisation changes.
• The model can say "I don't know". Below the confidence threshold a prediction is None. For an accessibility device, silence is better than a confident wrong answer.

Usage

Encode a clip

from islkit import encode_clip
from islkit.infer import HolisticExtractor

frames = []
with HolisticExtractor() as extractor:
    for frame_bgr in video_frames:          # BGR numpy arrays, e.g. from cv2
        _, raw, _ = extractor.process(frame_bgr)
        frames.append(raw)

clip = encode_clip(frames, T=48)            # (48, 352) float32

Recognise a sign

from islkit import SignRecogniser

recogniser = SignRecogniser("classifier.pt", threshold=0.6)  # labels_*.json beside it
prediction = recogniser.classify(frames)
print(prediction.gloss, prediction.confidence, prediction.top3)

prediction.gloss is None when the model declines.

Train

from islkit import build_model, load_include
from islkit.model import fit

data = load_include("path/to/isl-mediapipe-holistic-landmarks")   # data.X: (N, 48, 352)
model = build_model(n_classes=len(data.label_map))
fit(model, data.X, data.y)

load_include reads the Kaggle indian-sign-language-mediapipe-holistic-landmarks dump of INCLUDE and caches the encoded arrays. That dump records no signer or session, so a class-stratified k-fold over whole clips is the only honest split available. It still leaks signers, so report numbers from it as optimistic. For your own recordings, use leave-one-session-out: build_dataset returns a sessions array for this.

replace_head and freeze_backbone support fine-tuning a pretrained backbone on a smaller vocabulary. Always report per-class F1 (per_class_f1), not just aggregate accuracy.

Run the recognition service

import threading
from islkit.infer import ClipStore, HolisticExtractor, SignRecogniser
from islkit.pipeline import CameraSource, RecognitionPipeline
from islkit.server import EventHub, make_server

recogniser = SignRecogniser("classifier.pt")
hub = EventHub()
pipeline = RecognitionPipeline(
    recogniser=recogniser,
    source_factory=lambda: CameraSource(0, "1280x720"),
    on_event=hub.publish,
    extractor_factory=HolisticExtractor,
    store=ClipStore(),
    on_pause=hub.clear_take,
)
threading.Thread(target=pipeline.run, daemon=True).start()
make_server(pipeline, hub).serve_forever()     # 127.0.0.1:9978

islkit.view.make_view_server serves an optional annotated MJPEG debug view on a separate port.

Pretrained weights are not shipped with the package.

Guides and experiments

• docs/: how the features, dataset, baseline, pretraining and fine-tuning work, each argued from committed results.
• experiments/: the scripts that produce those results, runnable from a clone of this repository.

Development

uv sync --extra train
uv run pytest
uv run ruff check .

The tests cover the properties that would silently break recognition: encoder invariances, mask gating, label-map and checkpoint round-trips, and the saved-clip layout.

License

MIT. See LICENSE.