Sign Language Toolkit (SLTK)

| Tutorials | Documentation | Notebooks | Contributing | PyPI |

What SLTK Offers..

• SLTK is an open-source Python toolkit that accelerates sign language research and makes SOTA computer vision tools and resources easily availble to linguists and other people working in Sign Language and AI.

• It provides a complete pipeline for pose extraction, sign segmentation, gloss spotting, non-manual signal detection, and evaluation — all with a single CLI or Python API.

• SLTK brings together state-of-the-art models (WiLoR, NLF, TEASER, SignRep) behind a unified interface, so researchers can focus on linguistics rather than engineering.

Vision

• We believe the field needs a holistic toolkit that jointly supports the full research workflow: video processing, pose extraction, temporal analysis, and standardized evaluation.

• SLTK is designed for reproducibility: such that every metric, extraction step, and analysis tool uses the same data structures and can be run from a single CLI command or Python script.

Quick Start

Installation

# Core library (ELAN I/O, CLI, dataset loaders) — no GPU needed
pip install signlangtk

# With GPU extraction backends
pip install "signlangtk[wilor,nlf,teaser]"

# Everything
pip install "signlangtk[all]"

Package name: signlangtk on PyPI, sltk for Python imports.

git clone https://github.com/ed-fish/Sign-Language-Toolkit.git
cd Sign-Language-Toolkit
pip install -e ".[dev,wilor,nlf,teaser]"
pytest  # ~947 tests

One-Command Pipeline

Example sample data ships in sample-data/: a 7s How2Sign continuous clip (sample-data/example.mp4) and 59 isolated ASL Citizen sign clips for the gloss-spotting demo (sample-data/spotting_dictionary/). See sample-data/README.md.

sltk pipeline video.mp4 -o output/
# → output/video.eaf  (10+ tiers: segmentation, blinks, head nods, mouth, gaze, ...)

Python API

from sltk.extraction.wilor import WiLoRExtractor
from sltk.extraction.teaser import TeaserExtractor
from sltk.segmentation.runner import get_runner
from sltk.segmentation.h5_loader import h5_to_features
from sltk.segmentation.postprocess import extract_segments
from sltk.nms.runner import detect_nms
from sltk.io.elan_roundtrip import ElanDocument

VIDEO, FPS = "recording.mp4", 25.0

# ── Step 1: Extract 3D hand poses ─────────────────────────────────
with WiLoRExtractor() as ext:
    ext.load_model()
    result = ext.extract_from_video(VIDEO, "recording_wilor.h5")
    print(f"{result.num_detections} hand detections across {result.num_frames} frames")

# ── Step 2: Extract face parameters ───────────────────────────────
with TeaserExtractor() as ext:
    ext.load_model()
    ext.extract_from_video(VIDEO, "recording_teaser.h5")

# ── Step 3: Segment signs ─────────────────────────────────────────
features = h5_to_features("recording_wilor.h5")  # (T, 192) MANO features
labels = get_runner().predict(features)            # 0=OUT, 1=IN, 2=BEGIN
segments = extract_segments(labels)                # [(start, end), ...]

# ── Step 4: Detect non-manual signals ─────────────────────────────
blinks, nms_events, quality = detect_nms(
    "recording_teaser.h5", detectors={"all"}
)

# ── Step 5: Assemble multi-tier ELAN file ─────────────────────────
doc = ElanDocument.new(video_path=VIDEO)

doc.add_tier("Segmentation")
for s, e in segments:
    doc.add_segment("Segmentation", s / FPS, e / FPS, "SIGN")

for tier in dict.fromkeys(["BLINK"] + [ev.tier for ev in nms_events]):
    doc.add_tier(tier)
for b in blinks:
    doc.add_segment("BLINK", b.start_frame / FPS, b.end_frame / FPS, "blink")
for ev in nms_events:
    doc.add_segment(ev.tier, ev.start_frame / FPS, ev.end_frame / FPS, ev.label)

doc.save("recording.eaf")

Extraction

Three GPU extractors share the same interface: load_model() → extract_from_video(). Weights auto-download from HuggingFace Hub on first use (~3.4 GB total).

WiLoR — 3D Hand Reconstruction

21 keypoints per hand with MANO rotation matrices. Primary input for sign segmentation.

from sltk.extraction.wilor import WiLoRExtractor, WiLoRConfig

config = WiLoRConfig(
    device="cuda:0",
    img_batch_size=128,       # reduce for <8GB VRAM
    detection_confidence=0.3,
    use_amp=True,
)
with WiLoRExtractor(config) as ext:
    ext.load_model()
    result = ext.extract_from_video("video.mp4", "video_wilor.h5")

Output H5:

video_wilor.h5
├── attrs: fps, num_frames, resolution
├── img_idx        (M,)              # frame index per detection
├── kpts_3d        (M, 21, 3)       # 3D hand keypoints
├── kpts_2d        (M, 21, 2)       # 2D projections
├── right          (M,)             # True = right hand
├── confidence     (M,)             # detection score
├── bboxes         (M, 4)           # hand bounding boxes
└── mano/
    ├── hand_pose      (M, 15, 3, 3)   # joint rotations
    ├── global_orient  (M, 1, 3, 3)    # wrist rotation
    └── betas          (M, 10)          # shape parameters

NLF — Full-Body SMPL-X

55 SMPL-X joints (body + hands + face) with full pose parameters.

from sltk.extraction.nlf import NLFExtractor, NLFConfig

with NLFExtractor(NLFConfig(device="cuda:0")) as ext:
    ext.load_model()
    result = ext.extract_from_video("video.mp4", "video_nlf.h5")

TEASER — FLAME Face Parameters

FLAME 3D face parameters: jaw pose, expression, eyelid, shape, and head pose. Uses MediaPipe for face detection. This is the input for NMS detection.

from sltk.extraction.teaser import TeaserExtractor, TeaserConfig

with TeaserExtractor(TeaserConfig(device="cuda:0")) as ext:
    ext.load_model()
    result = ext.extract_from_video("video.mp4", "video_teaser.h5")

Batch Processing

from pathlib import Path
from sltk.extraction.wilor import WiLoRExtractor

with WiLoRExtractor() as ext:
    ext.load_model()
    result = ext.extract_from_video("video.mp4", "hands.h5")
    print(f"{result.num_detections} hands across {result.num_frames} frames")

Sign Segmentation

A 4-layer Transformer that reads WiLoR hand features and predicts per-frame BIO labels.

# CLI
sltk segment video_wilor.h5 -o segments.eaf -f elan --video video.mp4

# Or directly from video (auto-extracts WiLoR first)
sltk segment video.mp4 -o segments.eaf -f elan

# Python
from sltk.segmentation.runner import segment_h5
from sltk.segmentation.output import OutputFormat

segment_h5("video_wilor.h5", output_path="segments.eaf",
           output_format=OutputFormat.ELAN, fps=25.0, media_path="video.mp4")

Index Detection

Classify each detected sign segment as index (pointing) or lexical using the Index Proposal Network (IPN): an SL-GCN encoder over SMPL-X body + WiLoR hand keypoints. Two released models, selected by sign language (bsl / dgs).

# CLI (add --eaf-out video.eaf to also write an "Index" ELAN tier)
sltk index video_nlf.h5 video_wilor.h5 -s segments.json -l bsl -o index.json

# Python
from sltk.indexing import get_index_detector

detector = get_index_detector(language="bsl")   # cached; loads the model once
pred = detector.predict(body, right, left)       # body (T,55,3), right/left (T,21,3)
pred.label, pred.probability                      # ("index", 0.97)

# Straight from SLTK per-video H5 files:
preds = detector.predict_segments("video_nlf.h5", "video_wilor.h5", segments=[(10, 34)])

Install with pip install "signlangtk[indexing]" (requires torch >= 2.9). See the Indexing API reference for details.

Gloss Spotting

Match detected segments to a dictionary of known signs using SignRep embeddings (768-dim ViT features).

from sltk.embedding.pipeline import SignRepPipeline

pipeline = SignRepPipeline()
continuous = pipeline.extract_continuous("video.mp4", stride=4)
dictionary = pipeline.load_dictionary(["/data/dictionaries/bsldict/signrep/"])

result = pipeline.spot(
    features=continuous,
    segments=[{"segment_id": 0, "start_frame": 12, "end_frame": 45}],
    dictionary=dictionary,
    top_k=5,
)

for seg in result.segments:
    for gl in seg.top_glosses[:3]:
        print(f"  {gl['gloss']} ({gl['similarity']:.3f})")

Non-Manual Signal Detection

Detect blinks, head nods/shakes/tilts, mouth movements, eyebrow raises, gaze direction, and eye squints from TEASER face data.

# CLI
sltk nms video_teaser.h5 -o nms_output/

from sltk.nms.runner import detect_nms

blinks, nms_events, quality = detect_nms(
    "video_teaser.h5",
    detectors={"all"},
    smpl_path="video_nlf.h5",  # optional: enables gaze detection
)
print(f"{len(blinks)} blinks, {len(nms_events)} NMS events")
print(f"Tracking quality: {quality.detection_rate:.0%}")

Evaluation Metrics

Standardized metrics for translation, production (pose & video), and segmentation evaluation. See the Metrics tutorial and notebook.

ROI Cropping

GPU-accelerated region-of-interest cropping for lips, hands, and other body regions. See the Cropping API.

from sltk.cropping import crop_lips_from_video

crops, frame_indices, bboxes = crop_lips_from_video("video.mp4", output_size=96)

Visualization

Skeleton and 3D mesh overlay rendering on video frames. See the Visualization API.

from sltk.visualization import generate_overlay_video
generate_overlay_video("video.mp4", "video_wilor.h5", "overlay.mp4", viz_type="wilor")

ELAN I/O

See the ELAN tutorial.

Supported Datasets

Built-in loaders for major sign language datasets:

Additional Features

• Pipeline CLI: Single command from raw video to multi-tier ELAN file (sltk pipeline)
• Unified Pose Format: PoseSequence class normalizes all backends to (T, N, C) arrays with format conversion
• ROI Cropping: GPU-accelerated lip, hand, and generic region cropping for data loading and preprocessing
• Visualization: Skeleton overlay and 3D mesh projection rendering
• Batch Processing: All extractors support directory-level batch processing
• Model Weight Management: Auto-download from HuggingFace Hub with environment variable overrides
• Modular Install: Optional dependency groups ([wilor], [metrics], [nlf], etc.) keep the base lightweight

CLI Reference

# Full pipeline
sltk pipeline video.mp4 -o output/

# Pose extraction
sltk extract wilor video.mp4 -o hands.h5
sltk extract nlf video.mp4 -o body.h5
sltk extract teaser video.mp4 -o face.h5

# Sign segmentation
sltk segment hands.h5 -o segments.eaf -f elan --video video.mp4

# Non-manual signal detection
sltk nms face.h5 -o nms_output/

# Index (pointing) vs lexical classification
sltk index body.h5 hands.h5 -s segments.json -l bsl -o index.json

# Gloss spotting
sltk spot video.mp4 --segments segments.json --dictionary dict/

# Evaluation
sltk evaluate preds.txt refs.txt --task translation -m bleu4 -m chrf
sltk evaluate pred.h5 ref.h5 --task production -m mpjpe -m pck
sltk evaluate preds.eaf refs.eaf --task segmentation

# ELAN utilities
sltk to-elan segments.json --video video.mp4 -o annotations.eaf
sltk from-elan annotations.eaf -o segments.json --tier Gloss
sltk info video_wilor.h5

See the full CLI documentation.

Tutorials & Notebooks

Model Weights

All weights auto-download from HuggingFace Hub and cache at ~/.cache/sltk/weights/.

⚠️ MANO, SMPL-X, and FLAME require registration on the authors' websites and cannot be auto-downloaded. See the Installation Guide for step-by-step setup instructions.

Set SLTK_AUTO_DOWNLOAD=1 to skip the confirmation prompt for HF models. Set SLTK_WEIGHTS_DIR to override the cache location.

Testing

pytest                       # Full suite (~947 tests)
pytest -m "not slow"         # Skip slow GPU/neural tests
pytest tests/test_metrics.py # Single module

Documentation

Full documentation at sign-language-toolkit.readthedocs.io

License

CC-BY-NC-ND-4.0 (Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International)

Third-Party Licenses

SLTK bundles or depends on models with their own license terms:

Citations & Acknowledgements

SLTK integrates several third-party models and methods. If you use these components in your research, please cite the original papers. SLTK does not claim authorship of these models — it provides a unified interface to run them together.

Sign Segmentation — He et al., FG 2025

The sign segmenter uses the Hands-On model for temporal sign boundary detection from hand pose features.

@inproceedings{he2025hands,
  title={Hands-On: Segmenting Individual Signs from Continuous Sequences},
  author={He, Low Jian and Walsh, Harry and Sincan, Ozge Mercanoglu and Bowden, Richard},
  booktitle={2025 IEEE 19th International Conference on Automatic Face and Gesture Recognition (FG)},
  pages={1--5},
  year={2025},
  organization={IEEE}
}

Gloss Spotting — Wong et al., ICCV 2025

SignRep provides self-supervised sign language representations used for dictionary-based gloss matching.

@inproceedings{wong2025signrep,
  title={SignRep: Enhancing Self-Supervised Sign Representations},
  author={Wong, Ryan and Camgoz, Necati Cihan and Bowden, Richard},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  pages={22804--22814},
  year={2025}
}

Face Reconstruction — Liu et al., ICLR 2025

TEASER provides FLAME-based face parameter extraction (jaw pose, expression, eyelid) used for non-manual signal detection.

@article{liu2025teaser,
  title={TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction},
  author={Liu, Yunfei and Zhu, Lei and Lin, Lijian and Zhu, Ye and Zhang, Ailing and Li, Yu},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2025}
}

Hand Reconstruction — Potamias et al., CVPR 2025

WiLoR provides end-to-end 3D hand localization and MANO parameter estimation from in-the-wild images.

@inproceedings{potamias2025wilor,
  title={WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild},
  author={Potamias, Rolandos Alexandros and Zhang, Jinglei and Deng, Jiankang and Zafeiriou, Stefanos},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2025}
}

Body Pose — Sárándi & Pons-Moll, NeurIPS 2024

NLF estimates continuous 3D human body pose and shape (SMPL-X parameters, including eye gaze).

@inproceedings{sarandi2024nlf,
  title={Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation},
  author={S{\'a}r{\'a}ndi, Istv{\'a}n and Pons-Moll, Gerard},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2024}
}

Pose Estimation — Jiang et al., 2023

RTMPose provides real-time multi-person whole-body keypoint detection (133 COCO-WholeBody landmarks).

@article{jiang2023rtmpose,
  title={RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose},
  author={Jiang, Tao and Lu, Peng and Zhang, Li and Ma, Ningsheng and Han, Rui and Lyu, Chengqi and Li, Yining and Chen, Kai},
  journal={arXiv preprint arXiv:2303.07399},
  year={2023}
}