GloSL

Sign language translation corpora in one pose format, with audited splits.

GloSL converts continuous sign language translation corpora into one format: MediaPipe Holistic keypoints extracted the same way for every corpus, one annotation schema, and one frozen train/dev/test partition per corpus whose overlap with training is measured and tagged clip by clip. This repository holds the glosl Python package, the scripts that built the GloSL v1 corpora, the baseline models and the paper. No data is committed: every corpus but one is published on the Hugging Face Hub, and every corpus can be rebuilt from its original release (docs/corpora.md).

Install

pip install glosl                  # load, check and score GloSL corpora
pip install "glosl[torch]"         # also PyTorch datasets and the multilingual training mix
pip install "glosl[extract]"       # also extract poses for a new corpus (MediaPipe 0.10.21, needs ffmpeg)

Quickstart

import glosl
from glosl.pytorch import PoseDataset

ds = glosl.load("pedroodb/glosl-swisstxt_weather")          # card, annotations, splits
test = PoseDataset("pedroodb/glosl-swisstxt_weather", "test")  # PyTorch dataset: pose [T, 543, 3], text, ...
print(test[0]["text"], test[0]["pose"].shape)

examples/quickstart.py goes one step further: a clip's pose, a padded training batch and scoring, in a minute.

Use a corpus

import glosl

ds = glosl.load("pedroodb/glosl-phoenix14t")    # a Hub dataset repo or a local folder
test = ds.split("test")                          # annotations: id, text, signer, fps, n_frames, ...
pose = ds.pose(test.id[0])                       # a pose_format.Pose (543 points, pixels)

report = glosl.evaluate(ds, hypotheses)          # hypotheses: {clip id: translation}
# BLEU and chrF with video-level bootstrap intervals, on all test clips and on unseen sentences

or from the command line: glosl score pedroodb/glosl-phoenix14t predictions.csv (columns id, hypothesis).

Train with PyTorch

from torch.utils.data import DataLoader
from glosl.pytorch import PoseDataset, Mix, BalancedBatchSampler, collate

repos = ["pedroodb/glosl-phoenix14t", "pedroodb/glosl-how2sign", "pedroodb/glosl-lsat"]
train = Mix([PoseDataset(r, "train", layout="unisign69") for r in repos])
sampler = BalancedBatchSampler(train, batch_size=32)      # the mix the GloSL paper trains with
loader = DataLoader(train, batch_sampler=sampler, collate_fn=collate, num_workers=4)
for epoch in range(epochs):
    sampler.set_epoch(epoch)
    for batch in loader:
        batch["pose"], batch["mask"], batch["text"]       # [B, T, V, 3], [B, T], list of str

test = PoseDataset("pedroodb/glosl-lsat", "test", unseen_only=True)   # the unseen-sentence test clips

A split's poses are downloaded once and cached. Each pose is [T, V, 3]: x and y divided by the frame size, and the confidence; missing points are 0. Layouts: "holistic" (all 543 MediaPipe points), "unisign69" (the 69 points the baselines use) or a list of point indices. BalancedBatchSampler draws sign languages in proportion to their hours^0.5, corpora within a sign language likewise, caps each corpus at two passes per epoch and batches clips of similar length; sampler.table() shows the resulting shares.

Add a corpus

glosl init mycorpus          # fill glosl.yaml and segments.csv
glosl extract mycorpus       # poses, annotations.csv, card.json, README.md
glosl check mycorpus         # format, pose audit, split diagnostics
glosl push mycorpus --repo you/glosl-mycorpus
glosl submit you/glosl-mycorpus

Step by step: docs/adding-a-corpus.md.

Layout

Benchmark splits (glosl-splits-v1)

Every dataset has one frozen train/dev/test partition, in splits/glosl_v1.csv of its Hub repo and in the split column of annotations.csv. card.json → split_policy.origin says who defined it:

• imported: the corpus authors' official split, kept as published.
• glosl: defined here, with the method and criteria recorded in the card.

Leakage is measured, not hidden: diagnostics/summary.json holds per-split statistics (verbatim overlap, OOV, n-gram overlap, signer and source-video overlap, oracle-retrieval BLEU), and diagnostics/clip_tags.csv tags every dev/test clip. Spec: docs/splits.md.

License

Code: MIT. Each dataset keeps its own licence, stated in its card.json.