terpsi

Dance corpora from single-dancer videos: body landmarks (no video kept), audio, subtitles, on-screen text, and movement-versus-music analysis. Named for Terpsichore, the muse of dance.

pip install "terpsi[all]"   # mediapipe + opencv, librosa, an OCR backend, yb (yt-dlp); ffmpeg on PATH
terpsi ingest https://youtu.be/q_TUyxUhoEw
terpsi clips

A clip goes in as a URL (YouTube, Wikimedia Commons, anything yt-dlp reads) or a local file, and comes out as a folder under ~/.local/share/terpsi/clips/<clip_id>/ holding what a dance model needs and nothing a video needs: 553 landmarks per frame (33 body + 2×21 hands + 478 face, with metric 3D world coordinates for body and hands, and 52 face blendshapes), the audio stream as the source carried it, the description and metadata, the subtitles, the text that was on screen with its timestamps, a beat grid, and a movement-versus-music report. The video is deleted once extracted.

Python

from terpsi import ingest, Corpus

clip_id = ingest("https://youtu.be/q_TUyxUhoEw")      # download, extract, delete the video
clip = Corpus()[clip_id]
lm = clip.landmarks            # LandmarkSeries: lm.pose (F,33,4), lm.pose_world (F,33,3), lm.hand_left, lm.face, ...
clip.beats["tempo"]            # 129.2
clip.screen_text[:2]           # [{'t_start': 3.0, 't_end': 6.0, 'text': 'Mise en place', ...}, ...]
clip.report["beat_alignment"]  # AIST++ Beat Alignment Score of the body's landings against the music beats

Each stage is a function with one seam, a keyword argument whose default works out of the box:

import terpsi needs only numpy, scipy and cw; the heavy dependencies are extras (pose, audio, ocr, acquire) imported on first use.

Choosing clips

terpsi search "house dance tutorial footwork" --n 15          # YouTube, Creative-Commons-filtered by default
terpsi screen https://youtu.be/XXXXXXXXXXX                      # single dancer? whole body? (a pose count on sampled frames)
terpsi ingest-many sources.txt                                  # one URL per line

screen reports the fraction of sampled frames with exactly one body, the fraction where that body is whole (nose and an ankle visible), and the licence yt-dlp found. The licence of record per clip is yt-dlp's license field, stored in the manifest.

The analysis

terpsi.analysis implements the literature's movement-versus-music measures over a LandmarkSeries and a beat grid: motion energy (summed joint speed), kinematic beats (its local minima, AIST++), the Beat Alignment Score in both directions, the phase concentration of the body's landings within the beat cycle (a resultant length), the motion tempo by autocorrelation and its ratio to the music tempo, and the audio-visual lag by onset cross-correlation. terpsi analyze <clip_id> recomputes a clip's report.

Data layout

~/.local/share/terpsi/clips/<clip_id>/
  manifest.json       id, title, source, licence, duration, fps, coverage, mirrored hint, files
  info.json           yt-dlp's metadata
  description.txt
  subtitles/*.vtt
  audio.m4a           the source audio stream (no re-encode)
  landmarks.npz/.json the LandmarkSeries (NaN where nothing was seen) and its meta
  screen_text.json    [{t_start, t_end, text, lines}]
  beats.json          tempo, beats, onset envelope
  report.json         the movement-vs-music report

$TERPSI_DATA_DIR moves the root. Models (.task files) are fetched once into <root>/models/.

Related

thoremin (the live instrument whose body nodes and pulse follower this corpus is for), paces (step documents from tutorial video), kodokan (pose analysis of judo video), yb, ocracy. Research notes: docs/research/.