musicalgestures 1.6.9

Musical Gestures Toolbox for Python

MGT-python

The Musical Gestures Toolbox for Python is a collection of tools for visualizing and analysing audio and video files.

📖 Documentation & Examples

Quick Start

Installation

pip install musicalgestures

musicalgestures installs its core Python dependencies automatically. You still need a working ffmpeg installation on your system for video processing.

Basic Usage

import musicalgestures as mg

# Load a video (mp4, avi, mov, … all supported)
v = mg.MgVideo('dance.mp4')

# Create visualizations — call .show() to display the result
v.grid().show()
v.videograms().show()
v.average().show()
v.history().show()
v.heatmap().show()              # where the video changes most

# Motion analysis
v.motion().show()
v.motiontempo().show()          # dominant movement tempo (Hz/BPM)
v.motiondescriptors().show()    # motion energy, smoothness, entropy, spectral descriptors
v.eulerian(mode='motion').show()  # amplify subtle motion (EVM)

# Audio analysis
v.audio.waveform().show()
v.audio.spectrogram().show()
v.audio.mfcc().show()
v.audio.tempo().show()          # tempo + beat tracking
v.sonomotiongram().show()       # sonify the motiongram

# Pose estimation (MediaPipe is GPU-capable on the standard pip OpenCV)
v.pose(model='mediapipe').show()

Display happens via .show() — analysis methods return result objects (MgVideo/MgImage/MgFigure) and do not auto-render.

Runtime Notes

• import musicalgestures is fast: heavy dependencies are lazy-loaded, so the relevant backends load on first use rather than at import time.
• ffmpeg is required for video I/O and preprocessing.
• pose() defaults to the MediaPipe backend and downloads its weights on first use if they are missing; the OpenPose models ('body_25'/'coco'/'mpi') download their larger Caffe weights on first use instead.
• In notebooks and other non-interactive runs, missing pose weights are downloaded automatically when possible.
• If device='gpu' is requested but OpenCV CUDA support is unavailable, pose() falls back to CPU execution.
• flow.dense(), flow.sparse(), and blur_faces() use CPU by default (use_gpu=False). Set use_gpu=True to opt into CUDA acceleration with automatic CPU fallback.
• get_cuda_device_count() is available to quickly check whether OpenCV sees CUDA devices.
• blur_faces() returns the generated result object consistently, including when save_data=True.

Try Online

Quick Links

• Installation Guide
• Quick Start Tutorial
• API Reference
• Wiki & How-Tos
• Contributing

Features

• Video Analysis: Motion detection, optical flow, motion vectors, movement tempo, Eulerian Video Magnification, frame-rate/speed resampling (resample()), motion descriptors (energy/smoothness/entropy/spectral via motiondescriptors())
• Pose Estimation: MediaPipe (default; fast on plain CPU, GPU-capable) and OpenPose (multi-person) backends, with average-pose and trajectory summaries (per-marker quantity of motion + dominant frequency), optional marker motion trails, a 3D pose waterfall (pose_waterfall()), per-segment circular statistics (pose_segments()), centroid centring (pose_center()), and per-marker distance travelled (pose_distance())
• Audio–movement analysis: Compare a single performer's sound and motion — tempo similarity, phase synchrony, structural similarity, per-body-part audio coupling, and loudness/dynamics coupling
• Audio Processing: Waveforms, spectrograms, MFCC, chromagrams, tempo/beat tracking, spectral descriptors
• Visualizations: Motiongrams, videograms, motion history, heatmaps, sonomotiongrams (motion → sound)
• Space-time displays: Stroboscope (chronophotography), silhouette waterfall, Motion History Image, 3D space-time volume, combined motion SSM
• Integration: Works with NumPy, SciPy, librosa, and Matplotlib ecosystems
• Cross-platform: Linux, macOS, Windows support

Presentation

See this short video presentation made for the Nordic Sound and Music Computing Conference 2021:

Requirements

• Python 3.10+
• FFmpeg
• See installation guide for complete requirements

Research Background

This toolbox builds on the Musical Gestures Toolbox for Matlab, which again builds on the Musical Gestures Toolbox for Max. Many researchers and research assistants have helped its development over the years, including Balint Laczko, Joachim Poutaraud, Frida Furmyr, Marcus Widmer, Alexander Refsum Jensenius

The software is currently maintained by the fourMs lab at RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion at the University of Oslo.

Reference

If you use this toolbox in your research, please cite this article:

• Laczkó, B., & Jensenius, A. R. (2021). Reflections on the Development of the Musical Gestures Toolbox for Python. Proceedings of the Nordic Sound and Music Computing Conference, Copenhagen.

@inproceedings{laczkoReflectionsDevelopmentMusical2021,
    title = {Reflections on the Development of the Musical Gestures Toolbox for Python},
    author = {Laczkó, Bálint and Jensenius, Alexander Refsum},
    booktitle = {Proceedings of the Nordic Sound and Music Computing Conference},
    year = {2021},
    address = {Copenhagen},
    url = {http://urn.nb.no/URN:NBN:no-91935}
}

License

This toolbox is released under the GNU General Public License 3.0 license.