SheetSage-Infer

Inference-only version of SheetSage for music transcription.

AI-powered music transcription system that converts audio to lead sheets (melody + chord symbols) using deep learning models.

Renamed from openmirlab-sheetsage-infer. Releases through 0.2.0 were published as openmirlab-sheetsage-infer on PyPI; that name is now deprecated and will not receive further releases. Starting with 0.2.1, this package publishes as sheetsage-infer -- update your pip install / uv add commands accordingly. The import name is unchanged (import sheetsage).

📌 Overview

SheetSage-Infer is an inference-only version of SheetSage for music transcription, optimized for easy deployment with vendored Jukebox modules.

✨ Features

• ✅ CPU & GPU Support - Handcrafted features (CPU) or Jukebox embeddings (GPU, via jukebox-infer)
• ✅ Multiple Export Formats - LilyPond notation, MIDI files, PDF generation
• ✅ Audio from URLs - Support for YouTube, Bandcamp, and other sources
• ✅ Simple API - High-level sheetsage() function

🚀 Quick Start

Installation

From PyPI:

# Using pip
pip install sheetsage-infer

# Using uv (recommended - faster)
uv pip install sheetsage-infer

# Or add to your project with uv
uv add sheetsage-infer

madmom was replaced by madmom-infer (our maintained, PyPI-published replacement) as of 0.2.1 -- plain pip install now works with no extra steps or git installs.

For Development:

git clone https://github.com/openmirlab/sheetsage-infer.git
cd sheetsage-infer
pip install -e ".[dev]"
# Or, with uv:
uv sync --extra dev

Prerequisites

• Python: ≥3.10 (tested on 3.10, 3.11, 3.12)
• LilyPond (optional, for PDF generation)
  • Linux: sudo apt-get install lilypond
  • macOS: brew install lilypond
  • Windows: Download from lilypond.org

Simple API (Recommended for Python)

from sheetsage.infer import sheetsage
from sheetsage.utils import engrave
from sheetsage.align import create_beat_to_time_fn

# Transcribe audio URL
lead_sheet, segment_beats, segment_beats_times = sheetsage(
    'https://example.com/audio.mp3',
    use_jukebox=False,           # Use fast CPU-based features
    segment_start_hint=30,       # Start at 30 seconds
    segment_end_hint=60,         # End at 60 seconds
    beats_per_minute_hint=120    # Hint for BPM (improves accuracy)
)

# Export to LilyPond
lily_code = lead_sheet.as_lily()
print(lily_code)

# Export to MIDI
beat_to_time_fn = create_beat_to_time_fn(segment_beats, segment_beats_times)
midi_bytes = lead_sheet.as_midi(beat_to_time_fn)

# Save MIDI file
with open('output.mid', 'wb') as f:
    f.write(midi_bytes)

# Generate PDF (requires LilyPond)
pdf_bytes = engrave(lily_code, out_format='pdf')
with open('leadsheet.pdf', 'wb') as f:
    f.write(pdf_bytes)

Using Jukebox Features (Higher Quality, GPU Required)

from sheetsage.infer import sheetsage

# Requires GPU with >=12GB VRAM
lead_sheet, beats, beat_times = sheetsage(
    'audio.mp3',
    use_jukebox=True,  # Use Jukebox embeddings (vendored)
    segment_start_hint=0,
    segment_end_hint=30,
    beats_per_minute_hint=100
)

Note: Jukebox features require GPU with ≥12GB VRAM. Vendored modules work without external installation.

Command-Line Interface

# Basic transcription
python -m sheetsage.infer audio.mp3

# With options
python -m sheetsage.infer audio.mp3 \
    --segment_start_hint 30 \
    --segment_end_hint 60 \
    --beats_per_minute_hint 120 \
    --output_dir ./output

# See all options
python -m sheetsage.infer --help

📋 Requirements

• Python: ≥3.10
• PyTorch: ≥2.0.0
• GPU: Optional, but recommended for Jukebox features (12GB+ VRAM)
• OS: Linux, macOS, Windows

⚡ Performance

Transcription speed depends on audio length and feature extraction method:

• Handcrafted features (CPU): ~1-5 seconds per minute of audio
• Jukebox features (GPU): ~30-60 seconds per minute of audio (requires GPU with ≥12GB VRAM)

Note: Performance depends on audio length, hardware, and feature extraction method. Jukebox features provide higher quality but are slower.

📚 Examples

See examples/ directory for usage examples:

• basic_transcription.py - Basic usage
• jukebox_transcription.py - GPU-based transcription
• hooktheory_example.py - Working with Hooktheory data

🏗️ Project Structure

sheetsage-infer/
├── sheetsage/                    # Main package
│   ├── infer.py                 # Main transcription pipeline (public sheetsage()/CLI)
│   ├── pipeline/                 # Pipeline enums/constants + step helpers (used by infer.py)
│   ├── align.py                 # Beat-to-time alignment
│   ├── beat_track.py             # Beat detection
│   ├── utils.py                 # LilyPond engraving, audio I/O
│   ├── assets.py                 # Asset management
│   ├── assets/                   # Asset JSON files
│   │   ├── hooktheory.json
│   │   ├── jukebox.json
│   │   ├── rwc.json
│   │   ├── sheetsage.json
│   │   └── test.json
│   ├── modules/                  # Neural network models
│   │   └── modules.py            # Transformer architectures
│   ├── representations/          # Feature extractors
│   │   ├── handcrafted.py       # CPU-based mel-spectrograms
│   │   └── jukebox.py            # Jukebox embedding interface (imports jukebox-infer)
│   └── theory/                   # Music theory classes
│       ├── lead_sheet.py         # LeadSheet class with export methods
│       ├── basic.py              # Basic music theory primitives
│       ├── internal.py           # Internal theory classes
│       ├── theorytab.py          # TheoryTab integration
│       └── utils.py              # Theory utilities
├── tests/                        # Import smoke tests + env-guarded regression fixtures
├── examples/                     # Example scripts
│   ├── basic_transcription.py    # Basic usage
│   ├── jukebox_transcription.py  # GPU-based transcription
│   ├── hooktheory_example.py     # Hooktheory data examples
│   ├── hooktheory_simple.py     # Simple Hooktheory example
│   └── transcribe_hooktheory_segments.py  # Hooktheory segment transcription
├── hooktheory_data/              # Test data
│   ├── Hooktheory_Test_MIDI.tar.gz
│   └── Hooktheory_Test_Segments.json
├── .github/                      # GitHub configuration
│   └── workflows/
│       └── publish.yml           # PyPI publishing workflow (runs tests before build)
├── pyproject.toml               # Project configuration (single source of truth for deps)
├── uv.lock                      # UV lock file
├── CHANGELOG.md                 # Notable changes
├── CLAUDE.md                    # Orientation for AI coding agents working in this repo
├── LICENSE                      # MIT License (code)
├── NOTICE                       # License layering: code (MIT) vs weights/data (CC BY-NC-SA)
└── README.md                    # This file

🔄 Changes from Original SheetSage

SheetSage-Infer has been modified from the original SheetSage to make it more suitable for library use and easier to maintain.

Key Improvements

What We Maintain

• ✅ All core transcription functionality
• ✅ Same neural network models
• ✅ Same output formats (LeadSheet, LilyPond, MIDI)
• ✅ Same API interface for sheetsage() function
• ✅ Same theory classes (Note, Chord, Melody, Harmony, etc.)

What We Changed

• Jukebox via jukebox-infer: Uses the published, org-maintained jukebox-infer package instead of vendoring a copy of the Jukebox codebase
• Library-First Design: Optimized for pip install and programmatic use
• Better Dependency Management: Explicit version pins and compatibility

🙏 Acknowledgments

Original Research by Chris Donahue

SheetSage-Infer is built upon the excellent work of SheetSage by Chris Donahue. The original SheetSage represents a major advancement in music transcription, achieving state-of-the-art results through hierarchical transformer architectures.

Research Paper

SheetSage: A Hierarchical Transformer for Audio to Lead Sheet Transcription

This work introduced hierarchical music transcription with melody and harmony extraction, enabling high-quality lead sheet generation from audio.

Original Author

• Chris Donahue - Original SheetSage creator

About This Implementation

This package was created to continue the excellent work by providing easier deployment (via pip-installable dependencies, including jukebox-infer for Jukebox features), while preserving 100% of the original model quality and algorithms.

What we maintain:

• PyTorch 2.0+ compatibility
• Modern dependency management
• Inference-only packaging

What remains unchanged:

• All model architectures (100% original)
• All transcription algorithms (100% original)
• All model weights (100% original)
• All output formats (100% original)

📄 Citation

Please cite using the following bibtex entry:

@inproceedings{donahue2024sheetsage,
  title={SheetSage: A Hierarchical Transformer for Audio to Lead Sheet Transcription},
  author={Donahue, Chris},
  booktitle={ISMIR},
  year={2024}
}

If you use SheetSage-Infer in your research, please cite the original SheetSage paper above. This package is a maintenance fork to ensure easier deployment and continued compatibility - all credit for the models, algorithms, and research belongs to the original author.

📄 License

Licensing is two-tier — see NOTICE for the full breakdown:

• Code (this repository, including code adapted from SheetSage): MIT License. Copyright (c) 2022 Chris Donahue (Original SheetSage); Copyright (c) 2025 SheetSage-Infer contributors. See LICENSE for details.
• Weights and data downloaded at runtime via sheetsage.assets (trained model checkpoints, HookTheory-derived segments/MIDI) are CC BY-NC-SA 3.0, since they derive from user contributions on HookTheory. madmom's bundled DBN downbeat-tracking model is similarly CC BY-NC-SA (separate from madmom's own BSD-2-Clause source code). These are fetched on demand, not bundled in this package's source distribution or wheel.

⚠️ Limitations

• Inference only - No training capabilities
• Jukebox features require GPU - 12GB+ VRAM recommended for Jukebox embeddings
• Jukebox features require ≥60s of (reported) total audio length - the 5B prior's conditioning asserts total length is within [60s, 600s); this is an inherent Jukebox architecture constraint (not something this project's code controls), so very short clips will raise an AssertionError on the use_jukebox=True path even though the CPU (handcrafted-features) path has no such floor
• LilyPond required for PDF - Optional dependency for PDF generation
• Time signatures - Currently supports 4/4 and 3/4 only
• Audio length - Best results with segments 30-300 seconds

🤝 Contributing

We welcome contributions! Please:

1. Follow the code style (ruff/black)
2. Add tests for new features
3. Submit PRs with clear descriptions

Development Setup

# Install dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/

# Format and lint code
ruff format . && ruff check .

📞 Support

For issues and questions:

• GitHub Issues: github.com/openmirlab/sheetsage-infer/issues
• Examples: examples/ directory

🔗 Links

• Original SheetSage: https://github.com/chrisdonahue/sheetsage
• This Repository: https://github.com/openmirlab/sheetsage-infer
• PyPI Package: https://pypi.org/project/sheetsage-infer/ (previously openmirlab-sheetsage-infer, deprecated as of 0.2.1)

Made with ❤️ for the ML community

Based on the excellent work by Chris Donahue and the SheetSage project.