all-in-one-infer 3.0.1

All-In-One Music Structure Analyzer with source separation

All-In-One-Infer Music Structure Analyzer

An enhanced version of All-In-One with integrated source separation and modern PyTorch compatibility

📦 Available on PyPI: https://pypi.org/project/all-in-one-infer/

📢 Officially renamed: all-in-one-fix → all-in-one-infer

This project has officially moved from all-in-one-fix to all-in-one-infer (as of v3.0.0, 2026-07).

• Install: pip install all-in-one-infer (the old all-in-one-fix PyPI package will no longer receive updates)
• Import: import allin1_infer (formerly allin1fix)
• CLI: all-in-one-infer (formerly allin1fix)

Existing all-in-one-fix users: see the Migration section below.

🙏 Acknowledgments:

This package builds upon the exceptional work of the foundational project:

• All-In-One by Taejun Kim and Juhan Nam - The core music structure analysis algorithms and models. We are deeply grateful for their groundbreaking research in music information retrieval.

This enhanced version preserves all original research contributions while improving compatibility and workflow flexibility. All credit for the core algorithms belongs to the original authors.

This package provides models for music structure analysis, predicting:

1. Tempo (BPM)
2. Beats
3. Downbeats
4. Functional segment boundaries
5. Functional segment labels (e.g., intro, verse, chorus, bridge, outro)

🆕 What's New in v2.0.0 (historical)

🎵 Integrated Source Separation

• Source Separation: Uses demucs-infer package for high-quality source separation
• Clean Dependencies: Inference-only Demucs integration via demucs-infer package
• Model Caching: Intelligent model caching for improved performance (6x faster on repeated use)
• GPU Memory Management: Automatic GPU cleanup prevents out-of-memory errors
• Better Error Messages: Fuzzy matching suggestions for model names

🔧 Enhanced Compatibility

• PyTorch 2.x Support: Compatible with PyTorch 2.0 through 2.7+ and CUDA 11.7-12.8
• NATTEN 0.17.x Verified: Fully tested and working with PyTorch 2.0-2.7+
  • Automatic version detection supports NATTEN 0.17.x-0.19.x
  • Extensively tested with real music analysis workloads
  • Note: NATTEN 0.20+ is not compatible due to API changes requiring dimensional validation updates
• Unified Package: Single package with all functionality included
• Modern Packaging: UV-style packaging with full pip compatibility

🎛️ Flexible Source Separation

• Custom Models: Integrate your own source separation models via pluggable architecture
• Pre-computed Stems: Use existing separated stems from any source separation tool
• Direct Stems Input: Skip source separation entirely by providing stems directly
• Hybrid Workflows: Mix custom separation, pre-computed stems, and default separation

🗂️ Cache Management

• View Cache: all-in-one-infer --cache-info to see cached separation models
• Clear Cache: all-in-one-infer --clear-cache to free up disk space
• Python API: allin1_infer.print_cache_info(), allin1_infer.clear_model_cache()

📦 Enhanced CLI & API

• Backward Compatible: All original functionality preserved with allin1_infer namespace
• Rich CLI Options: New stems handling and cache management options
• Python API: Enhanced analyze function with new stem provider system

---

Table of Contents

• Motivation & Changes
• Installation
• Usage for CLI
  • New Stems Features
  • Cache Management
  • Technical Improvements
• Usage for Python
  • New Stems API Features
• Visualization & Sonification
• Available Models
• Speed
• Advanced Usage for Research
• Concerning MP3 Files
• Migration from All-In-One
• Citation
• About All-In-One-Infer
• Documentation

💡 Motivation & Changes

Why This Fork?

The original All-In-One package is an excellent music structure analysis tool, but needed updates for modern PyTorch environments:

1. PyTorch 2.x Compatibility: NATTEN library needed upgrade for PyTorch 2.x
2. Source Separation: Required separate source separation setup
3. Performance: No model caching, repeated model loading
4. Modern Tooling: Packaging and dependency management improvements

Note: This fork uses demucs-infer, a maintained inference-only package with PyTorch 2.x support for source separation.

What Changed in v2.0.0?

This fork addresses these issues through strategic integration and improvements:

1. NATTEN Dependency Removed (Pure-PyTorch Neighborhood Attention) 🔧

Before:

# Original All-In-One required NATTEN, a compiled CUDA/C++ extension that must
# be built against the exact installed torch version at install time.
dependencies = ["natten>=0.15.0"]

After:

# No natten needed — neighborhood attention is implemented in pure PyTorch.
dependencies = ["torch>=2.0.0"]  # no upper bound, no compiled extension

Changes Made:

• Reimplemented NATTEN's neighborhood attention (1D + 2D, with relative positional biases) in pure PyTorch: src/allin1_infer/models/neighborhood_attention.py
• Numerically identical to NATTEN 0.17.5 (verified by golden-fixture tests, forward and backward)
• Pretrained checkpoints load unchanged — no weight conversion
• If a compatible NATTEN (0.17.x-0.19.x) is installed, it is used automatically as a faster fused-kernel backend: pip install "all-in-one-infer[natten]" — see the Installation section's "Optional: NATTEN fused kernels" for the required torch pin and --no-build-isolation steps

Impact: Installs anywhere with a single pip install — any torch >= 2.0 (including 2.8+), CPU-only machines, and platforms NATTEN never supported (e.g. macOS / Apple Silicon). No C++ toolchain required.

2. Streamlined Source Separation 🎵

Before:

# Required external demucs package (no longer maintained)
dependencies = ["demucs"]  # PyTorch 1.x only, not actively maintained

After:

# Uses demucs-infer (maintained, PyTorch 2.x compatible)
dependencies = ["demucs-infer"]

Changes Made:

• Switched to demucs-infer (maintained fork of Demucs for inference)
• PyTorch 2.x compatibility (no torchaudio<2.1 restriction)
• Added intelligent model caching for 6x performance improvement
• Implemented automatic GPU memory cleanup
• Enhanced error messages with model name suggestions

Impact: Actively maintained dependencies, faster processing, modern PyTorch support

3. Enhanced Cache Management 🗂️

Added Features:

• View cached models: all-in-one-infer --cache-info
• Clear cache: all-in-one-infer --clear-cache (with --clear-cache-dry-run preview)
• Python API: get_cache_size(), list_cached_models(), clear_model_cache()
• Tracks model files (.th, .pth) in ~/.cache/torch/hub/checkpoints/

Impact: Better disk space management, visibility into cached models

4. Documentation & Code Cleanup 📝

Changes:

• Updated to use demucs-infer package instead of embedded code
• Added proper acknowledgments to both All-In-One and Demucs projects
• Clarified integration source and original authorship
• Improved docstrings and code comments

Impact: Clear attribution, easier maintenance, better developer experience

5. Modern Packaging with UV 📦

Before:

# Traditional setup.py or older pyproject.toml

After:

# Modern pyproject.toml with UV support
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Changes Made:

• Converted to modern UV-style packaging using pyproject.toml
• Uses hatchling as build backend
• Maintains full pip compatibility - works with both uv and pip
• Follows PEP 621 for project metadata

Installation Methods:

# With UV (recommended, faster)
uv pip install all-in-one-infer

# With traditional pip (still supported)
pip install all-in-one-infer

# Editable install for development
uv pip install -e .
pip install -e .

Impact: Faster dependency resolution with UV, while maintaining compatibility with traditional pip workflows

Respect for Original Work

This project integrates two foundational open-source projects:

Original Projects:

• All-In-One - Music structure analysis by Taejun Kim & Juhan Nam
• demucs-infer - Source separation package (PyTorch 2.x compatible)

What's New in v2.0.0:

• ✅ NATTEN removed — pure-PyTorch neighborhood attention (any PyTorch >= 2.0, no compilation)
• ✅ demucs-infer integration for source separation
• ✅ Performance improvements (6x faster with model caching)
• ✅ Enhanced error handling and cache management
• ✅ Modern packaging with UV support
• ✅ 100% backward compatible with All-In-One API

What Stayed the Same:

• ✅ All-In-One model architectures (unchanged)
• ✅ Beat/downbeat tracking algorithms (unchanged)
• ✅ Tempo estimation (unchanged)
• ✅ Structure segmentation (unchanged)
• ✅ Research quality and accuracy (unchanged)

Credit:

• All-In-One research → Taejun Kim & Juhan Nam (original paper)
• Source separation → demucs-infer package (openmirlab/demucs-infer)
• This fork → PyTorch 2.x compatibility, performance improvements, modern tooling

📦 Installation

📦 Available on PyPI: https://pypi.org/project/all-in-one-infer/

Quick Install from PyPI 🚀

pip install all-in-one-infer

# madmom is required at runtime; if it wasn't installed automatically:
pip install git+https://github.com/CPJKU/madmom

Or if you prefer UV (faster):

uv add all-in-one-infer
uv add git+https://github.com/CPJKU/madmom

That's it — no --no-build-isolation, no "install torch first". Since NATTEN is no longer a dependency, there is nothing to compile at install time.

Requirements

• Python: 3.9 or later (required for scipy>=1.13 and madmom)
• PyTorch: 2.0.0 or later (no upper bound)
• OS: Linux, macOS, Windows

GPU Support (Optional)

For GPU acceleration, install PyTorch with CUDA support:

# Example: CUDA 12.1
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install all-in-one-infer

Optional: NATTEN fused kernels

Neighborhood attention runs on a built-in pure-PyTorch implementation that is numerically identical to NATTEN. If you want NATTEN's fused CUDA kernels as a speed optimization (Linux + CUDA + torch < 2.8 only, compiles at install time):

pip install torch"<2.8"
pip install "all-in-one-infer[natten]" --no-build-isolation

It is picked up automatically when importable; otherwise the pure-PyTorch backend is used. Results are identical either way.

✅ Verify Installation

After installation, verify it worked:

# Check if installed
python -c "import allin1_infer; print('✅ allin1_infer installed successfully!')"

# Check version
python -c "import allin1_infer; print(allin1_infer.__version__)"

# Test CLI
all-in-one-infer --help

Troubleshooting

Installation fails with scipy version error

• ✅ Cause: Using Python < 3.9
• ✅ Solution: Ensure Python 3.9+ is used

ImportError: No module named 'madmom'

• ✅ Cause: madmom must be installed separately from git (PyPI limitation)
• ✅ Solution: Run pip install git+https://github.com/CPJKU/madmom before using all-in-one-infer

Installation fails with madmom error

• ✅ Cause: Installing madmom from GitHub requires git and internet
• ✅ Solution: Ensure git is installed (apt install git or brew install git) and you have internet access

---

Installation from GitHub (Development)

If you want to install the latest development version from GitHub:

Using UV (Recommended):

# Install UV if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install from GitHub
uv pip install git+https://github.com/openmirlab/all-in-one-infer.git

Using pip:

pip install git+https://github.com/openmirlab/all-in-one-infer.git

Development Installation (Editable):

git clone https://github.com/openmirlab/all-in-one-infer.git
cd all-in-one-infer
pip install -e .

Note: All PyPI dependencies (including demucs-infer) are installed automatically. madmom still needs to come from git if it isn't picked up automatically: pip install git+https://github.com/CPJKU/madmom.

(Optional) Install FFmpeg for MP3 support

For Ubuntu:

sudo apt install ffmpeg

For macOS:

brew install ffmpeg

Usage for CLI

Basic Usage

To analyze audio files:

all-in-one-infer your_audio_file1.wav your_audio_file2.mp3

🎛️ New Stems Features

1. Direct stems input from directory:

all-in-one-infer --stems-from-dir ./my_stems --stems-id "my_song" -o ./results
# Expects: ./my_stems/{bass,drums,other,vocals}.wav

2. Custom stem filename patterns:

all-in-one-infer --stems-from-dir ./stems --stems-pattern "track_{stem}.wav" -o ./results
# Expects: ./stems/track_{bass,drums,other,vocals}.wav

3. Individual stem files:

all-in-one-infer \
  --stems-bass path/to/bass.wav \
  --stems-drums path/to/drums.wav \
  --stems-other path/to/other.wav \
  --stems-vocals path/to/vocals.wav \
  --stems-id "my_track" -o ./results

4. Pre-computed stems mapping:

# Create stems_mapping.json:
{
  "song1.wav": "/path/to/song1_stems/",
  "song2.wav": "/path/to/song2_stems/"
}

all-in-one-infer song1.wav song2.wav --stems-dict stems_mapping.json -o ./results

5. Skip separation (use existing stems in demix-dir):

all-in-one-infer track.wav --skip-separation --demix-dir ./existing_stems -o ./results

Results will be saved in the ./struct directory by default:

./struct
└── your_audio_file1.json
└── your_audio_file2.json

The analysis results will be saved in JSON format:

{
  "path": "/path/to/your_audio_file.wav",
  "bpm": 100,
  "beats": [ 0.33, 0.75, 1.14, ... ],
  "downbeats": [ 0.33, 1.94, 3.53, ... ],
  "beat_positions": [ 1, 2, 3, 4, 1, 2, 3, 4, 1, ... ],
  "segments": [
    {
      "start": 0.0,
      "end": 0.33,
      "label": "start"
    },
    {
      "start": 0.33,
      "end": 13.13,
      "label": "intro"
    },
    {
      "start": 13.13,
      "end": 37.53,
      "label": "chorus"
    },
    {
      "start": 37.53,
      "end": 51.53,
      "label": "verse"
    },
    ...
  ]
}

🗂️ Cache Management

Separation models are downloaded to ~/.cache/torch/hub/checkpoints/ and can use several GB of disk space.

View cache information:

all-in-one-infer --cache-info

Output:

============================================================
Model Cache Information
============================================================
Cache directory: /home/user/.cache/torch/hub/checkpoints
Total size: 3.19 GB
Number of models: 23

Cached models:
------------------------------------------------------------
  7fd6ef75-a905dd85.th                        37.61 MB  2025-09-08 12:20:50
  14fc6a69-a89dd0ee.th                        36.71 MB  2025-09-08 12:20:46
  ...
============================================================

Preview what would be deleted (dry run):

all-in-one-infer --clear-cache-dry-run

Clear all cached models:

all-in-one-infer --clear-cache

Python API:

import allin1_infer

# View cache info
allin1_infer.print_cache_info()

# Get cache size
size_gb = allin1_infer.get_cache_size()

# List models
models = allin1_infer.list_cached_models()

# Clear cache (dry run first!)
count = allin1_infer.clear_model_cache(dry_run=True)
count = allin1_infer.clear_model_cache()  # Actually delete

🔧 Technical Improvements

All-In-One-Infer includes several technical enhancements over the original:

• Modern PyTorch Support: Compatible with PyTorch 2.x and CUDA 12.x
• Pure-PyTorch Neighborhood Attention: NATTEN is no longer required — a numerically identical pure-PyTorch implementation ships by default; NATTEN 0.17.x can optionally be installed via the [natten] extra as a faster fused-kernel backend
• Source Separation: Uses demucs-infer package with model caching and GPU cleanup
• Memory Optimization: Automatic GPU memory cleanup prevents OOM errors on batch processing
• Performance: 6x faster on repeated use with intelligent model caching
• Error Handling: Better error messages with fuzzy matching and helpful suggestions
• Modular Architecture: Clean separation of concerns for easier maintenance and extension
• Cache Management: Built-in tools to view and manage cached separation models

📋 All Available CLI Options

$ all-in-one-infer -h

usage: all-in-one-infer [-h] [-o OUT_DIR] [-v] [--viz-dir VIZ_DIR] [-s]
                 [--sonif-dir SONIF_DIR] [-a] [-e] [-m MODEL] [-d DEVICE] [-k]
                 [--demix-dir DEMIX_DIR] [--spec-dir SPEC_DIR] [--overwrite]
                 [--no-multiprocess] [--compile-model] [--demucs-overlap DEMUCS_OVERLAP]
                 [--demucs-fp16] [--stems-dict STEMS_DICT]
                 [--stems-dir STEMS_DIR] [--skip-separation] [--no-demucs]
                 [--stems-bass STEMS_BASS] [--stems-drums STEMS_DRUMS]
                 [--stems-other STEMS_OTHER] [--stems-vocals STEMS_VOCALS]
                 [--stems-from-dir STEMS_FROM_DIR]
                 [--stems-pattern STEMS_PATTERN] [--stems-id STEMS_ID]
                 [paths ...]

positional arguments:
  paths                 Path to tracks (for single track mode) or omit for
                        stems input mode

Core Options:
  -o, --out-dir         Path to store analysis results (default: ./struct)
  -v, --visualize       Save visualizations (default: False)
  -s, --sonify          Save sonifications (default: False)
  -m, --model          Model to use (default: harmonix-all)
  -d, --device         Device to use (default: cuda if available else cpu)
  -k, --keep-byproducts Keep demixed audio and spectrograms (default: False)

Performance Options (opt-in, experimental):
  --compile-model       torch.compile the model(s) (reduce-overhead mode).
                        ~57s one-time compile cost, ~38% faster steady-state
                        forward -- only worth it for long-lived/batch
                        processing (default: False)
  --demucs-overlap DEMUCS_OVERLAP
                        Accuracy-affecting: overlap fraction for demucs
                        separation chunks. Default 0.25 matches demucs' own
                        default (unchanged behavior); other values can shift
                        segment boundaries
  --demucs-fp16         Accuracy-affecting: run demucs separation under
                        fp16 autocast (CUDA only) for faster separation
                        (default: False)

Stems Input Options:
  --stems-dict         JSON file mapping audio paths to stem directories
  --stems-from-dir     Directory containing bass.wav, drums.wav, other.wav, vocals.wav
  --stems-pattern      Pattern for stem files (e.g. "song_{stem}.wav")
  --stems-bass         Path to bass stem file
  --stems-drums        Path to drums stem file  
  --stems-other        Path to other stem file
  --stems-vocals       Path to vocals stem file
  --stems-id           Identifier for the stem set
  --skip-separation    Skip source separation, use existing stems

Usage for Python

Basic Usage

from allin1_infer import analyze

# Analyze audio files (uses demucs-infer for separation)
results = analyze(['song1.wav', 'song2.mp3'])

🎛️ New Stems API Features

1. Custom separation models:

from allin1_infer import analyze, CustomSeparatorProvider

class MyCustomSeparator:
    def __init__(self, model_path):
        self.model = load_my_model(model_path)
    
    def separate(self, audio_path, output_dir, device):
        # Your separation logic here
        # Must return Path to directory with bass.wav, drums.wav, other.wav, vocals.wav
        stems_dir = output_dir / 'my_model' / audio_path.stem
        stems_dir.mkdir(parents=True, exist_ok=True)
        
        # Your model inference
        stems = self.model.separate(audio_path, device)
        
        # Save stems
        for stem_name, audio_data in stems.items():
            save_audio(audio_data, stems_dir / f"{stem_name}.wav")
        
        return stems_dir

# Use your custom model
separator = MyCustomSeparator("path/to/model.pth")
provider = CustomSeparatorProvider(separator)
results = analyze(['song.wav'], stem_provider=provider)

2. Pre-computed stems:

from allin1_infer import analyze, PrecomputedStemProvider

# Use stems from any source separation tool
stems_mapping = {
    'song1.wav': '/path/to/spleeter_output/song1/',
    'song2.wav': '/path/to/mdx_output/song2/',
    'song3.wav': '/path/to/custom_stems/song3/'
}
provider = PrecomputedStemProvider(stems_mapping)
results = analyze(['song1.wav', 'song2.wav', 'song3.wav'], stem_provider=provider)

3. Direct stems input:

from allin1_infer import analyze, StemsInput, create_stems_input_from_directory

# Method 1: Manual specification
stems = StemsInput(
    bass='path/to/bass.wav',
    drums='path/to/drums.wav', 
    other='path/to/other.wav',
    vocals='path/to/vocals.wav',
    identifier='my_song'
)

# Method 2: From directory (expects bass.wav, drums.wav, other.wav, vocals.wav)
stems = create_stems_input_from_directory('/path/to/stems_folder')

# Method 3: Multiple tracks with different stems
stems_list = [
    create_stems_input_from_directory('/path/to/song1_stems'),
    create_stems_input_from_directory('/path/to/song2_stems')
]

results = analyze(stems_input=stems_list)

4. Hybrid workflows:

# Mix different approaches in the same analysis
from allin1_infer import analyze, PrecomputedStemProvider, StemsInput

# Some tracks have pre-computed stems
stems_mapping = {'song1.wav': '/path/to/stems/'}
provider = PrecomputedStemProvider(stems_mapping)

# Other tracks use default separation  
regular_tracks = ['song2.wav', 'song3.wav']

# Process each group
results1 = analyze(['song1.wav'], stem_provider=provider)
results2 = analyze(regular_tracks)  # Uses default HTDemucs

Available functions:

• analyze()
• load_result()
• visualize()
• sonify()

analyze()

Analyzes the provided audio files and returns the analysis results.

import allin1_infer

# You can analyze a single file:
result = allin1_infer.analyze('your_audio_file.wav')

# Or multiple files:
results = allin1_infer.analyze(['your_audio_file1.wav', 'your_audio_file2.mp3'])

A result is a dataclass instance containing:

AnalysisResult(
  path='/path/to/your_audio_file.wav', 
  bpm=100,
  beats=[0.33, 0.75, 1.14, ...],
  beat_positions=[1, 2, 3, 4, 1, 2, 3, 4, 1, ...],
  downbeats=[0.33, 1.94, 3.53, ...], 
  segments=[
    Segment(start=0.0, end=0.33, label='start'), 
    Segment(start=0.33, end=13.13, label='intro'), 
    Segment(start=13.13, end=37.53, label='chorus'), 
    Segment(start=37.53, end=51.53, label='verse'), 
    Segment(start=51.53, end=64.34, label='verse'), 
    Segment(start=64.34, end=89.93, label='chorus'), 
    Segment(start=89.93, end=105.93, label='bridge'), 
    Segment(start=105.93, end=134.74, label='chorus'), 
    Segment(start=134.74, end=153.95, label='chorus'), 
    Segment(start=153.95, end=154.67, label='end'),
  ]),

Unlike CLI, it does not save the results to disk by default. You can save them as follows:

result = allin1_infer.analyze(
  'your_audio_file.wav',
  out_dir='./struct',
)

Parameters:

• paths : Union[PathLike, List[PathLike]]
  List of paths or a single path to the audio files to be analyzed.

• out_dir : PathLike (optional)
  Path to the directory where the analysis results will be saved. By default, the results will not be saved.

• visualize : Union[bool, PathLike] (optional)
  Whether to visualize the analysis results or not. If a path is provided, the visualizations will be saved in that directory. Default is False. If True, the visualizations will be saved in './viz'.

• sonify : Union[bool, PathLike] (optional)
  Whether to sonify the analysis results or not. If a path is provided, the sonifications will be saved in that directory. Default is False. If True, the sonifications will be saved in './sonif'.

• model : str (optional)
  Name of the pre-trained model to be used for the analysis. Default is 'harmonix-all'. Please refer to the documentation for the available models.

• device : str (optional)
  Device to be used for computation. Default is 'cuda' if available, otherwise 'cpu'.

• include_activations : bool (optional)
  Whether to include activations in the analysis results or not.

• include_embeddings : bool (optional)
  Whether to include embeddings in the analysis results or not.

• demix_dir : PathLike (optional)
  Path to the directory where the source-separated audio will be saved. Default is './demix'.

• spec_dir : PathLike (optional)
  Path to the directory where the spectrograms will be saved. Default is './spec'.

• keep_byproducts : bool (optional)
  Whether to keep the source-separated audio and spectrograms or not. Default is False.

• multiprocess : bool (optional)
  Whether to use multiprocessing for extracting spectrograms. Default is True.

• compile_model : bool (optional)
  EXPERIMENTAL. Wrap the loaded model(s) with torch.compile(mode='reduce-overhead'). Adds a ~57s one-time compilation cost on first inference, then ~38% faster steady-state forward passes -- only worth it for long-lived/batch processing, not single tracks. Default is False.

• demucs_overlap : float (optional)
  EXPERIMENTAL, accuracy-affecting. Overlap fraction between chunks passed to demucs' apply_model(). Default is 0.25 (demucs' own default; unchanged from prior behavior). Profiling showed different values can shift segment boundaries slightly.

• demucs_fp16 : bool (optional)
  EXPERIMENTAL, accuracy-affecting. Run demucs separation under torch.autocast('cuda', dtype=torch.float16) for faster separation. Default is False (fp32, unchanged from prior behavior). Only takes effect on CUDA.

Returns:

• Union[AnalysisResult, List[AnalysisResult]]
  Analysis results for the provided audio files.

load_result()

Loads the analysis results from the disk.

result = allin1_infer.load_result('./struct/24k_Magic.json')

visualize()

Visualizes the analysis results.

fig = allin1_infer.visualize(result)
fig.show()

Parameters:

• result : Union[AnalysisResult, List[AnalysisResult]]
  List of analysis results or a single analysis result to be visualized.

• out_dir : PathLike (optional)
  Path to the directory where the visualizations will be saved. By default, the visualizations will not be saved.

Returns:

• Union[Figure, List[Figure]] List of figures or a single figure containing the visualizations. Figure is a class from matplotlib.pyplot.

sonify()

Sonifies the analysis results. It will mix metronome clicks for beats and downbeats, and event sounds for segment boundaries to the original audio file.

y, sr = allin1_infer.sonify(result)
# y: sonified audio with shape (channels=2, samples)
# sr: sampling rate (=44100)

Parameters:

• result : Union[AnalysisResult, List[AnalysisResult]]
  List of analysis results or a single analysis result to be sonified.
• out_dir : PathLike (optional)
  Path to the directory where the sonifications will be saved. By default, the sonifications will not be saved.

Returns:

• Union[Tuple[NDArray, float], List[Tuple[NDArray, float]]]
  List of tuples or a single tuple containing the sonified audio and the sampling rate.

Visualization & Sonification

This package provides a simple visualization (-v or --visualize) and sonification (-s or --sonify) function for the analysis results.

all-in-one-infer -v -s your_audio_file.wav

The visualizations will be saved in the ./viz directory by default:

./viz
└── your_audio_file.pdf

The sonifications will be saved in the ./sonif directory by default:

./sonif
└── your_audio_file.sonif.wav

For example, a visualization looks like this:

You can try it at Hugging Face Space.

Available Models

The models are trained on the Harmonix Set with 8-fold cross-validation. For more details, please refer to the paper.

• harmonix-all: (Default) An ensemble model averaging the predictions of 8 models trained on each fold.
• harmonix-foldN: A model trained on fold N (0~7). For example, harmonix-fold0 is trained on fold 0.

By default, the harmonix-all model is used. To use a different model, use the --model option:

all-in-one-infer --model harmonix-fold0 your_audio_file.wav

Speed

With an RTX 4090 GPU and Intel i9-10940X CPU (14 cores, 28 threads, 3.30 GHz), the harmonix-all model processed 10 songs (33 minutes) in 73 seconds.

Advanced Usage for Research

This package provides researchers with advanced options to extract frame-level raw activations and embeddings without post-processing. These have a resolution of 100 FPS, equivalent to 0.01 seconds per frame.

CLI

Activations

The --activ option also saves frame-level raw activations from sigmoid and softmax:

$ all-in-one-infer --activ your_audio_file.wav

You can find the activations in the .npz file:

./struct
└── your_audio_file1.json
└── your_audio_file1.activ.npz

To load the activations in Python:

>>> import numpy as np
>>> activ = np.load('./struct/your_audio_file1.activ.npz')
>>> activ.files
['beat', 'downbeat', 'segment', 'label']
>>> beat_activations = activ['beat']
>>> downbeat_activations = activ['downbeat']
>>> segment_boundary_activations = activ['segment']
>>> segment_label_activations = activ['label']

Details of the activations are as follows:

• beat: Raw activations from the sigmoid layer for beat tracking (shape: [time_steps])
• downbeat: Raw activations from the sigmoid layer for downbeat tracking (shape: [time_steps])
• segment: Raw activations from the sigmoid layer for segment boundary detection (shape: [time_steps])
• label: Raw activations from the softmax layer for segment labeling (shape: [label_class=10, time_steps])

The frame rate of these activations (frames per second, i.e. 1 / time_steps in seconds) is available as result.activation_fps whenever include_activations=True -- read it from there instead of hardcoding 100, since it reflects the actual loaded model's cfg.fps rather than assuming a fixed value.

You can access the label names as follows:

>>> allin1_infer.HARMONIX_LABELS
['start',
 'end',
 'intro',
 'outro',
 'break',
 'bridge',
 'inst',
 'solo',
 'verse',
 'chorus']

Embeddings

This package also provides an option to extract raw embeddings from the model.

$ all-in-one-infer --embed your_audio_file.wav

You can find the embeddings in the .npy file:

./struct
└── your_audio_file1.json
└── your_audio_file1.embed.npy

To load the embeddings in Python:

>>> import numpy as np
>>> embed = np.load('your_audio_file1.embed.npy')

Each model embeds for every source-separated stem per time step, resulting in embeddings shaped as [stems=4, time_steps, embedding_size=24]:

1. The number of source-separated stems (the order is bass, drums, other, vocals).
2. The number of time steps (frames). The time step is 0.01 seconds (100 FPS).
3. The embedding size of 24.

Using the --embed option with the harmonix-all ensemble model will stack the embeddings, saving them with the shape [stems=4, time_steps, embedding_size=24, models=8].

Python

The Python API allin1_infer.analyze() offers the same options as the CLI:

>>> allin1_infer.analyze(
      paths='your_audio_file.wav',
      include_activations=True,
      include_embeddings=True,
    )

AnalysisResult(
  path='/path/to/your_audio_file.wav', 
  bpm=100, 
  beats=[...],
  downbeats=[...],
  segments=[...],
  activations={
    'beat': array(...), 
    'downbeat': array(...), 
    'segment': array(...), 
    'label': array(...)
  }, 
  embeddings=array(...),
)

Concerning MP3 Files

Due to variations in decoders, MP3 files can have slight offset differences. I recommend you to first convert your audio files to WAV format using FFmpeg (as shown below), and use the WAV files for all your data processing pipelines.

ffmpeg -i your_audio_file.mp3 your_audio_file.wav

In this package, audio files are read using Demucs. To my understanding, Demucs converts MP3 files to WAV using FFmpeg before reading them. However, using a different MP3 decoder can yield different offsets. I've observed variations of about 20~40ms, which is problematic for tasks requiring precise timing like beat tracking, where the conventional tolerance is just 70ms. Hence, I advise standardizing inputs to the WAV format for all data processing, ensuring straightforward decoding.

🔄 Migration from All-In-One

All-In-One-Infer is designed to be a drop-in replacement with enhanced features. Here's how to migrate:

Package Name Changes

# Old (All-In-One)
from allin1 import analyze

# New (All-In-One-Infer)  
from allin1_infer import analyze

CLI Command Changes

# Old
allin1 track.wav -o ./results

# New
all-in-one-infer track.wav -o ./results

Dependency Changes

# Old dependencies (All-In-One - original)
dependencies = ["demucs", "natten>=0.15.0"]

# Current (3.0.0+) dependencies — NATTEN is no longer required
dependencies = ["torch>=2.0.0", "demucs-infer", ...]  # pure-PyTorch neighborhood attention
# Optional fused-kernel backend: pip install "all-in-one-infer[natten]"  (natten>=0.17.1,<0.20)

Installation Methods

All-In-One-Infer supports both UV (recommended, faster) and pip (traditional):

# With UV (recommended, faster dependency resolution)
uv pip install git+https://github.com/openmirlab/all-in-one-infer.git

# With traditional pip (still fully supported)
pip install git+https://github.com/openmirlab/all-in-one-infer.git

# Editable install for development (works with both)
git clone https://github.com/openmirlab/all-in-one-infer.git
cd all-in-one-infer
uv pip install -e .
# or
pip install -e .

Note: The package uses modern pyproject.toml with hatchling backend, following PEP 621 standards. Dependencies are automatically installed from GitHub (demucs-infer, madmom) and PyPI (other packages).

What Stays the Same

• ✅ All analysis results format (JSON structure unchanged)
• ✅ All function signatures and return types
• ✅ All model names and parameters
• ✅ All core functionality and accuracy
• ✅ All visualization and sonification features

What's Enhanced

• 🆕 NATTEN removed as a hard dependency — pure-PyTorch NA ships by default (numerically identical, no compiled extension, any torch>=2.0)
• 🆕 Optional [natten] extra (natten>=0.17.1,<0.20, torch<2.8) as a speed optimization
• 🆕 PyTorch 2.0-2.7.0 and CUDA 11.7-12.8 compatibility
• 🆕 Uses demucs-infer package for PyTorch 2.x compatible separation
• 🆕 Clean dependency management via demucs-infer
• 🆕 Flexible source separation options
• 🆕 Direct stems input capability
• 🆕 Custom model integration
• 🆕 Performance improvements (model caching, GPU cleanup)
• 🆕 Better error handling and stability
• 🆕 Modern packaging (UV-style with pip compatibility)

Training

This package is inference-only; training code has been removed. TRAINING.md is kept as a historical guide. To retrain models, refer to the upstream mir-aidj/all-in-one repository.

Citation

If you use this package for your research, please cite the following papers:

All-In-One (core music structure analysis algorithms):

@inproceedings{taejun2023allinone,
  title={All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audio},
  author={Kim, Taejun and Nam, Juhan},
  booktitle={IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)},
  year={2023}
}

Demucs (source separation models):

@inproceedings{defossez2021hybrid,
  title={Hybrid Spectrogram and Waveform Source Separation},
  author={Défossez, Alexandre},
  booktitle={Proceedings of the ISMIR 2021 Workshop on Music Source Separation},
  year={2021}
}

📝 About All-In-One-Infer

What is This Project?

All-In-One-Infer is a unified package that combines:

• Music structure analysis from All-In-One by Taejun Kim & Juhan Nam
• Source separation via demucs-infer package
• Pure-PyTorch neighborhood attention by default (no NATTEN required); optional [natten] extra for NATTEN's fused-kernel backend
• Performance improvements and integration work

Key Principles

🎯 Respect Original Work:

Core Research: Unchanged ✅

• ✅ All-In-One model architectures (100% original)
• ✅ Beat/downbeat/tempo algorithms (100% original)
• ✅ Structure segmentation (100% original)
• ✅ Research quality and accuracy (100% original)

This Fork's Contributions 🔧

• PyTorch 2.x compatibility (NATTEN dependency removed; pure-PyTorch neighborhood attention by default, optional [natten] extra)
• Performance optimizations (model caching, GPU management)
• Modern packaging and dependency management
• Enhanced error handling and user experience
• Source separation via demucs-infer package

Attribution 📚

• All-In-One research → Taejun Kim & Juhan Nam (original)
• Source separation → demucs-infer (openmirlab/demucs-infer)
• PyTorch 2.x compatibility → This fork

For Researchers

When using this package, please cite the All-In-One paper for music structure analysis. See Citation section for BibTeX.

Project Information

License: MIT (same as All-In-One and Demucs) Original All-In-One: github.com/mir-aidj/all-in-one Original Demucs: github.com/facebookresearch/demucs

What Changed in v2.0.0? (historical)

See Motivation & Changes section above for detailed breakdown of modifications.

📚 Documentation

Comprehensive documentation is available in the docs/ directory:

• USAGE_EXAMPLES.md - Detailed usage examples and code snippets
• TRAINING.md - Historical training guide (this package is inference-only; see note in the doc)
• INTEGRATION.md - Details about Demucs integration
• IMPROVEMENTS.md - Performance improvements documentation
• CHANGELOG.md - Version history and release notes

For more information, see the Documentation Index.