Skip to main content

AllMusicCaps: Album Reviews as Complementary Supervision for Music CLAP

Pablo Alonso-Jiménez, Xavier Lizarraga-Seijas, Xavier Serra, Dmitry Bogdanov

License PyPI version Dataset Tests

Music-text contrastive (CLAP) models trained with captions derived from professional album reviews, plus the AllMusicCaps caption dataset.

Install

From PyPI

pip install amclap

From source

For embedding extraction or fine-tuning:

pip install .

For development including pre-training your own models:

pip install -e .[train]

Inference

Load a model by specifying its Hugging Face model ID:

import torch
from amclap import get_model

x_a = torch.randn(1, 24000 * 10).cpu()   # mono audio at 24 kHz
x_t = ["dreamy shoegaze with washed-out guitars"]

model_id = "mtg-upf/allmusiccaps_te_trained_sigreg"
model = get_model(model_id=model_id, device="cpu")

with torch.no_grad():
    z_a = model.forward_audio(x_a)       # torch.Size([1, 512])
    z_t = model.forward_text(x_t)        # torch.Size([1, 512])

Audio must be mono at 24 kHz. Both towers output 512-dimensional embeddings in a shared space, comparable with cosine similarity.

Note: pip install amclap is enough for inference on every released model. Training with the SigReg objective additionally needs lejepa, which has no PyPI release and so is not part of the [train] extra — install it from git:

pip install "lejepa @ git+https://github.com/rbalestr-lab/lejepa.git"

Available models

All models use the OMAR-RQ audio encoder and an all-mpnet-base-v2 text encoder (TE). Step is the checkpoint the paper reports.

Model Data Layers Objective TE Step
baseline baseline last InfoNCE frozen 147k
amcquotes quotes last InfoNCE frozen 150k
amcstruct struct last InfoNCE frozen 150k
baseline_quotes baseline+quotes last InfoNCE frozen 150k
baseline_struct baseline+struct last InfoNCE frozen 150k
layer6 baseline+quotes 6 InfoNCE frozen 150k
all_layers baseline+quotes all InfoNCE frozen 150k
sigmoid baseline+quotes all sigmoid trained 150k
lejepa baseline+quotes all LeJEPA (cosine) frozen 150k
infonce_sigreg baseline+quotes all LeJEPA (InfoNCE) frozen 150k
te_trained baseline+quotes all InfoNCE trained 60k
te_trained_sigreg baseline+quotes all InfoNCE+SigReg trained 60k

te_trained_sigreg is the best overall model; all_layers is the frozen-TE recipe it builds on.

Note: models with a trainable text encoder overfit past ~40--80k steps, so they are released at their 60k checkpoint rather than the final one.

Hugging Face Model IDs

The AllMusicCaps dataset

540,454 rows pairing YouTube tracks with captions derived from AllMusic album reviews, in two styles: review quotes and LLM-filled structured attributes. It contains identifiers and captions, no audio.

from datasets import load_dataset

ds = load_dataset("mtg-upf/allmusiccaps", split="train")
print(ds[0]["generated_quotes_captions"])

v1 is the default. allmusiccaps_v0.jsonl is what the paper's models were trained on, kept for exact reproducibility: it holds the raw LLM output, in which generated_quotes_captions is not consistently typed, so Arrow-backed readers reject it. v1 normalizes that field to list[string] and changes nothing else. See the dataset card for the mapping and how to read v0, and normalize_allmusiccaps_v1.py to reproduce it. A 50-row sample lives in data/allmusiccaps/_samples/.

Training

  1. Install development dependencies:
pip install -e .[train]
  1. Prepare the data

Audio is stored downsampled to 24 kHz mono as 16-bit raw bytes (numpy memmap files); captions come from the JSONL above. Check the preprocessing scripts.

  1. Configuration

Experiment configuration is controlled with gin-config; see cfg/README.md. At least the dataset paths need to point at your own data.

  1. Run the experiment
python -m amclap.train cfg/<config>.gin

Citation

If you find this work useful, please cite the paper:

@inproceedings{alonso2026allmusiccaps,
  title = {{AllMusicCaps}: Album Reviews as Complementary Supervision for Music {CLAP}},
  author = {Alonso-Jim{\'e}nez, Pablo and Lizarraga-Seijas, Xavier and Serra, Xavier and Bogdanov, Dmitry},
  booktitle = {International Society for Music Information Retrieval Conference (ISMIR)},
  year = {2026},
}

Licensing information

The code in this repository is available under AGPL-3.0 license. The model weights are available under CC BY-NC-SA 4.0 license for non-commercial applications. The AllMusicCaps dataset is released for non-commercial scientific research purposes only, and any publication of results based on it must cite AllMusic as the source of the data. Contact us for more information.

Metadata

Release files for amclap 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for amclap 0.1.0
File Size Uploaded Provenance Actions
amclap-0.1.0.tar.gz 128.4 kB With provenance Details for amclap-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for amclap 0.1.0
File Interpreter ABI Platform Actions
amclap-0.1.0-py3-none-any.whl Python 3 none any Details for amclap-0.1.0-py3-none-any.whl

Total release size: 313.8 kB

Release files / amclap-0.1.0.tar.gz

Download URL amclap-0.1.0.tar.gz
Size 128.4 kB
Tags Source
SHA-256 checksum
How to use checksums
3af05e886244995b70448e4841890dad0e37077124390f52c9223ed4d1370e3d
BLAKE2b-256 checksum
How to use checksums
25c4c670731469cb36553e31310ca558c2a51204a7a1241219173fbce9a8e50f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 25, 2026.

Transparency log

Release files / amclap-0.1.0-py3-none-any.whl

Download URL amclap-0.1.0-py3-none-any.whl
Size 185.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c42c387cb80ec1c77b3667085ee994fd4779329a30b9d5845797b91dc52ae5df
BLAKE2b-256 checksum
How to use checksums
25f7826a6bf5fdf7c1b1930fdb6130eee1a9c5cd1e118d9496c1a861c39a1ea0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 25, 2026.

Transparency log

Security for amclap 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for amclap-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 25, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page