AllMusicCaps: Album Reviews as Complementary Supervision for Music CLAP
Pablo Alonso-Jiménez, Xavier Lizarraga-Seijas, Xavier Serra, Dmitry Bogdanov
Music-text contrastive (CLAP) models trained with captions derived from professional album reviews, plus the AllMusicCaps caption dataset.
Install
From PyPI
pip install amclap
From source
For embedding extraction or fine-tuning:
pip install .
For development including pre-training your own models:
pip install -e .[train]
Inference
Load a model by specifying its Hugging Face model ID:
import torch
from amclap import get_model
x_a = torch.randn(1, 24000 * 10).cpu() # mono audio at 24 kHz
x_t = ["dreamy shoegaze with washed-out guitars"]
model_id = "mtg-upf/allmusiccaps_te_trained_sigreg"
model = get_model(model_id=model_id, device="cpu")
with torch.no_grad():
z_a = model.forward_audio(x_a) # torch.Size([1, 512])
z_t = model.forward_text(x_t) # torch.Size([1, 512])
Audio must be mono at 24 kHz. Both towers output 512-dimensional embeddings in a shared space, comparable with cosine similarity.
Note:
pip install amclapis enough for inference on every released model. Training with the SigReg objective additionally needslejepa, which has no PyPI release and so is not part of the[train]extra — install it from git:pip install "lejepa @ git+https://github.com/rbalestr-lab/lejepa.git"
Available models
All models use the OMAR-RQ audio encoder and an all-mpnet-base-v2 text encoder (TE). Step is
the checkpoint the paper reports.
| Model | Data | Layers | Objective | TE | Step |
|---|---|---|---|---|---|
| baseline | baseline | last | InfoNCE | frozen | 147k |
| amcquotes | quotes | last | InfoNCE | frozen | 150k |
| amcstruct | struct | last | InfoNCE | frozen | 150k |
| baseline_quotes | baseline+quotes | last | InfoNCE | frozen | 150k |
| baseline_struct | baseline+struct | last | InfoNCE | frozen | 150k |
| layer6 | baseline+quotes | 6 | InfoNCE | frozen | 150k |
| all_layers | baseline+quotes | all | InfoNCE | frozen | 150k |
| sigmoid | baseline+quotes | all | sigmoid | trained | 150k |
| lejepa | baseline+quotes | all | LeJEPA (cosine) | frozen | 150k |
| infonce_sigreg | baseline+quotes | all | LeJEPA (InfoNCE) | frozen | 150k |
| te_trained | baseline+quotes | all | InfoNCE | trained | 60k |
| te_trained_sigreg | baseline+quotes | all | InfoNCE+SigReg | trained | 60k |
te_trained_sigreg is the best overall model; all_layers is the frozen-TE recipe it builds on.
Note: models with a trainable text encoder overfit past ~40--80k steps, so they are released at their 60k checkpoint rather than the final one.
Hugging Face Model IDs
- mtg-upf/allmusiccaps_baseline
- mtg-upf/allmusiccaps_amcquotes
- mtg-upf/allmusiccaps_amcstruct
- mtg-upf/allmusiccaps_baseline_quotes
- mtg-upf/allmusiccaps_baseline_struct
- mtg-upf/allmusiccaps_layer6
- mtg-upf/allmusiccaps_all_layers
- mtg-upf/allmusiccaps_sigmoid
- mtg-upf/allmusiccaps_lejepa
- mtg-upf/allmusiccaps_infonce_sigreg
- mtg-upf/allmusiccaps_te_trained
- mtg-upf/allmusiccaps_te_trained_sigreg
The AllMusicCaps dataset
540,454 rows pairing YouTube tracks with captions derived from AllMusic album reviews, in two styles: review quotes and LLM-filled structured attributes. It contains identifiers and captions, no audio.
from datasets import load_dataset
ds = load_dataset("mtg-upf/allmusiccaps", split="train")
print(ds[0]["generated_quotes_captions"])
v1 is the default. allmusiccaps_v0.jsonl is what the paper's models were trained on, kept for
exact reproducibility: it holds the raw LLM output, in which generated_quotes_captions is not
consistently typed, so Arrow-backed readers reject it. v1 normalizes that field to list[string]
and changes nothing else. See the dataset card
for the mapping and how to read v0, and
normalize_allmusiccaps_v1.py to
reproduce it. A 50-row sample lives in data/allmusiccaps/_samples/.
Training
- Install development dependencies:
pip install -e .[train]
- Prepare the data
Audio is stored downsampled to 24 kHz mono as 16-bit raw bytes (numpy memmap files); captions come from the JSONL above. Check the preprocessing scripts.
- Configuration
Experiment configuration is controlled with gin-config; see
cfg/README.md. At least the dataset paths need to point at your own data.
- Run the experiment
python -m amclap.train cfg/<config>.gin
Citation
If you find this work useful, please cite the paper:
@inproceedings{alonso2026allmusiccaps,
title = {{AllMusicCaps}: Album Reviews as Complementary Supervision for Music {CLAP}},
author = {Alonso-Jim{\'e}nez, Pablo and Lizarraga-Seijas, Xavier and Serra, Xavier and Bogdanov, Dmitry},
booktitle = {International Society for Music Information Retrieval Conference (ISMIR)},
year = {2026},
}
Licensing information
The code in this repository is available under AGPL-3.0 license. The model weights are available under CC BY-NC-SA 4.0 license for non-commercial applications. The AllMusicCaps dataset is released for non-commercial scientific research purposes only, and any publication of results based on it must cite AllMusic as the source of the data. Contact us for more information.
Metadata
Release files for amclap 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| amclap-0.1.0.tar.gz | 128.4 kB | With provenance | Details for amclap-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| amclap-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for amclap-0.1.0-py3-none-any.whl |
Total release size: 313.8 kB
Release files / amclap-0.1.0.tar.gz
| Download URL | amclap-0.1.0.tar.gz |
|---|---|
| Size | 128.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3af05e886244995b70448e4841890dad0e37077124390f52c9223ed4d1370e3d
|
|
BLAKE2b-256 checksum How to use checksums |
25c4c670731469cb36553e31310ca558c2a51204a7a1241219173fbce9a8e50f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 25, 2026.
Transparency logRelease files / amclap-0.1.0-py3-none-any.whl
| Download URL | amclap-0.1.0-py3-none-any.whl |
|---|---|
| Size | 185.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c42c387cb80ec1c77b3667085ee994fd4779329a30b9d5845797b91dc52ae5df
|
|
BLAKE2b-256 checksum How to use checksums |
25f7826a6bf5fdf7c1b1930fdb6130eee1a9c5cd1e118d9496c1a861c39a1ea0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 25, 2026.
Transparency logSecurity for amclap 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for amclap-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 25, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.