Skip to main content

OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction

Pablo Alonso-Jiménez, Pedro Ramoneda, R. Oguz Araz, Andrea Poltronieri, Dmitry Bogdanov

arXiv License

Lobster

Install

For embedding extraction or fine-tuning:

pip install .

For development including pre-training your own models:

pip install -e .[train]

Inference

Load a model by specifying its Hugging Face model ID:

import torch
from omar_rq import get_model

# Embedding extraction example
x = torch.randn(1, 24000 * 4).cpu()

model_id = "mtg-upf/omar-rq-multifeature-25hz-fsq"
model = get_model(model_id=model_id, device="cpu")

embeddings = model.extract_embeddings(x, layers=[6])

timestamps = torch.arange(embeddings.shape[2]) / model.eps

get_model reference:

Returns an OMAR-RQ Module from the provided  model_id or config_file.

Args:
    model_id (str): Hugging Face's Model ID or local path to the model
    config_file (Path): Path to the model config of a trained model.
    device (str): Device to use for the model. Defaults to "cpu".
    quantization_targets (bool): If True, it will create the quantization
        targets for SSL pre-training of the model. Defaults to False.
    load_weights (bool): If True, it will load the weights from the
        checkpoint. Defaults to True.

Output:
    module: The model from the provided config file.


Module usage:

Args:
    audio (torch.Tensor): 2D mono audio tensor (B, T'). Where B is
        the batch size and T' is the number of samples.
    layers (set): Set of layer indices to extract embeddings from.
        By default, it extracts embeddings from the last layer (logits).

Output:
    torch.Tensor: Extracted embeddings. The output tensor has shape
        (L, B, T, C,) where L = len(layers), B is the batch size, T is
        the number of output timestamps, and C = embedding dimension.


Example:

>>> x = torch.randn(1, 24000 * 4).cpu()
>>>
>>> model = get_model(config_file, device="cpu")
>>>
>>> embeddings = model.extract_embeddings(x, layers=(6))
>>>
>>> # use the `eps` field to compute timestamps
>>> timestamps = torch.arange(embeddings.shape[2]) / model.eps



>> NOTE: The model's embedding rate depends on the model's configuration.
    For example, the melspectrogram model has an embedding rate of 16ms.
    audio should be a sequence with a sample rate as inditacted in the
    config file and up to 30s.

extract_embeddings reference:

Extract embeddings from an input audio batch.

Args:
    audio (torch.Tensor): 2D mono audio tensor (B, T'). Where B is 
        the batch size and T' is the number of samples.
    layers (set): Set of layer indices to extract embeddings from.
        By default, it extracts embeddings from the last layer (logits).

Output:
    torch.Tensor: Extracted embeddings. The output tensor has shape 
        (L, B, T, C,) where L = len(layers), B is the batch size, T is
        the number of output timestamps, and C = embedding dimension.

Available models

Model Input Rate SR Tagging Difficulty Pitch Chord Beat Structure
Hz Hz mAP MSE acc. acc. F1 acc.
base mel 15.63 16000 .482 1.65 .892 .657 .783 .647
multicodebook mel 15.63 16000 .488 1.66 .897 .675 .775 .639
multifeature audio 18.75 24000 .467 1.76 .938 .734 .833 .623
multifeature-25hz audio 25 24000 .463 1.79 .932 .728 .848 .628
multifeature-25hz-fsq audio 25 24000 .463 1.71 .940 .749 .855 .628

Note: Different models were trained with different sample rates. It is responsibility of the user to ensure that the input audio is sampled at the correct rate.

OMAR-RQ models are offered in different configurations, each with its own strengths and weaknesses. Models based on mel spectrogram (base and multicodebook) tend to perform better on semantic tasks such as auto-tagging, structure recognition, and difficulty estimation. On the other hand, multifeature-24hz-fsq offers the best performance in tonal and temporal tasks such as pitch and chord estimation, and beat tracking.

Hugging Face Model IDs

Pre-training OMAR-RQ models

  1. Install development dependencies:
pip install -e .[train]
  1. Prepare the experiment data

We downsample our data to 16 kHz mono and store it as 16-bit raw bytes (numpy memmap files). Check our data preparation scripts.

  1. Configuration

Our experiment configuration is controlled with gin-config. Check the default config file to see the different parameters that can be modified.

At least the following parameters should be modified:

  • DiscotubeMultiViewAudioDataModule.data_dir -> Your base data folder.
  • DiscotubeMultiViewAudioDataModule.filelist_train -> Filelist of training audio paths relative to the data_dir (one file per line).
  • DiscotubeMultiViewAudioDataModule.filelist_val -> Same for the tracks on the validation split.
  1. Run the experiment
python src/train.py cfg/rq_single_view/config.gin

Citation

If you find this work useful, please cite the paper:

@inproceedings{alonso2025omar,
  title = {{OMAR-RQ}: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction},
  author = {Alonso-Jim{\'e}nez, Pablo and Ramoneda, Pedro and Araz, R. Oguz and Poltronieri, Andrea and Bogdanov, Dmitry},
  booktitle = {ACM Multimedia Conference (ACMMM), Open Source Track},
  year = {2025},
  doi = {10.1145/3746027.3756871},
}

Licensing information

The code in this repository is available under AGPL-3.0 license license. The model weights are available under CC BY-NC-SA 4.0 license for non-commercial applications. Contact us for more information.

Metadata

Release files for omar-rq 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for omar-rq 0.2.1
File Size Uploaded Provenance Actions
omar_rq-0.2.1.tar.gz 65.5 kB With provenance Details for omar_rq-0.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for omar-rq 0.2.1
File Interpreter ABI Platform Actions
omar_rq-0.2.1-py3-none-any.whl Python 3 none any Details for omar_rq-0.2.1-py3-none-any.whl

Total release size: 142.1 kB

Release files / omar_rq-0.2.1.tar.gz

Download URL omar_rq-0.2.1.tar.gz
Size 65.5 kB
Tags Source
SHA-256 checksum
How to use checksums
fcea4a747334db9be847cea6e7cab7e5c6c6a810e07bc2082207b35a4c976442
BLAKE2b-256 checksum
How to use checksums
2cb02e5b7d422645ef6d78ac59c86f3df2fa140a8fb0e3e976584691cf894294
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 22, 2025.

Transparency log

Release files / omar_rq-0.2.1-py3-none-any.whl

Download URL omar_rq-0.2.1-py3-none-any.whl
Size 76.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
46e627014d7f2a39ac960d7a6ef0eb9d89ccc6852a471583aa014e171f075032
BLAKE2b-256 checksum
How to use checksums
f7c6ce82a97758a51e64d5be532bed9d692f8219affca5867bbb93fdc8bed8a4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 22, 2025.

Transparency log

Security for omar-rq 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for omar_rq-0.2.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 22, 2025.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page