MORTM: Metric-Oriented Rhythmic Transformer for Music Generation
Takaaki Nagoshi
Project.MORTM Research Group
Abstract
Autoregressive models based on the Transformer architecture have achieved remarkable success in symbolic music generation. However, maintaining long-term structural coherence and rhythmic consistency remains a significant challenge, as standard tokenization methods often neglect the hierarchical nature of musical time.
We present MORTM (Metric-Oriented Rhythmic Transformer for Music), a novel framework that explicitly models metric structures through a bar-centric tokenization strategy. Version 4.5 introduces a scalable Sparse Mixture of Experts (MoE) architecture and FlashAttention-2 integration, enabling efficient training on extended contexts. Furthermore, we propose a Reinforcement Learning from Music Feedback (RLMF) pipeline using Proximal Policy Optimization (PPO), where the generator is aligned with stylistic objectives defined by a BERT-based reward model (BERTM).
1. Key Contributions
- Metric-Oriented Tokenization: A specialized vocabulary and encoding scheme that encapsulates musical events within a metric grid, enforcing bar-level structural integrity.
- Sparse Mixture of Experts (MoE): Implementation of Top-2 gating MoE layers to decouple model capacity from inference cost, allowing for massive parameter scaling.
- Efficient Long-Context Modeling: Integration of FlashAttention-2 and relative positional embeddings (ALiBi/RoPE) to handle extended musical sequences with linear memory complexity.
- Reinforcement Learning Alignment: A complete PPO-based RLHF pipeline that fine-tunes the autoregressive policy using rewards derived from a bidirectional discriminator (BERTM).
- Multimodal Scalability: Extensions for audio spectrogram modeling (V_MORTM) and piano-roll vision processing (MORTM Live).
2. Architecture
MORTM is built upon a decoder-only Transformer backbone, optimized for the nuances of symbolic music data.
2.1 Sparse Mixture of Experts (MoE)
To enhance the model's representational power without incurring prohibitive computational costs, we replace standard Feed-Forward Networks (FFNs) with MoE layers in selected blocks.
- Routing Mechanism: A learnable gating network routes each token to the Top-$k$ experts (default $k=2$).
- Expert Specialization: This allows different experts to specialize in distinct musical textures (e.g., rhythmic accompaniment vs. melodic phrasing).
2.2 Attention Mechanism
We employ FlashAttention-2 to accelerate the attention computation. $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$ Combined with Rotary Positional Embeddings (RoPE), the model effectively captures relative timing dependencies across thousands of tokens.
2.3 Reward Modeling (BERTM)
BERTM (Bidirectional Encoder Representations for Music) acts as a critic. Pre-trained on masked language modeling (MLM) and fine-tuned for genre/quality classification, it provides scalar rewards that guide the PPO training phase.
3. Installation & Prerequisites
This research code is implemented in PyTorch. For optimal performance, especially with FlashAttention-2, an NVIDIA GPU (Ampere architecture or newer) is recommended.
# Clone the repository
git clone [https://github.com/Ayato964/mortm.git](https://github.com/Ayato964/mortm.git)
cd mortm
# Install core dependencies
pip install torch torchvision torchaudio --index-url [https://download.pytorch.org/whl/cu118](https://download.pytorch.org/whl/cu118)
pip install flash-attn --no-build-isolation
# Install project requirements
pip install -r requirements.txt
1 maintainer
Metadata
1 maintainer
Release files for mortm 4.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| mortm-4.9.tar.gz | 98.1 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mortm-4.9-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 205.8 kB
Release files / mortm-4.9.tar.gz
| Download URL | mortm-4.9.tar.gz |
|---|---|
| Size | 98.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2d7c700705e5aef38b4c8f93902e238b2ac10e65461ae2553ab110e17acb6115
|
|
BLAKE2b-256 checksum How to use checksums |
c720bcebf81e25690031d809b6b1dd356967657bfe478fd27f54979c4a28c300
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / mortm-4.9-py3-none-any.whl
| Download URL | mortm-4.9-py3-none-any.whl |
|---|---|
| Size | 107.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0b6005c82c8eb07f56beae0bc2b6bf0c7fa9a5e17f76ecf7002f66d2e4db798f
|
|
BLAKE2b-256 checksum How to use checksums |
9c6620737ae79abc8343738e2c518f57502332fe0fdf3dc2ba52bb7e44fa8327
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Security for mortm 4.9
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
