muscriptor 0.1.0

Audio-to-MIDI transcription using a transformer language model

MuScriptor

MuScriptor is a multi-instrument music transcription model developed by Kyutai and Mirelo. MuScriptor is the first music transcription model that has been trained on a large scale dataset of 170k songs from classical music to heavy metal.

Online Demo | [Paper] (coming soon) | HuggingFace

Try it locally

You can try it locally with the web UI with:

uvx muscriptor serve

or with the CLI:

uvx muscriptor transcribe

Installation

with uv (recommended):

uv add muscriptor

pip install muscriptor

Models

Three variants are published under the MuScriptor HuggingFace organization. Everywhere a model is selected (load_model(), the CLI's --model, serve --model) you can pass the bare size keyword and the weights are downloaded and cached automatically. The architecture is a transformer decoder only. Here are the detailed model sizes:

Variant	Parameters	Layers	Dim	HuggingFace repo
small	103M	14	768	muscriptor-small
medium (default)	307M	24	1024	muscriptor-medium
large	1.4B	48	1536	muscriptor-large

small is the practical choice on CPU-only machines, medium is the default speed/accuracy trade-off, and large is the most accurate but really wants a GPU.

Usage

from pathlib import Path
from muscriptor import TranscriptionModel

# Downloads the default "medium" variant from HuggingFace (cached under
# ~/.cache/muscriptor/). Also accepts "small"/"large", a local safetensors
# path, or an hf:// / http(s):// URL.
model = TranscriptionModel.load_model()

# Stream events as they're transcribed. Optionally tell the model which
# instruments to expect — run `muscriptor list-instruments` for the names.
for event in model.transcribe("audio.wav", instruments=["acoustic_piano", "drums"]):
    print(event)

# Or get a MIDI file directly
midi_bytes = model.transcribe_to_midi("audio.wav")
Path("out.mid").write_bytes(midi_bytes)

from dataclasses import dataclass
from typing import Generator


@dataclass
class NoteStartEvent:
    pitch: int
    # Start of the note in seconds, from the beginning of the audio.
    start_time: float
    # A unique index for this note, used to match the corresponding
    # NoteEndEvent.
    index: int
    instrument: str


@dataclass
class NoteEndEvent:
    end_time: float
    # The NoteStartEvent this end matches. Convenient for consumers —
    # when serializing (e.g. to JSON) drop this field and rely on
    # `start_event_index` to refer back to the start by id.
    start_event: NoteStartEvent

    @property
    def start_event_index(self) -> int:
        return self.start_event.index


@dataclass
class ProgressEvent:
    # A coarse progress anchor woven into the event stream: `completed` of
    # `total` 5-second chunks have been transcribed. One is emitted up front
    # with completed == 0 (so consumers learn `total`), then one per finished
    # chunk. Advisory only — consumers that just build notes can ignore them.
    completed: int
    total: int


class TranscriptionModel:
    ...
    def transcribe(
            self,
            audio: str | Path | tuple[torch.Tensor, int],
            use_sampling: bool = False,
            temperature: float = 1.0,
            cfg_coef: float = 1.0,
            instruments: list[str] | None = None,
            batch_size: int | None = None,
            no_eos_is_ok: bool = True,
            beam_size: int = 1,
        ) -> Generator[NoteStartEvent | NoteEndEvent | ProgressEvent, None, None]:
        """Transcribe audio into a stream of note events.

        Args:
            audio: Path to an audio file, or a tuple `(tensor, sample_rate)`
                with a float tensor of shape [T] or [1, T] at `sample_rate`
                Hz. The tuple form is useful when the audio is already
                loaded in memory.
            use_sampling: Use temperature sampling instead of greedy decoding.
            temperature: Sampling temperature (only used when use_sampling=True).
            cfg_coef: Classifier-free guidance coefficient. Keep to 1 for the released models (they are post-RL)
            instruments: Optional list of instrument group names to
                condition the model on (exact names, e.g.
                ["acoustic_piano", "drums"]). Run `muscriptor
                list-instruments` (or GET /instruments on the server)
                for the full list of valid names.
            batch_size: Number of 5-second chunks processed per forward
                pass. `None` (default) picks a value based on the device:
                1 on CPU, 4 on GPU. Use `batch_size=1` for the lowest
                streaming latency — larger batches process several chunks
                together, so events belonging to later chunks of a batch
                won't arrive until the whole batch finishes. Within a
                batch, events are always yielded in temporal order; all
                events from chunk N are emitted before any event from
                chunk N+1.
            no_eos_is_ok: If True, a chunk that doesn't emit EOS within
                the generation budget produces a warning instead of raising.
            beam_size: Beam search width. 1 (default) uses greedy decoding
                (or sampling, with use_sampling=True); >= 2 enables beam
                search, which is slower but can be more accurate.

        Returns:
            Generator of NoteStartEvent, NoteEndEvent and ProgressEvent
            objects. Every
            NoteStartEvent is guaranteed to be followed by exactly one
            matching NoteEndEvent later in the stream (with the same
            `index`). Drum hits appear as a NoteStartEvent immediately
            followed by its matching NoteEndEvent at the same start time
            plus a tiny duration. Note: this tokenizer does not preserve
            velocity (loudness) — only onset/offset timing, pitch, and
            instrument are recovered.
        """

    def transcribe_to_midi(
            self,
            audio: str | Path | tuple[torch.Tensor, int],
            use_sampling: bool = False,
            temperature: float = 1.0,
            cfg_coef: float = 1.0,
            instruments: list[str] | None = None,
            batch_size: int | None = None,
            no_eos_is_ok: bool = True,
            beam_size: int = 1,
        ) -> bytes:
        """Same as `transcribe`, but returns a MIDI file as bytes instead
        of a generator of events. Useful when you want to save the MIDI
        to disk or send it over a network without going through the
        event stream.
        """

CLI

# Transcribe to MIDI (defaults to <audio_file>.mid next to the input)
muscriptor transcribe audio.wav -o out.mid

# Pick a model variant: small / medium / large (default: medium),
# a local safetensors path, or an hf:// / http(s):// URL
muscriptor transcribe audio.wav --model large

# Tell the model which instruments to expect (comma-separated names;
# run `muscriptor list-instruments` for the full list). Case-insensitive,
# and unambiguous abbreviations work: 'timp,dist' = timpani + distorted
# electric guitar
muscriptor transcribe audio.wav --instruments acoustic_piano,drums

# Get the event stream instead of MIDI: json (single array) or
# jsonl (one event per line, streamed while transcribing); -o - = stdout
muscriptor transcribe audio.wav --format jsonl -o -

# Decoding options: temperature sampling, or beam search (slower, can be
# more accurate)
muscriptor transcribe audio.wav --sampling -t 0.8
muscriptor transcribe audio.wav --beam-size 4

# Render a stereo check-mix of the result (left channel = original audio,
# right channel = synthesized MIDI; requires fluidsynth on PATH)
muscriptor transcribe audio.wav -o out.mid --auralize check.wav

See muscriptor transcribe --help for the full list of options.

Web UI

A browser client is included under web/. The FastAPI server serves both the UI and a POST /transcribe endpoint that streams NoteStart/NoteEnd events back as Server-Sent Events. The UI accepts an audio file (WAV, or any format soundfile/libsndfile can read — mp3, flac, ogg, m4a, …) via drag-and-drop, renders a live piano roll while events arrive, auto-plays once enough notes are available, and crossfades between the original WAV and the synthesized MIDI playback.

One-time setup

uv sync
cd web && pnpm install && pnpm run build && cd ..

pnpm run build is required once — the FastAPI server auto-mounts web/dist/ if it exists. The first build downloads two soundfonts: MuseScore_General.sf2 (215 MB, to the repo root, used by /auralize) and MuseScore_General.sf3 (38 MB, the compressed build the UI plays); later builds skip files already on disk.

Run

uv run muscriptor serve \
    --model medium \
    --device cuda \
    --host 0.0.0.0 \
    --port 8222

--model accepts a size keyword (small, medium, large) that downloads the matching variant from HuggingFace (cached under ~/.cache/muscriptor/), a local safetensors path, or an hf:// / http(s):// URL. It defaults to medium when omitted.

Then open http://127.0.0.1:8222/ (or the LAN address of the host) and drop a WAV onto the page.

• Drop --device cuda if running CPU-only.
• --host 0.0.0.0 makes it reachable on the LAN; the default 127.0.0.1 is local-only.
• Playback runs a full SoundFont synthesizer (SpessaSynth) in the browser, fed with MuseScore_General.sf3 — the same soundfont the /auralize endpoint uses, served by the app itself from /soundfonts/, no third-party CDN.

License

The code in this repository is released under the MIT license.

The model weights, published on HuggingFace, are released under the CC BY-NC 4.0 license (non-commercial use).

The MuseScore General SoundFont downloaded for playback / auralization is distributed under its own (MIT) license.

Citation

The paper is coming soon — a citation entry will be added here with it.