piano-song-to-visual

Turn a MIDI or MusicXML score into a Synthesia-style falling-notes practice video — no hands, just notes flowing onto a keyboard — arranged so a human can actually play it.

Status: done, at 1.0. One command turns a score into a practice video with sound, arranged to fit your hands, at whatever tempo and over whatever bars you want to drill. There is also a showcase mode: colour schemes and optional effects, off by default, for when you want the render to look like something rather than teach you something.

Done means finished rather than abandoned. It does what it was built to do, so the work from here is bug fixes, not features.

Two things were left out on purpose. Exporting the soundtrack on its own, since the mux backend already takes a finished audio file back. And a better built-in synth tone, which is the only one you would notice, and only if you have no SoundFont.

Why this exists

If you learn piano by watching and listening rather than by reading sheet music, you're stuck with whatever arrangements happen to exist on YouTube. For a lot of music — game soundtracks especially — nothing exists.

This tool takes a MIDI or a MusicXML score and produces the video you'd have wanted someone to make, with three things standard falling-note videos leave out:

• Dynamics you can see. Note brightness tracks velocity, so loud and soft are visible rather than guessed at.
• Pedal lanes. A lane to the right of the keyboard where pedal presses fall like notes do, showing exactly when the sustain pedal goes down and for how long.
• An alignment grid. Faint horizontal and vertical rules, so you can tell that two notes an octave and a half apart are actually simultaneous.

Plus a mode that is the opposite of all that, for when the video is the point rather than the practice: gradient backgrounds, colour schemes, sparks off the strike line. Off by default, because a practice aid and a piece of spectacle want opposite things. See Themes and Effects.

And one thing no falling-note renderer does at all:

• A hand-span constraint that is guaranteed, not suggested. You set your maximum comfortable simultaneous reach; the arrangement is rewritten so nothing in the output ever exceeds it. Difficulty is a separate knob — a harder setting gives you more notes and faster passages, never a wider stretch.

Scope

A written score in, video out. MIDI or MusicXML, and MusicXML is the better input where you have the choice, because it states which hand plays each note instead of leaving it to be guessed.

Audio in is not planned. Transcribing a recording sets a ceiling on quality that nothing downstream can raise, and it would drag a machine-learning stack into a repo that currently installs in seconds. Public-domain sheet music covers the classical repertoire this is aimed at, and it arrives with the hands, the dynamics and the pedalling already written down.

Pipeline

score ──▶ parse ──▶ arrange ──▶ constrain ──▶ render ──▶ video

Any stage runs on its own, so you can hand-fix an intermediate MIDI and pick up from there. docs/ARCHITECTURE.md describes how it fits together, and docs/CONSTRAINT-ENGINE.md covers the part no other tool does: how a passage too wide for your hands is rewritten until it fits.

Install

Requires Python 3.12+. No system ffmpeg needed: the render extra brings its own binary.

git clone https://github.com/RinkyDinkyNooble/piano-song-to-visual
cd piano-song-to-visual
pip install -e ".[all]"

[all] is what you want: psv run needs the renderer. A lighter install is possible if you only care about the MIDI stages:

pip install -e .            # inspect, export, arrange, constrain
pip install -e ".[render]"  # + video and audio, which is everything below

A command that needs an extra you do not have says which one to install rather than failing on whichever module it happened to reach first.

Sound works with no further setup: the built-in synth needs nothing but numpy. For a real sampled piano see A real piano sound below.

Usage

One command does everything:

psv run song.mid -o practice.mp4

  arrange          reduced to two hands, 12 note(s) dropped
  constrain        403 span violation(s) found, resolved in 9 pass(es)
    drop           57
    octave-shift   272
    reassign       50
    truncate       34
  audio            builtin
  notes            6082
wrote practice.mp4

That is a Beethoven string quartet becoming a piano piece you can reach every chord of, with sound, in about four seconds.

MusicXML as well as MIDI

Any command that takes a file takes either, told apart by content rather than by extension:

psv run sonata.musicxml -o practice.mp4
psv run sonata.mxl      -o practice.mp4    # the zipped form

MusicXML is the better input where you have the choice, because it states what MIDI leaves to be guessed. A piano score is written on two staves and the file says which staff every note is on, so the hands are read rather than inferred from track names. Dynamics arrive as pp and ff rather than as velocity bytes, and pedalling as a mark with a start and a stop.

Repeats are unrolled into a linear timeline, so a repeated section arrives twice: |: :|, first- and second-time bars, D.C., D.S., segno, coda and fine. docs/MUSICXML.md covers what the reader handles and where it stops.

How long a render takes

Frames are drawn independently, so the timeline is cut into spans and each span is rendered and encoded by its own process. Two settings control it, and they matter far more together than apart:

psv run song.mid -o out.mp4                   # both defaults
psv run song.mid -o out.mp4 --encode fast     # quickest, biggest file
psv run song.mid -o out.mp4 --workers 1       # one process, as it used to be

--encode chooses how long the encoder spends looking for things to compress. It changes the file size, not the picture:

On its own a faster encoder buys almost nothing, because a single-process render waits on the drawing rather than on the encoder. It is the combination that pays. Für Elise at 1080p60, on a six-core machine:

A short render is never split, since a worker costs a Python interpreter and an ffmpeg process before it draws anything.

Before committing to a file, it is worth asking what is actually in it:

psv inspect song.mid

beethoven-op18-no4-i-quartet
  duration       514.8s
  notes          6151 in 4 part(s)
  range          C2 to C7
  polyphony      peak 11, mean 3.7
  widest span    51 semitones at 503.5s
  tempo          138 BPM, constant
  meter          4/4
  dynamics       none (every note the same velocity)
  pedal          none
  hands          not separated; needs the arrange stage

The last three lines are the useful ones: they tell you whether the file carries the dynamics and pedal data the visuals depend on, and whether it needs arranging at all.

Presets

Named bundles, so a sensible result does not need a TOML file first:

psv run song.mid -o practice.mp4 --preset beginner
psv presets                                  # what each one actually sets

small-hands (a 9-semitone reach), beginner (small hands, thinner texture, 0.7x tempo, two bars of count-in), as-written (no span limit and nothing thinned for difficulty), and draft (640x360, no audio, for iterating).

as-written still reduces a multi-instrument score to two hands, because two hands is the premise. What it turns off is the span limit and difficulty thinning.

Themes

A preset changes how the piece is played. A theme only changes how it looks, so the two compose:

psv run song.mid -o practice.mp4 --theme midnight
psv presets                                  # lists themes as well as presets

midnight (deep blue, blue against amber), ember (warm dark red, amber against teal), neon (violet and hard white edges), aurora (deep teal, violet against green). None is on by default: the plain look is the one that stays out of the way while you are learning something.

Every theme leaves hue carrying which hand and brightness carrying how loud. That is the readability rule, and there is a test that no theme spends it.

Effects

Off unless you ask. Sparks off the strike line, a glow on the pressed key, a flash as a note lands:

psv run song.mid -o showcase.mp4 --theme neon --effects showcase

subtle (a glow and a flash), showcase (that plus sparks), maximum (everything but bloom), none (turns off whatever a config file asked for).

Or list them yourself, in the order they draw:

[[visual.effects]]
kind = "key_glow"
intensity = 0.6

[[visual.effects]]
kind = "strike_flash"
intensity = 0.8

The kinds are strike_flash, key_glow, trail, particles, halo, pulse and bloom. Each takes an intensity from 0 to 1, and 0 is a no-op rather than something faint.

Measured at 1080p, against the 10.6 ms it takes to draw a frame at all: subtle 1.4 ms, showcase 2.6 ms, maximum 11.7 ms. bloom on its own is 39 ms, nearly four times a whole frame, so it is in none of the bundles and you have to name it. It is the only effect that reads the frame it is given, and it weighs every pixel at full resolution before shrinking, which is what stops identical notes glowing by different amounts.

Precedence runs least specific to most: the config file, then the preset, then the theme, then the effects, then the individual flags. --preset small-hands --span 14 gives you 14.

A video you would post

Off unless you ask, because a practice video wants none of it:

psv run song.musicxml -o out.mp4 --title-card 4 --fade-out 3 --hold-black 2

A title card at the front, fading to nothing to reveal the music already playing behind it, then a fade to black at the end and a few seconds held there. The card is an overlay rather than a separate clip spliced on the front, which is the only way the notes can already be falling while it clears.

You do not type the words. MusicXML carries <work-title> and <creator type="composer">, so a real score names itself; --title and --composer override when the file is wrong or when the input is a MIDI, which has nowhere to put either. [title] in the config file has the rest: the screen colour, a footer line for a channel name, a font, and which way the opacity falls.

The card clears before it leaves the screen — 70% of the way through by default — so there is a moment of clear picture with the first notes visibly falling before any of them lands.

This is the one step that re-encodes the finished file. Everything else is copied, so it costs a generation and a minute or so on a long render, which is why it stays off until asked for.

Practising with it

A video of the piece at full speed, both hands, from the top, is not how anyone learns a piece. Four flags cover how it is actually done:

psv run song.mid -o practice.mp4 --bars 20-40 --tempo 0.6 --hands left --count-in 2

• --tempo 0.6 plays at six-tenths of the written speed, same notes. Work a hard passage up from something you can play cleanly.
• --bars 20-40 renders only those bars, counting from 1 and including both ends. Bars are what you think in; --start and --seconds are still there when you want wall-clock time instead, and cannot be combined with --bars.
• --hands left sounds one hand. The other stays on screen, faintly, so you can still see where it is.
• --count-in 2 puts two bars of clicks in front of the music, at the tempo and meter you are about to play. --silent-count-in keeps that time and drops the clicks, for when the notes falling toward the line are counting for you. --metronome keeps clicking through the piece.

None of these touch the arrangement. They run after arrange and constrain, so the piece you practise at half speed is note for note the piece you practise at full speed. All four can also live in a config file:

[practice]
tempo = 0.75
hands = "both"       # both | left | right
count_in_bars = 2
count_in_clicks = true
metronome = false

A flag on the command line beats the file.

Playing it without the pedal

Transcriptions often carry pedalling you do not want, and a piano you are practising on may not have a working pedal at all:

psv run song.mid -o practice.mp4 --no-pedal

This is not a display setting. The pedal events are deleted as the score is read, so every stage below sees a piece written without them: the arrangement changes, no lane is drawn, and no CC64 reaches the synthesiser.

It costs a little music, and that is the honest part. The constraint engine's cheapest repair is to lift a finger early while the pedal holds the note ringing, which nobody can hear. With no pedal it has to move an octave or drop a note instead, in the same places a player without a pedal would lose them. Every one of those is reported, so -vv shows you exactly what it cost.

To hide the lane without changing anything you hear, use --pedals-lanes 0 instead. That is a display setting.

Iterating quickly

A full 1080p60 render of a long piece takes minutes. While you are trying settings, keep it small and short:

psv run song.mid -o preview.mp4 --start 30 --seconds 10 --width 640 --height 360 --fps 30

That costs about a second.

Running one stage at a time

Each stage reads and writes MIDI, so you can stop after any of them, fix the file by hand in any MIDI editor, and carry on:

psv arrange   song.mid      -o two-hands.mid
psv constrain two-hands.mid -o playable.mid    # add -vv to see every decision
psv render    playable.mid  -o practice.mp4

psv export song.mid -o copy.mid parses and writes straight back out, which is how you check that ingest understood a file.

Configuration

Everything meaningful is configurable via a TOML file rather than flags you have to remember. The sketch:

Every setting below also has a flag, named after where it lives: visual.grid.opacity is --visual-grid-opacity. The flags are generated from the config itself, so -h lists all of them and cannot fall behind. Shorter names for the ones used most (--fps, --span, --tempo, --no-pedal) are kept alongside the long forms.

Values layer from least specific to most: the defaults, then the TOML file, then a --preset, then the flags. A flag you do not pass changes nothing, so a config file and a one-off override do not fight.

The grid keys are named after what each line marks, not which way it runs. In a falling-notes view the horizontal axis is pitch and the vertical axis is time, so "horizontal lines" and "vertical lines" are easy to get backwards.

[hands]
max_span_semitones = 12   # hard limit on simultaneously held notes; 18 = 1.5 octaves
                          # never relaxed, at any difficulty.
                          # 0 means no limit: the piece exactly as written,
                          # which psv then says loudly rather than implying
                          # it checked something
overlap_tolerance_s = 0.03 # overlaps shorter than this do not count as
                          # simultaneous. A note released 10 ms after the next
                          # one starts is sloppy MIDI, not a stretch to make

[difficulty]
level = "original"        # note density, ornamentation, harmonic detail

[visual]
width = 1920
height = 1080             # both must be even: h264 encodes in 2x2 blocks, and
                          # an odd size would be quietly padded
fps = 60
lookahead_s = 3.0         # seconds of music visible above the keyboard at once
background = "#101010"    # any hex colour. Grey by default so nothing back here
                          # competes with the hues that say which hand is playing
black_key_bar_width = 0.6 # relative to white-key bars, so black keys read from far away
black_key_darkening = 0.2 # applied on top of the note's colour
note_border = 0.0016      # outline on each bar, as a fraction of frame width.
                          # This is what separates four fast repeats on one key
                          # from one long block. 0 turns it off
note_border_shade = -0.45 # -1 black, 0 the bar's own colour, +1 white. Negative
                          # cuts the bar out of the background, positive lights
                          # it from inside
note_radius = 0.0         # rounds the ends of each bar, as a fraction of the
                          # bar's own width. 0.5 makes each end a half-circle.
                          # A fraction of the bar, not the frame, because a
                          # black-key bar is narrower and wants less
bar_gradient = 0.0        # brightness ramp along each bar. Positive fades the
                          # top, negative fades the bottom
gradient_top = ""         # a vertical gradient behind everything. Set both ends
gradient_bottom = ""      # to use it; it then replaces `background` and may
                          # have a hue, which `background` may not
workers = 0               # processes to render with; 0 is one per core,
                          # 1 renders in a single process
encode = "balanced"       # small | balanced | fast: how long the encoder
                          # spends compressing, against how big the file is

[visual.colors]           # hue = which hand, brightness = how loud
left_hand  = "#4a90d9"
right_hand = "#5fb87a"
unassigned = "#9aa0ac"    # before hand assignment has run
pedal      = "#c8a44a"
quiet = 0.35              # brightness at pp
loud  = 1.0               # brightness at ff. Setting these equal is how you
                          # turn dynamics off: every note the same brightness

[visual.grid]
pitch_lines = "octave"    # vertical rules at every C, for finding a key
beat_lines  = "beat"      # horizontal rules on the beat, for spotting simultaneity
opacity     = 0.15        # faint: an aid, not decoration

[[visual.effects]]        # optional, off by default, drawn in the order listed.
kind = "strike_flash"     # strike_flash | key_glow | trail | particles
intensity = 0.8           # halo | pulse | bloom. 0 draws nothing at all

[pedals]
enabled = true            # false reads the piece as if written without pedals:
                          # different repairs, no lane, no CC64. Not the same
                          # question as lanes = 0, which only hides the lane
lanes = 1                 # up to 3; sustain is the one MIDI reliably carries
threshold = 1             # controller value at which a pedal counts as engaged.
                          # 1 shows half-pedalling; 64 is the on/off convention

[audio]
backend = "builtin"       # builtin | fluidsynth | mux | none
soundfont = ""            # for backend = "fluidsynth": path to a .sf2
fluidsynth_bin = ""       # folder holding the native library, so it does not
                          # have to be on PATH for one optional backend
program = 0               # which instrument in that SoundFont; `psv instruments`
reverb = 0.5              # how much room the piano is played in, 0 dry to 1 a
                          # large hall. 0.5 is what it has always sounded like:
                          # FluidSynth's reverb is on unless you turn it off.
                          # fluidsynth backend only, and the others say so
audio_file = ""           # for backend = "mux": your own recording
offset_s = 0.0            # nudge that recording into sync
stereo_width = 0.5        # low notes left, high notes right, as at the keyboard

[title]                   # a card at the front and a fade at the end, for a
                          # video you will post. Off until `seconds` is set,
                          # and the only step that re-encodes the finished file
seconds = 0.0             # how long the card is up. 0 turns the whole thing off
text = ""                 # what it says; empty takes the score's own title
composer = ""             # the second line; MusicXML carries this, MIDI does not
footer = ""               # a third line, fainter and lower. For a channel name
font = ""                 # a .ttf or .otf; empty finds a serif, and falls back
                          # to a built-in face rather than failing the render
screen = "#0a0a0a"        # the screen behind the text
opacity = 1.0             # how opaque it starts, before it fades to nothing
clear_at = 0.0            # when it reaches nothing. 0 means 70% of `seconds`,
                          # leaving clear screen so the first notes are visible
                          # falling before any of them lands
curve = "ease"            # ease | linear | slow: how the opacity falls
fade_out_s = 0.0          # fade the picture and the sound to black over this
hold_s = 0.0              # then hold on black for this long

[practice]                # how the finished arrangement is presented
tempo = 1.0               # 0.75 renders at three-quarters speed
hands = "both"            # both | left | right
count_in_bars = 0         # bars of lead-in before the music
count_in_clicks = true    # false keeps the time and drops the beeps, which is
                          # what you want once the falling notes count for you
metronome = false         # keep clicking through it

A real piano sound

The built-in synth is sine harmonics with an envelope: fine for keeping your place, obviously synthetic. For a sampled instrument, point at a SoundFont:

[audio]
backend = "fluidsynth"
soundfont = "~/.local/fluidsynth/GeneralUser-GS.sf2"
fluidsynth_bin = "~/.local/fluidsynth/bin"      # folder holding the DLL
program = 0    # 0 grand, 1 bright, 4 Rhodes, 5 FM electric, 6 harpsichord
reverb = 0.5   # 0 dry, 1 a large hall

reverb is one number driving FluidSynth's room size, damping, width and level together, because exposing all four means picking four numbers to find out that three of them barely matter. 0.5 is what psv has always sounded like: FluidSynth enables its own reverb unless told not to, so this has never been dry, and the middle of the range is those settings rather than a new opinion. --reverb 0.8 overrides it for one run. The other backends do not go through FluidSynth and say so rather than implying it happened.

You need the FluidSynth binaries matching your Python's architecture, and any .sf2 SoundFont (GeneralUser GS is a good 30 MB starting point). fluidsynth_bin exists so you do not have to put the library on your PATH. If anything is missing, the render falls back to the built-in synth and says which piece it could not find.

psv instruments lists what audio.program can select, reading the SoundFont's own preset names where one is configured: General MIDI is a convention, and a font may put anything at any number. docs/SOUNDS.md covers where SoundFonts come from, why bigger is usually better, and how to audition one in a couple of seconds.

Tests

pytest
python scripts/fetch_test_songs.py   # optional: the two CC BY-SA test songs

Two public-domain songs are committed, so the suite runs green on a fresh clone with no network. Every feature the tool promises is registered in tests/features.toml and cannot be marked done until a test claims it.

python scripts/fetch_test_scores.py  # optional: the MusicXML test suite

That second set is 29 small MusicXML files, each built to break one corner of the format. They are MIT and therefore not committed, for the same reason the CC BY-SA songs are not: this repository is 0BSD and should not quietly attach a condition it says is not there. Tests needing them skip when they are absent.

Contributing

This is a personal tool that happens to be public, so bug reports are more welcome than large features; ask first if the change is more than a fix. CONTRIBUTING.md has the setup, the four checks CI runs, and the three rules the project will not trade away.

Licence

0BSD — do whatever you want, no attribution required.

Note that the licence covers this code. Music you feed it is your responsibility; this is a tool for learning to play things yourself.