song2mv

Song in, beat-synced lyric timing and a code-drawn music video out — built for coding agents.

song2mv turns an audio file and its lyrics into synced lyrics (LRC, SRT, word-timed JSON) on the song's beat grid, then into a lyric or music video drawn entirely in code. Each step is its own command, so a coding agent such as Claude Code or Codex — or a person — can use just the part they need.

Two songs made with Suno, each turned into a music video and a Short by Claude Code with song2mv:

Till Money Means Nothing (4:55, Short): a launch broadcast that ends on Mars, inked with the toon style pack	Carry the Light (4:48, Short): a child's crayon drawing of a launch tower that becomes real, in watercolour from the paint style pack

How they were made: one Claude Code session (Opus 5.5, max effort) split the work among agents, as step 4 of the skill describes: five drew the shared pieces, then seven built the sections, in about two hours for both songs. One of these videos is about 12,000 lines of drawing code, where a single agent wrote 750–1,200 in our trial videos: expect a simpler video from one agent, or give each brief to a session of its own.

Alpha (0.x). Commands, flags and file formats may change between releases; every JSON file carries a schema_version and every change is listed in CHANGELOG.md. sync and beats are the most tested parts. check and render are experimental.

Install

Install it with uv:

uv tool install --python 3.13 "song2mv[sync]"

Or run it once without installing anything:

uvx --python 3.13 --from "song2mv[sync]" song2mv --help

Inside a virtual environment, pip install "song2mv[sync]" works too.

You also need ffmpeg (brew install ffmpeg, apt install ffmpeg). render also needs Node 18+.

Python 3.10–3.14. The sync extra needs faster-whisper, which has no Python 3.15 wheels yet, hence --python 3.13 above (Homebrew's python3 may already be 3.15). check, beats and render also run on 3.15.

Install only what you use — the parts differ a lot in size:

8 GB of RAM is plenty, and no GPU is needed. On a 12-core M4 Pro, a 3-minute song's sync (whisper small, three runs on the CPU) took 160 s at 1.4 GB at most; a 15 s render at 1080p took 20 s at 2.3 GB, Chrome's tabs included, and a draft of the whole song 24 s. A slower CPU takes longer. Everything installed takes about 2.2 GB of disk, of which about 1.2 GB downloads on first use. Tested on macOS (Apple silicon) and Linux; not yet on Windows.

Quick start

song2mv sync song.mp3 --lyrics lyrics.txt --language en --title "My Song"   # → song2mv-work/song/
song2mv render song2mv-work/song/mv.json --from 60 --to 75 --aspect both
song2mv check song2mv-work/song/renders/song-16x9-60-75.mp4 --song song2mv-work/song/mv.json --from 60 --to 75
song2mv sheet song2mv-work/song/renders/song-16x9-60-75.mp4 --song song2mv-work/song/mv.json --from 60 --every 1

The work folder is relative to where you run it: run it outside your code repositories, or pass --out. If the audio file has a lyrics tag (Suno writes one), --lyrics is optional. Want subtitles for CapCut, Premiere or DaVinci Resolve? Stop after sync and import song.srt or song.lrc.

With a coding agent

Open this repository (or your own project with song2mv installed) in Claude Code, Codex or another agent and ask, for example: "make a music video for ~/Music/my-song.mp3" or "give me word-timed LRC for this song".

• AGENTS.md — the operating rules: which command to run when, what to check by ear, what never to commit. Codex reads it, and Claude Code reads it through CLAUDE.md, when this repository is open.
• .claude/skills/song2mv/ — the full music-video loop (timing → reading the lyrics → scene design → render → check → fix) as a Claude Code skill. Copy it to ~/.claude/skills/ to use it in any project.
• Both ship in the package, in song2mv/guide/: song2mv --help (and the help of every command) prints the installed skill's path, under a line telling the agent to read it first, so an agent in any project finds it.
• Every command takes --json: progress goes to stderr and stdout carries one JSON result. See docs/contract.md.

song2mv sync

song2mv sync song.mp3 [--lyrics lyrics.txt] [--language ko] [--title "My Song"] [--out DIR] [--runs 5] [--model medium] [--bpm 90]
song2mv sync DIR/song.json [--from grid] [--runs 5]   # rerun after editing song.json; options given here are saved
                                                     # to it, finished whisper runs are reused

Why it works the way it does:

• Whisper gives a different answer every run. On a test song, two runs with the same file and settings found 342 and 380 words, and one of them skipped 15 seconds (five sung lines). So run k uses seed k and a different lead-in silence, and the runs are merged in lyric space: per line, only runs that matched at least half of it count, outliers more than 1 s from the rest are dropped, and each word takes the median.
• A chorus repeats word for word. Matched to the sheet by difflib alone, a run that heard the second chorus better gave its audio to the sheet's first chorus, and the lines in between were left with none: on the JamendoLyrics benchmark that was nearly all of the large error. Each run is also aligned over the whole song, which keeps each copy on its own audio; where the two alignments disagree, the one that matched more of that stretch wins. A line the runs still place out of the sheet's order is merged again from the runs that kept the order, and goes on check_by_ear.
• librosa.beat.beat_track can be off by a ratio (it reported 123 BPM for a 90 BPM song with a dotted-8th rhythm). The grid is fitted to percussive onsets instead, and auto tempo prefers the candidate where most gaps between lyric lines are whole bars, unless its grid is off the music for more of the song than the strongest onsets' grid is. The bar's first beat is the beat position with the strongest kick.
• The grid has one tempo, and songs do not always keep one. A live band drifts, jazz speeds up for a solo, a song slows down at the end, an AI-generated song can jump by a 16th note between sections. So tempo and phase are fitted again in windows of 4 bars, and where the music's beats are more than half a 16th note off the grid (a tempo change, a drift, or a beat that moved), grid.json lists the stretch under off_grid and the --json result says so under warnings. Where those windows can be chained beat by beat, as in AI-made songs that drift, grid.json gets a beat map (beats, the time of every beat): words snap to it, and the clicks and the video's beat effects follow it. Where they cannot (a beat that jumps, a weak pulse), lyric words in the stretch keep whisper's times instead of being snapped, and beat effects stay on the one tempo.
• Shouts, chants and whispers are what whisper misses. Lines no run placed are re-heard with just that stretch of audio and the expected words as the prompt. Those lines are marked reprompt and listed in the --json result under check_by_ear; pin anything wrong in manual. A line heard there more often than the lyric sheet has it takes the earliest hearing whose word timing is believable (whisper can stretch a word over seconds of music or time it at zero length), and warnings lists the other hearings.
• A pin only constrains. Pinning a line in manual splits the gap around it, and hearing the pieces again with shorter prompts moved lines that were right: on a test song, two pins sent the re-heard line after them 1.2 s early, into a pinned line. So a re-heard line keeps its time while it still fits between the pinned and placed lines around it, and only the lines a pin contradicts are heard again, in the window between the pins. Pin what is wrong; the lines you did not pin stay where they were.
• One run is not enough. A line only one of the runs placed can be seconds off, so it goes on check_by_ear too.
• A word no run heard is a guess. It is spread between the words around it, and where those are more than 1 s apart per word, half such words were more than 1 s off on the benchmark; that line goes on check_by_ear.
• A lyric sheet can leave out what was sung. One had a hook's couplet once that Suno sang twice: the couplet's lines were matched on both copies, one stayed on screen for 8 s with its last two words 6 s late, and nothing was listed. A line spread over far more time than it is sung in now goes on check_by_ear, off_sheet lists stretches where the runs agree on words that no line has (chants, ad-libs, changed lines), and where the sheet and the audio's lyrics tag differ, warnings says which of the two the runs support.
• Both can be wrong in the same way. Suno sang a repeated hook line as "Back it, back it up" where both say "Beep, beep": the line's last words match, so nothing looked off. When the runs hear a line alike and not as written, while two or more of them hear another copy of the same line as written, warnings names it and what they heard, and it goes on check_by_ear. Whisper can mishear a line the same way in every run, too (all 8 lines it names on the JamendoLyrics benchmark), so only an ear can tell which words are sung there.
• The runs can agree on a line and still be wrong about a word. Three words of a line at almost one time (the runs timed "Beep, beep" on "Brooklyn"), runs whose starts for a line are more than 0.7 s apart, and a word whose merged time is more than 0.5 s from every run's time for it (two runs that disagree, merged to their middle) put a line on check_by_ear: on the JamendoLyrics benchmark, 56 of the 65 lines these add have a word more than 0.3 s off (of all lines, 37 % have). Two words at one time are listed where another copy of the line has them 0.2 s apart or more ("beep" and "Brooklyn" on one 16th in one hook, 0.33 s apart in the other); alone they are not, since fast singing puts two words on one 16th all the time.

To hear whether the grid and timing are right, mix a click track over a stretch — a high click on the first beat of each bar, a low click on the other beats and, with --words, a soft tick on every lyric word:

song2mv click song2mv-work/song --from 90 --to 110 --words    # → song2mv-work/song/click-90-110.wav

Song config (song.json)

sync writes a minimal song.json (the audio, a title and the options you passed) into the work folder. Add or change keys there and rerun song2mv sync DIR/song.json. Only audio is required; paths are relative to the file. See examples/song.example.json.

Line ids are <section>-<nn>: in-01, v1-03, pc2-01, ch1-05, br-02, ou-01. A section name that appears more than once without a number ([Chorus] twice) is numbered by occurrence (ch1, ch2); one that appears once keeps the bare id (ch-01 in a song with one chorus).

song2mv beats

song2mv beats song.mp3 [--out song.grid.json] [--bpm 128] [--downbeat 0]

Tempo, first beat and bar start for cutting video to the beat — no lyrics needed. Without lyrics, auto tempo picks the candidate with the strongest onset score, so check the result by ear or pass --bpm. Stretches where the song leaves that one tempo are in off_grid (and warnings): cut to beats there when the file has a beat map, and don't cut to the grid there when it has none.

song2mv check (experimental)

song2mv check video.mp4 --song mv.json [--from 92 --to 102] [--goals my-goals.json] [--src "video/src/scenes/**/*.tsx"]

--from is the song time of the video's first frame; the measured stretch ends with the video unless --to ends it earlier. Checks that cannot run on a stretch (no repeated section in it, too few cuts) are listed as skipped, not passed. It works on any mp4 that follows the song's timing, not only on song2mv renders. A failing line says where: the stretch measured, the cuts in it, and its still or empty part, or for a frantic line its biggest frame change and its motion without the cuts (without a cut: whether the whole frame changes, as under a camera drift over fine detail). A long stretch without a cut names the sudden changes in it that did not count as one, with their correlation and difference: a reframe of a smooth picture can stay too alike to count, and one of a dark picture too dim.

The default goals are a starting point measured on one video; your own goals file only needs the values you change. Passing is necessary, not proof of fun.

song2mv sheet

song2mv sheet video.mp4 --song mv.json --every 3                  # a frame from the middle of every 3 s
song2mv sheet video.mp4 --song mv.json --from 92 --times 93.1,97.6   # frames at these song times
song2mv sheet video.mp4 --song mv.json --lines v1-03,ch1-02       # a frame in the last word of each line
song2mv check video.mp4 --song mv.json --json > check.json
song2mv sheet video.mp4 --cuts check.json                         # a frame 0.3 s into each shot
song2mv sheet video.mp4 --every 2 --crop 700,380,312,400 --cols 3 # one region, zoomed

A contact sheet of the video as rendered, captions, title and strips included: a way to review a video without watching it. Each tile is the frame on screen at its song time, labelled with that time and, with --song, the id of the line sung then (or of the section between lines). --from is the song time of the video's first frame, as for check; without it, a stretch keeps the start in the name render gave it (song-9x16-49.72-72.2.mp4 starts at 49.72). --cuts reads the start and the cuts of a saved check --json result and also takes its --from and --song. --lines ID,ID takes a frame in the last word of each line named, halfway from the word's start to the line's end in --song (the line's middle without word times). --crop X,Y,W,H shows one region of each frame, in the video's pixels, scaled to the tile width (--cols sets the zoom; the label shows it). The sheet is a PNG 1920 px wide, video-sheet.png next to video.mp4 unless --out names it; frames that do not fit on one page, about 2400 px tall, continue on video-sheet-2.png and so on. The labels come from a built-in bitmap font, so it needs nothing but ffmpeg and numpy: no drawtext, no image library.

song2mv render (experimental)

song2mv render mv.json                                        # whole song, 16:9, plain lyric video
song2mv render mv.json --from 95 --to 111 --aspect 9x16 --hold 2.5   # a hook cut whose first frame shows the payoff
song2mv render mv.json --from 95 --to 111 --draft             # 10 fps at half size, for the render → check loop
song2mv render mv.json --still 98.5 --aspect both            # PNG frames to look at
song2mv render mv.json --max-mb 200                          # the finished video, at most 200 MB (an upload limit)
song2mv render mv.json --cover 3.2                           # a 1080×1080 cover from the frame at 3.2 s
song2mv render mv.json --template video/                     # your own copy of the template, for your own scenes

--draft renders the same frames at 10 fps, the rate check measures at, and at half size (960×540), and names the file …-draft.mp4. On 30 s of a paper-cut scene it rendered in 15 s, against 55–70 s at full size and about 2.5 min for the finished video, and check gave drafts the same passes, fails and cuts as full-size renders. Iterate on drafts and render the finished video once without it.

A finished video is capped at 8 Mbit/s, the rate YouTube asks for at 1080p30. Flat styles stay far below it; the paper pack's grain ran near 14 Mbit/s, and at the cap a 4-minute video is about 250 MB instead of 440. --max-mb N encodes a video that is still over N MB again, in two passes, to fit an upload limit, and keeps its sound. --cover T writes a square cover for album and playlist pages: the 16:9 frame at song time T, whole, over a blurred fill of itself. Next to them, sync already wrote song.lrc (line times, which most players and music platforms import), song.words.lrc (word times) and song.srt (subtitles).

Out of the box you get a plain lyric video that follows the song's structure: calm sections with a low moon and embers, mid sections with a floor grid rolling on the beat, hot sections with rays on the beat and the line itself huge in the middle, lit word by word. Colours come from theme in the song config.

The real music video is your own scenes, with a look made for the song. Start from a concept: two or three ideas in different art styles, a rough still of each, and pick one (the skill walks through it). With --template DIR, DIR gets its own copy of the template: write the scenes in DIR/src/scenes/, reuse and change what fits the concept from the kit in DIR/src/kit/, and draw the rest. Besides useSong(), beat-grid helpers, Slam, BigLine and karaoke Captions, the kit holds:

• people: a posable person (standing, walking, running, lying), a crowd;
• places: a city, a street, a room, a bed, hills, the sea;
• sky: sky by time of day, sun, moon and clouds;
• objects: a phone with a chat, a clock, a heart, a car, a plane, lanterns;
• camera: one world filmed as shots that cut on bars and words, with a punch on hits;
• effects: rain, fireworks, confetti, hits (burst, sparkle, ring), flames, lightning and storms, tears, falling cash. Their colours and shapes are props, so they fit any art style;
• recipes: new assets combined from these parts in a few lines, such as a sleepless night in a storm;
• style packs (kit/styles/): drawing pieces that put any object in one art style. paper cuts outlines by hand and stacks them with shadows and grain; pixel draws sprites on a canvas of big pixels, big outlined type (Korean too) and a game dialog box that letters the line as it is sung, and turns any kit drawing into the same pixels; comic inks drawings over off-register colour and Ben-Day dots, characters in one heavy outline with tapered inner lines, in panels that each film the scene with their own camera, with speech balloons and captions lettered as the line is sung; toon puts one ink outline round any drawing, kit pieces included, with shading and shines; line traces any drawing as ink on paper, chalk on a blackboard, a blueprint or neon on brick; paint turns any drawing into watercolour over pencil; finish lays an old film, a VHS tape or a halftone print over a whole frame. New assets and packs are welcome (CONTRIBUTING.md).

The people, places and objects are drawn in one flat style. To see everything, run node render.mjs src/sample-song.json --composition Kit-Gallery in DIR. Rules that held up: a scene is a pure function of song time T; text pops once and then holds still while it is read; lay out on the 1920×1080 stage. For 9:16, a song can also export a VerticalScene that draws the whole 1080×1920 frame, instead of the stage shrunk into a band; the kit's verticalView films the 16:9 world for it, its subject above the captions and inside the apps' safe box. In 16:9 a storyboard films through wideView, which keeps each shot's subject above the captions in the same way.

Remotion is not MIT: it is free for individuals and companies with up to 3 employees; larger companies need a company license (remotion.dev/license). Only render uses it.

How it compares

Other open-source projects turn songs into videos in five broad ways:

In short, song2mv is a timing layer — word-timed lyrics on the beat grid, as plain files — plus a code-drawn video that an agent can measure and fix. grid.json and lyrics.json can drive cuts in other tools too, generative ones included.

Where it is weaker today:

• Timing: on the JamendoLyrics benchmark (79 songs in English, German, Spanish and French), 87 % of a song's words land within 0.3 s of where people put them, on average; the best dedicated aligner reaches 94 % (docs/benchmark.md). Most wrong line starts are a line's first word alone, early or late by a fraction of a second, and lines whisper hears badly (vocalisations, shouts, chants) can still be far off and need a few manual values. Separating the vocals first, as karaoke tools do, made the timing worse on our Korean test song.
• No generated imagery: out of the box you get a plain lyric video, and anything richer is scene code that you or your agent write.
• No karaoke formats (CDG, karaoke ASS).
• render needs Node and Remotion (a company license above 3 employees) and takes ~700 MB of disk on first use.

Roadmap

• A basic render without Node (Python + ffmpeg), so everyone gets a lyric video with no extra install.
• check measured against the song itself (music energy vs. motion, chorus vs. verse) instead of fixed numbers.
• An MCP server (song2mv mcp) for agents and apps without a terminal.

Rights

This repository contains no songs and no lyrics. Before you publish a video, make sure you may: the song's licence (including an AI music service's plan terms) and the lyrics' copyright apply to the video too.

License

MIT — see LICENSE. song2mv is developed with the help of coding agents (Claude Code, Codex).