lyric-align

You already have the correct lyrics. You only need the times.

lyric-align anchors known lyric lines onto ASR word timings by character-level fuzzy matching, and emits line- or character-level timestamps as LRC / SRT / ASS / JSON.

It is built for two things most aligners handle poorly:

• Space-less languages (Japanese, Chinese). Word-level tools split on whitespace, which does not exist here. lyric-align compares characters, so it works where str.split() fails.
• Sung vocals, including rap. ASR mis-hears lyrics constantly, but you don't care about its text — you have the official lyrics. You only borrow its timing. Character-level fuzzy matching tolerates the ASR errors and keeps the timestamps.

Install

Python 3.9–3.14. The core has no dependencies at all; the ASR extra pulls in faster-whisper (ctranslate2, not torch).

As a command-line tool (recommended — puts lyric-align on your PATH, in its own isolated environment):

uv tool install "lyric-align[asr]"
# or: pipx install "lyric-align[asr]"

lyric-align --version

Add separate to the extras (lyric-align[asr,separate]) if you want Demucs vocal splitting; it pulls in torch, so leave it out until you need it.

As a library, into your own environment:

pip install lyric-align            # core only, pure stdlib — no dependencies
pip install "lyric-align[asr]"     # + faster-whisper, to transcribe audio

If you already have segments with word timings from somewhere else, the core install is enough: feed them in with --segments and nothing gets downloaded.

Try it in one minute

Both the recording and the lyrics below are public domain, so this runs end-to-end with nothing of your own:

curl -L -o amazing_grace.mp3 \
  "https://upload.wikimedia.org/wikipedia/commons/8/8f/Amazing_Grace_%28vocalist_with_guitar%29_-_Southern_Aire_-_United_States_Air_Force_Reserve_Band.mp3"
# examples/amazing_grace.txt ships with this repo
lyric-align amazing_grace.mp3 examples/amazing_grace.txt \
  --language en --pairing 1 --no-vad -o amazing_grace.lrc

segments: 12
match threshold: 0.5 (alphabetic script)
aligned 12/12 lines

[00:06.60]Amazing grace, how sweet the sound
[00:14.30]That saved a wretch like me
[00:23.70]I once was lost, but now am found
[00:34.52]Was blind, but now I see
...
[01:46.24]We've no less days to sing God's praise
[01:57.46]Than when we first begun

Note --no-vad: this is a slow hymn, and the ASR's voice-activity filter mistakes sustained singing for silence. With the filter on, the same file yields one garbage segment for 130 seconds; with it off, twelve clean ones. Keep the filter for rap, drop it for anything sung slowly. (The --pairing 1 is what auto picks here anyway — this ASR already split one lyric line per segment.)

Drop --no-vad and you can watch the honest-gap contract hold: every line is reported unmatched, nothing is written, and the cause is named.

only 0/12 lines matched from 1 segments — try --no-vad (the voice-activity
filter silences slow singing), or --separate (a full mix hides the vocal
from the ASR)

Use

# transcribe audio and align known lyrics → LRC
lyric-align song.wav lyrics.txt -o out.lrc

# full mix? split the vocal first — this matters a lot (see below)
lyric-align song.wav lyrics.txt --separate -o out.lrc

# already have Whisper segments? skip ASR
lyric-align --segments segments.json lyrics.txt -f srt

# keep the transcription, so fixing a typo doesn't re-run Whisper (or Demucs)
lyric-align song.wav lyrics.txt --separate --dump-segments segments.json -o out.lrc
lyric-align --segments segments.json lyrics.txt -o out.lrc        # seconds, not minutes

# per-character karaoke ASS (\k tags)
lyric-align song.wav lyrics.txt -f ass --karaoke -o out.ass

lyrics.txt is plain text, one lyric line per line. Blank lines, # comments and section markers ([Verse 1], [Hook]) are skipped, so a pasted lyric sheet works as-is. segments.json is a list of {"start", "end", "text", "words": [{"start","end","word"}]} — the shape any Whisper flavor produces, and the shape --dump-segments writes. The dump lands before alignment, so a run that dies on the cheap half still leaves the expensive half on disk.

Output: pick the format your next tool eats

-f all -o out writes every format at once.

Per-syllable formats split by script: alphabetic text is grouped into words (<00:06.60>Amazing <00:08.82>grace), CJK stays one unit per character (<00:21.78>治<00:21.94>部).

Recipes

# burn subtitles into a video
ffmpeg -i video.mp4 -vf "ass=out.ass" -c:a copy out.mp4

# karaoke sweep instead of plain lines
lyric-align song.wav lyrics.txt -f ass --karaoke -o out.ass

# fix a mistimed line by hand — the correction loop, in three steps
lyric-align song.wav lyrics.txt -f aud -o out.labels.txt   # 1. export labels
#   2. Audacity: File > Import > Labels, drag the wrong line over the waveform,
#      then File > Export > Export Labels
lyric-align --from-labels out.labels.txt -f lrc -o final.lrc   # 3. back to LRC

# web player
lyric-align song.wav lyrics.txt -f vtt -o out.vtt

# a file shaped for an AMLL TTML DB submission
lyric-align song.wav lyrics.txt -f ttml -o out.ttml \
  --meta musicName="Song" --meta artists="Artist" --meta album="Album" \
  --meta ncmMusicId=1234567

Feed it a vocal stem

Alignment quality is dominated by this one choice. Same track, same settings, 20 human-marked lines — only the input differs:

On the full mix the ASR returned 11 segments for a 4-minute song instead of 41, merging whole sections, and everything past the first chorus went unmatched. Use --separate (or point the tool at a stem you already have). Separation is the slow step, so the stem is cached and reused.

Library

from lyric_align import align, Segment

segments = [Segment.from_dict(d) for d in whisper_output]
aligned = align(segments, lyric_lines, karaoke=True)  # pairing="auto"
for a in aligned:
    print(a.start, a.matched, a.line)

Design: honest gaps over silent drift

When a line can't be confidently matched, lyric-align marks it unmatched rather than inventing a timestamp. Forced aligners always emit an answer and so fail silently — drifting into the intro, or smearing a chorus. For a review-in-the-loop workflow (subtitling, MV production) an honest gap you can see beats a wrong number you can't. Pass --interpolate if you do want gaps filled (they stay flagged as guessed).

How it works

1. (optional) vocal separation — Demucs, if installed ([separate]).
2. ASR — faster-whisper with word timestamps ([asr]), or bring your own segments. --dump-segments writes steps 1–2 out, so iterating on the lyrics never pays for them twice.
3. anchor — each known lyric line/stanza is matched to a segment by character-level similarity, scanning forward monotonically so repeated choruses consume segments in order.
4. breath split — when one segment covers several lines, they're cut at the largest inter-word silence (a breath), near the split point implied by character counts.
5. char map — known characters are distributed across the word span by proportional interpolation, for karaoke \k timing.

The core (steps 3–5) is pure Python standard library — no numpy, no torch. The heavy pieces (Whisper, Demucs) are optional extras behind lazy imports.

Where this came from

This was written by TORYU (屠龍), a Japanese artist who writes the music, builds the videos and designs the software alone, to put lyrics on the timeline for a set of music videos, where the lyrics are Japanese and the delivery is rap. That is the whole reason the matcher compares characters instead of words, and why the accuracy below is measured on sung Japanese rather than on read speech.

The three tracks the numbers come from are 過ぎたるもの (20 lines, ±0.5 s ground truth), 黒砂の誓い (33 lines, 1 s granularity) and 誠の虎徹 (38 lines, from the video's own beat grid). Others from the same catalogue: 六の巷, 永遠の炎 — toryu.tokyo.

Three is not thoroughness, it is the minimum that stops a change from looking good. With two tracks every proposal traded one against the other and the call came down to which track was weighted more; the third was added to take that discretion away. It did, but not by arbitrating — it abstained. Seven of the eight candidates leave it bit-identical, because it has nothing broken for them to repair, and that turned out to be the more useful answer: the trade the two tracks appeared to show was never a trade. See Known limits.

Short lyric fragments from those tracks appear in the test fixtures. They are the author's own work and are not covered by this project's MIT license — see NOTICE. The runnable example (examples/amazing_grace.txt) is public domain, so anyone can reproduce it end to end.

Accuracy

Measured against 20 human-marked lyric lines of a 4-minute Japanese rap track (±0.5 s ground-truth precision), aligning from a Demucs-separated vocal stem:

lyric-align and stable-ts are practically equivalent: they split the columns, and every gap between them is smaller than the ±0.5 s the ground truth was eyeballed to. WhisperX's batched VAD merges whole verses into single segments, which is fine for captions but loses line-level timing (and it can't take known text as input).

A second track (3-minute Japanese rap, 33 human-marked lines, 1 s ground-truth granularity) reproduces this: 33/33 matched, median |err| 0.30 s, 26/33 within 0.5 s. Its mean of 0.94 s comes almost entirely from four outliers, all on the same line — see below.

A third (3.5-minute Japanese rap, 38 lines) is the mildest of the three: 36/38 matched, mean |err| 0.35 s, median 0.32 s, 29/38 within 0.5 s, and a worst case of 0.81 s — no multi-second outlier anywhere, including on a hook line that repeats six times.

The ground truth on all three tracks is marked per two-line pair, so these figures score the first line of each pair — 20, 33 and 38 lines, not every line placed. Second lines have a weaker but free check: each must start inside its pair's window. The shipped configuration passes it (19/19 and 29/33, the four misses being the repeated hook already described), so the tables are not hiding a second failure mode — but the distinction matters as soon as a change is judged on how many lines it places, because placements land mostly on the unscored half. One did, and it is the last entry in Known limits.

A bigger ASR model is a trap unless the pairing follows

Lines are matched in stanza units, and pairing says how many lyric lines make one unit. That is not a property of the song — it is a property of the ASR's segmentation, and models differ. Upgrading medium → large-v3 at a pairing tuned for medium is worse than not upgrading; with the pairing it deserves, the same upgrade removes the worst outlier entirely. This is why the default is --pairing auto, which reads lines-per-segment off the ASR output.

Against Vilm, the one other maintained tool here

Vilm Lyrics Aligner solves the same problem for a different audience — live performance, Korean/English code-switching, SRT into DaVinci Resolve, with a GUI and a Resolve panel where this has a CLI. On two Japanese rap tracks we split the columns when there are no repeats; with a four-times-repeated hook this tool is ahead two to one. Notably, character-level matching is not the differentiator — Vilm compares characters too. What differs is a script-aware threshold and locality instead of global optimality.

Coming from stable-ts?

stable-ts was archived on 2026-05-30, its last commit being "Add note about paused development". The version pip installs is older than that: 2.19.1, from 2025-08, which predates its own final alignment work (committed 2025-10, never released).

It did this job well, and this is not a claim to have beaten it. It is also not a claim to have lost — the head-to-head splits, and every gap in it is smaller than the ±0.5 s the ground truth was marked to. What actually differs is everything around the accuracy: the install (torch, unconditionally, vs. nothing for the core), the matching (word-level forced alignment vs. character-level fuzzy anchor), and what happens to a line it cannot place — always given a time, vs. reported as unmatched.

# stable-ts
import stable_whisper
model = stable_whisper.load_model("medium")
result = model.align(audio, "\n".join(lines), language="ja", original_split=True)
result.to_srt_vtt("out.srt")

# lyric-align — format inferred from the extension
lyric-align audio.wav lyrics.txt --language ja -o out.srt

Match threshold is script-aware

A line is accepted when its character similarity clears a threshold, and the right floor depends on how many characters the language has to choose from. So the default is picked from the lyrics themselves (--threshold overrides it):

Using the CJK floor on English silently invents matches — measured on the hymn above, "Through many dangers, toils and snares" was placed on the line "We've no less days to sing God's praise" (similarity 0.34).

Known limits

Every sweep below has now been re-run on all three tracks. The two-track figures quoted in each table are unchanged; the third track did not overturn a single rejection. What it did instead is worth reading, because it is not what the two-track framing predicted:

Seven of the eight candidates do nothing at all on the third track — bit-identical to what ships, at every value swept. That includes λ = 1.2 on the timing prior, a setting strong enough to collapse the first track from 19/20 placed to 1/20. The third track has no misplacement to repair (worst 0.81 s, zero outliers), and every one of these candidates is a repair mechanism, so none of them has anything to fire on.

That reframes the rejections. Two tracks made this look like a trade — a gain here paid for by a loss there, with the weighing left to whoever read the table. It is not a trade. The upside exists only on tracks that are already broken, while the downside — a greedy scan whose idx carries every decision into its neighbour — applies everywhere. The eighth candidate, word-start snapping, is the one exception to the inertness, and it is worse on the third track too (mean 0.35 s → 0.48 s, ≤0.5 s 29/38 → 20/38), making it worse on three of three.

Heavily repeated refrains can land on the wrong repetition

A hook line sung four times is four identical strings; if the ASR segments the repeats unevenly, the forward scan can consume the neighbouring one. On the second track, one 4×-repeated hook line produced errors of +6.4 s, −3.8 s, −4.5 s and +5.7 s while every non-repeated line stayed within ~0.5 s. Check hook sections by hand, or align verses and hooks as separate passes.

Repetition alone does not cause this. The third track repeats two hook lines six times each — twelve of its thirty-eight lines are duplicates — and none of them breaks: the worst is 0.81 s, and the repeated lines are no worse than the rest of that track. What differs is the ASR: where the repeats are segmented evenly, the forward scan consumes them in order and the ambiguity never has to be resolved. So the failure needs repetition and uneven segmentation together, and you cannot predict it from the lyrics alone — check the hooks on a track where the numbers matter, rather than assuming either outcome.

Two obvious fixes were implemented and measured. Both are worse than shipping the limitation, and the reason generalises: identical repetitions carry identical similarity, so no better search over similarity can tell them apart.

Correcting one placement tends to break the next one

The first track's worst case (3.6 s) has a fully diagnosed cause, and four independent fixes follow from it. All four fix the outlier; all four cost more elsewhere than they return. The mechanism is the greedy scan itself: idx advances past whatever was chosen, so every neighbour is downstream of every decision, and gains and losses arrive in adjacent pairs.

A varying unit size places more lines, and some of them badly

pairing is a rounded average, so it is wrong for part of any track. Choosing (lines, segment) jointly at each step fixes the placement count and looks nearly free — until you score the half of the lines the shipped metric does not look at, where one of the extra placements is 13.3 s wrong.

The ASR is the floor, and singing can put it there

• Slow, sustained singing is much harder than rap — hymns, ballads and school songs stretch vowels until the ASR stops producing usable segments. Reach for --no-vad first (see the one-minute example); dense, consonant-rich delivery is the sweet spot. This is an ASR limit, not an anchoring one.
• A quiet or lo-fi recording can defeat the ASR entirely. On a −33 dBFS amateur recording of an unaccompanied Japanese art song, Whisper returned zero segments, and returned 音楽 ("music") or a row of repeated single characters once its silence thresholds were relaxed — it classified the singing as music rather than speech. Loudness-normalizing to −16 LUFS did not help. When the transcription is empty there is nothing to anchor to; check for segments before blaming the alignment.

The nearest match, stable-ts, was archived in 2026-05 — see Coming from stable-ts? for what carries over and what does not.

License

MIT, for the code, tests and documentation. The song lyrics quoted in the test fixtures are not covered by it — see NOTICE.