fx-dub 1.2.0

English | 日本語 | 中文 | Español | Français | हिन्दी | Italiano | Português (BR)

Verify a generated dub before anyone hears it.

Your text-to-speech model returned 48 kHz stereo at exactly the right duration and a textbook −18 LUFS. It also said a line you never wrote, in a voice that isn't your character's, with a two-second hole in the middle.

None of that is visible to sample rate and duration. fx-dub gives you two receipts — one for the container, one for what was actually said — and exits non-zero when either fails.

pip install fx-dub

$ fxdub-dialogue docs/scenes/night-street.json words.json --only-speaker VOICE
9/10 checks pass
| PASS | line_present:0:VOICE    | Hey, how's it going?
| FAIL | no_invented_speech      | 4 unscripted word(s): not bad can't complain
| PASS | no_overlap              | clean
| PASS | no_internal_straggle    | clean
| PASS | one_voice_per_character | clean

That failure is real. A model in audio reference mode reproduced its reference clip's dialogue, not just its timbre — so a stem meant to carry one character's lines quietly re-spoke the other's. Mixed under the real take, it sounded like two men talking over each other. Sample rate: perfect. Duration: perfect.

The two receipts

A failing check is a finding, not a bug in the tool. Report it; never tune the threshold to make it green. Every check cites the standard or the measured defect it traces to, so you can argue with it on the evidence.

The scene script is the contract

Direction lives in the script, not in an agent's head:

{
  "clip_duration_s": 10.062,
  "cast": {
    "VOICE": { "description": "off-frame, deep and gritty", "on_frame": false },
    "MAC":   { "description": "on-frame, gritty, weary", "on_frame": true,
               "face": { "frame": 60, "x": 348, "y": 122 } }
  },
  "lines": [
    { "speaker": "VOICE", "text": "Hey, how's it going?" },
    { "speaker": "MAC",   "text": "Not bad. Can't complain.",
      "max_gap_s": 0.15,
      "direction": "There's no pause in between. A gap here runs into VOICE's next cue." },
    { "speaker": "VOICE", "text": "Good to hear, good to hear." }
  ]
}

max_gap_s on that line is why the verifier rejects a take a global threshold would wave through. The note beside it is why the number is 0.15 and not something else.

on_frame is what lets a package with no eyes catch a caption defect. Pass --scene to fxdub-receipt and it compares the number of people the caption claims against the number the contract declares visible. On the delivered run that check fails: the captioner wrote "two men … facing each other" over a one-man shot, and that caption is what feeds the audio prompt.

--only-speaker MAC narrows the contract to one character, which is how you check a per-character stem: it should carry that character's lines and silence where anyone else talks. Checking a stem against the whole scene hides exactly the bug above.

Getting a transcript

fxdub-dialogue reads a word-level diarized transcript — {text, start, end, speaker_id} per word. Any diarizing ASR will do. fxdub.vo_graphs.transcribe() builds the ComfyUI graph for one:

from fxdub import vo_graphs

graph = vo_graphs.transcribe("<storage-key>.flac", "run/words")
# -> API-format dict, ready for your own submit path. Nothing is sent from here.

The public API — fxdub.verify

New in v1.2.0. The two console scripts verify a video dub. The core underneath them — match a script against what was actually spoken — is not specific to video, and it is exported as a declared, stable surface so other tools can build on it instead of importing internals that may move.

from fxdub import verify

lines = [{"speaker": "narrator", "text": "Chapter two continues the tale."}]
result = verify.align_lines(lines, words)      # words: the transcript above

result.missing            # scripted lines that were never spoken
result.invented_words     # rendered tokens no scripted line claimed

verify.check_all_lines_present(result)   # -> Check(name, ok, detail, traces_to)
verify.check_no_invented_speech(result)
verify.check_one_voice_per_line(result)

Those three transfer to any pipeline that turns authored text into generated speech: a line the script asked for must be spoken, a line it did not ask for must not be, and a character must be rendered by one voice, their own. They were proven against a second codebase before being declared — four content defects measured in an EPUB→audiobook renderer (a narrated footnote sentinel, a chapter dropped by a partial render and reported as success, three characters collapsed into one voice, markup read aloud as prose) are fixtures in tests/test_verify.py, beside a clean control that must report nothing.

What is deliberately not in the API: transcription (fx-dub consumes word timings it is given — that is why it has no dependencies), thresholds (every one of them traces to a defect caught by ear in one medium, so they belong to the caller that owns the policy), and the medium-specific checks — overlap, mid-line straggle, clip fit, ducking depth, caption-vs-cast. A check that cannot meaningfully fire in your medium is worse than no check, because it reads as a pass.

fxdub-dialogue is itself a consumer of this API — it computes its content verdicts through verify and adds only what makes a receipt a dub receipt. That is enforced by tests that break verify and require the CLI to break with it, so the public path cannot quietly become one nobody exercises.

Graph builders

fxdub.vo_graphs also builds the VO-stage graphs: voice design, same-engine audio reference, clone-and-speak, splice, place-on-timeline, mix — and the picture stage: frame extract, lip-sync, and mux. They exist because the alternative — hand-typing API JSON into a chat window — produces graphs that vanish with the session and quietly reintroduce defects already paid for once.

Every builder is linted by the repo's trap detectors, so the shapes that cost real failed jobs cannot be re-authored by accident. Two examples of what that encodes:

• The ElevenLabs clone node's auto-grow input is addressed as files.audio0 at runtime — not the files.item_1 its own schema advertises — and a dry run accepts the wrong name without complaint.
• ByteDance's pitch_rate is node-global, so one node cannot voice two characters at different pitches. Its timestamps address an absolute output timeline, so the fix is one pass per character, layered.
• The lip-sync node's speaker_selection defaults to let the model decide. Leave it unpinned and the job completes, returns a correctly-framed MP4 at the right duration, and passes every container check — with the wrong person's mouth moving. The builder pins coordinates; the detector fails the graph that doesn't.

Building a graph is a pure function from arguments to a dict. Nothing in this package submits, uploads, or spends.

Threat model

fx-dub runs locally and makes no network calls of any kind.

• Data touched: only the files you name on the command line — FLAC/MP4 masters, LUFS manifests, caption text, transcript JSON. It writes one receipt, at the --json path you choose.
• Data NOT touched: no credentials, no API keys, no environment secrets, no files outside the paths you pass.
• Permissions required: filesystem read on the inputs; filesystem write only if you pass --json.
• Network egress: none. There is no HTTP client here and the runtime dependency list is empty by design — CI fails the build if that ever changes.
• Telemetry: none. Nothing is collected, counted, or transmitted.

Media parsing is standard-library only: FLAC STREAMINFO and MP4 atoms are decoded directly rather than shelling out to ffprobe. Malformed input yields a failed check, not a crash. Full policy in SECURITY.md.

Exit codes

1 and 2 stay distinct on purpose: in CI the first wants its receipt read, the second means the invocation is wrong. Errors print {code, message, hint} on stderr; --debug re-raises instead.

The pipeline these receipts verify

fx-dub began as a ComfyUI-native dubbing pipeline and still is one. It runs on Comfy Cloud:

video ─► describe (Florence-2, pinned, single mid-clip frame)
              │ caption.txt
              ▼
        audio prompt (positive claims only — negation collapses in audio-text models)
              ├──────────► ambience bed (ElevenLabs eleven_sfx_v2, 48 kHz, exact duration)
              │                    │ stem_bed.flac
   your script ──────────► dialogue (per-character passes, layered on an absolute timeline)
                                   │ stem_vo.flac
                                   ▼
                    mix bus (48 kHz · dialogue-anchored · −18 LUFS)
                                   │ mix.flac + LUFS manifests
                                   ▼
                        re-mux ─► dubbed.mp4
                                   │
                       (optional)  ▼
                    lip-sync ─► sync one named face to that
                                character's own track, then
                                re-mux the full mix back over it

"Re-mux" = re-multiplex: the finished soundtrack is written back into the video container, pixels untouched. Not a typo for "remix" — the mixing happens one stage earlier; this is the step that hands you a playable dubbed.mp4.

Gain-stage from the meter, never from remembered numbers. Engines differ by 8 dB on the same line: swapping one TTS for another moved a VO stem from −18.34 to −25.03 LUFS. Reusing the previous recipe's fixed gain would have buried the dialogue by 7 dB while every other check stayed green.

What's honest about this design

• Captions carry meaning, not timing. A caption-mediated pipeline is ambience- and dialogue-grade; it will never sync a door-slam by prose alone. Impact-grade timing needs an event timeline — the Knowledge Base maps the direct video→audio models that do it natively, and their licences.
• A scene description is not a script. You write the words your characters say; the pipeline makes them sound right.
• Voice identity is not free. Prompt-designed voices are non-deterministic regardless of seed — a voice you approve cannot be recalled by re-running the same prompt. Cast once, keep the approved audio, then reference or splice it forever after. Cross-engine cloning does not preserve identity either. This is the most expensive lesson in the repo's trap ledger, and the verifier's one_voice_per_character check is how it stays learned.
• Lip-sync drives one face, so it needs one character's track. Feed it the mix and it will mouth every line — including the ones belonging to someone who is not in the shot — and still pass every audio check, because the audio never changed. Feed it a per-character track and silence becomes the correct performance: the character listens. Results are non-deterministic regardless of seed, so an approved take is kept, never re-rendered. The node also re-times the picture; assert frame count on the deliverable, not on its raw output.
• Mix numbers come from standards and listening studies (BS.1770-5, AES TD1008, JAES ducking research), not vibes — and they're knobs, because preferences measurably differ.
• Governance is a feature. Do not clone a real person's voice without consent. Synthetic speech published in the EU carries an Article 50 machine-readable-marking obligation; the receipt JSON is built to be part of that provenance trail, and the KB's publishing section tells you what disclosure you owe where you post. No person-specific voice packs, ever. Not for robocalls.

Status

v1.2.0 — the pipeline is delivered, both receipts are green, the picture is lip-synced, and the alignment core is now a declared public API. A two-character night-street scene scores 19/19 on the container contract (48 kHz, −18.09 LUFS, dialogue +11.17 LU over the bed, 161 frames intact, 10.069 s) and 11/11 on the content contract. The lip-synced variant holds the same contract — 832 × 480, 161 frames, both tracks — with MAC's mouth on his line and closed while the off-frame character speaks.

It scores 19/20 once you pass --scene, and the failure is real: the delivered run's caption claims two men over a one-man shot. That check caught a defect that had been shipping green. 284 tests, CI green. Full history in the CHANGELOG.

This release adds fxdub.verify — the declared, medium-agnostic surface other tools build on — and closes two defects that nothing in the suite could see: a committed receipt carrying the absolute path of the machine that wrote it, and a CI trigger that ran the full matrix twice on every release. Both now have detectors, each proven red against the real pre-fix bytes.

For agents and LLMs

Start at AGENTS.md — the durable operating manual — then HANDOFF.md for live state, then query kb/fxdub.db for the trap ledger. A machine-readable summary is published at /fx-dub/llms.txt.

Provenance

This repo practises receipts-first development: graphs are pulled from the platform and verified (billing feed, decoded output headers) rather than trusted from reports; design citations pass an external different-family verifier before they become architecture; measured numbers carry their job UUIDs. When a trap is found, the same commit adds the detector, the database seed, and the test.

License

MIT — the repo and the package. Model weights carry their own licences; the Knowledge Base is the honest map. © 2026 mcp-tool-shop.

Built by MCP Tool Shop