nw

Narrative Workflow — the substrate audiovisual production apps are built on.

A project is a folder. A genre — music video, explainer, commentary weave, slideshow — is the reusable specialization on top: pure data, declared over the substrate, carrying no engine of its own. nw owns the engine: the typed project facade, the prepare → plan → execute split with a cost gate, the Transform contract, an async job layer, a provenance graph with freshness queries, and QA reports. Apps (reelee, muvid, braidio) supply their own body schemas, Transforms, and genres — without modifying nw.

import nw

# Declare a kind of production (apps do this once, in their own package).
nw.register_genre(
    nw.Genre(
        slug="slideshow",
        title="Slideshow",
        description="Stills over narration, assembled to a video.",
        transform_names=("clips_to_animatic.ffmpeg",),
        projection_entrypoint="clips_to_animatic.ffmpeg",
        templates=(nw.Template(slug="lecture", title="Lecture deck"),),
    )
)

nw.list_genres()  # ['slideshow']
nw.genre_catalog()  # JSON-able: what a CLI / HTTP route / MCP tool serves
nw.resolve_genre(
    "slideshow", "lecture"
)  # {'genre': ..., 'template': ..., 'params': {...}}

Install

pip install nw

Genre and Template — the central abstraction

A Genre is a reusable definition of a production kind. It is a frozen dataclass that references substrate pieces by name rather than owning them:

A Template is a named preset within a genre — a filled-in default configuration ("Deep Dive", "Children's book", "Math explainer"). The substrate owns a Template's identity (slug / title / description) and carries a genre-defined params payload it deliberately does not interpret; the app that owns the genre validates and resolves those params. That keeps a genre self-describing for any consumer (a CLI, an HTTP catalog, an MCP connector) while app-specific meaning stays in the app.

nw ships no built-in genres. Concrete genres register themselves from their own packages, so adding one is a one-file registration — the same open-closed shape as nw.transforms and nw.renderers:

nw.register_genre(nw.Genre(slug="music-video", title="Music video", ...))
nw.get_genre("music-video").is_ready()  # every declared transform/strategy present?
nw.recommend_genre("essay")  # an intake answer -> a genre slug (or None)

Genre → project: resolve, initialize, create

Three registries connect a chosen genre to an actual project. Each is optional, and each is registered by the genre's owning app — so a host that aggregates many genres can serve any of them without knowing which app owns which.

# 1. resolve — pure: (genre, template) -> the creation envelope
nw.resolve_genre("music-video", "cinematic_clip")
# {'genre': 'music-video', 'template': 'cinematic_clip', 'params': {...}}

# 2. initialize — the side-effecting twin: seed a freshly-created project
proj = nw.Project.init("my_video")
nw.initialize_genre("music-video", proj, template="cinematic_clip")

# 3. create — for a *plugged-in* genre a host aggregates but doesn't own:
#    the owning app supplies "make a project for this in the caller's own space"
nw.create_genre_project("commentary-weave", caller_id, "ep_01")

# ...and when the host will SERVE the project, it says where it goes:
nw.create_genre_project(
    "commentary-weave", caller_id, "ep_01", projects_dir=my_projects_dir
)

An initializer must confine its side effects to the project it is given, so a failed create can be reverted by removing the project folder — create_genre_project rolls back automatically and is all-or-nothing.

A genre project factory places a project where its caller asks; it does not own the location. projects_dir is the directory the project folder is created in (the new project's root is projects_dir/<project_id>), so a host that has to serve a guest genre's project can put it where its own resolver and lister look — without which the project is a sibling of nothing the host can address. None (the default) leaves placement to the genre's app, so every pre-existing caller is unchanged. Ask nw.can_place_genre_project(slug) first: a factory written before this argument existed is refused rather than quietly satisfied somewhere else. One that accepts the argument and ignores it is caught by an outcome check on the created root — acceptance is not the guarantee, the outcome is — and an outcome nw cannot verify (a factory that accepts a placement and returns no project) is a failure rather than a pass. The rollback that follows is bounded by the placement: nothing outside it is deleted, because that branch is precisely the one where nw has concluded it does not know what the factory did.

The naming rationale (Genre / Template over kind / format / recipe / …) is in thorwhalen/nw#10 and the architecture discussion that settled it.

Transforms — the A → B arrow

Every step in an audiovisual workflow is a Transform: screenplay → treatment, beat → storyboard panel, panel → image, clips → animatic, shot → rendered clip. A Transform is a swappable, costed function from A-annotations to B-annotations, in two phases:

1. plan() — pure data. Returns a falaw.Plan plus skeleton output annotations that already carry provenance, so even a dry-run inspection shows what will be produced, from what, and at what cost. No billable calls.
2. execute() — runs the Plan, completes the skeletons with real artifact references, writes them to the project graph, and returns a TransformResult with actual cost and cache savings.

Transforms are keyed by name in an xdol.Registry, following <from_kind>_to_<to_kind>[.<flavor>[.<variant>]]:

nw.register_transform(MyTransform())
t = nw.get_transform("beat_to_panel.llm.default")
plan, skeleton = t.plan(proj, inputs, params=params)
result = t.execute(proj, plan, skeleton)
result.cost_usd_actual, result.cache_hit_savings_usd

Most Transforms subclass nw.BaseTransform and override only plan() plus the class-level name / input_kinds / output_kind / params_model / is_batch. params_model is a Pydantic model, which is what gives an MCP server or a CLI a JSON Schema for the Transform for free.

A blocked or failed output still tells you why, after reload

execute(..., on_failure="isolate") runs what can be run and reports the rest instead of raising: result.failed / result.blocked are FailedOutputs carrying a reason (and, for a blocked one, blocked_by). That reason is also persisted — proj.graph.unproduced_outputs() reads it back after a reload, and it disappears on its own the moment a retry produces the real output.

result = t.execute(proj, plan, skeleton, on_failure="isolate")
[f.reason for f in result.failed]  # in this response
[u.body.reason for u in proj.graph.unproduced_outputs()]  # survives reload

A caller's key reaches execute and nothing else

A server rendering on a caller's bring-your-own credential hands it to execute(..., secrets=) — a read-only {provider_name: key} mapping (nw.Secrets) — and to nothing else. It never enters the Plan, the skeleton, provenance, a cache key, a run record, the job index or a log line; the type redacts its repr, refuses pickling and is not JSON-serializable, so the accident raises instead of leaking. fan_out_execute and nw.jobs.enqueue pass it accepts-it-or-not, exactly like on_failure, and bind it around the call regardless — so a "fal" secret is the fal credential even for an execute override that predates the seam. BaseTransform.execute (and nw's own shot renderers) bind it the same way; an app declares the keyword on its own paid Transforms and reads the provider it calls (braidio reads "elevenlabs"). A failure message that quotes the key is redacted before it reaches a run record or the job index.

result = t.execute(proj, plan, skeleton, secrets={"fal": caller_fal_key})
nw.fan_out_execute(t, proj, fan_out, secrets={"elevenlabs": caller_key})
nw.jobs.enqueue(proj, "weave", params, dispatch=..., secrets=secrets)  # not in params

prepare → plan → execute, with a budget gate

The shot render unit makes the same split concrete, and it is why cost is knowable before the network goes near a credit card. It is shot-typed by construction (ShotSpec in, output.mp4 out), so it is not the extension point for a new render kind — register a Transform for that. A new way to render a shot still belongs here, and is adapted into a Transform for free (see "Render strategies" below). Details: misc/docs/Rendering Provenance and Partial Re-render.md.

1. prepare (nw.prepare_shot) — local work: audio slice, anchor resolution, storyboard prompt assembly. No billable calls.
2. plan (nw.plan_render_shot) — pure data: a falaw.Plan. Inspect plan.total_cost_usd before executing.
3. execute (nw.execute_render) — the only phase that talks to fal. Materializes shots/<id>/output.mp4 and records a render-decision in the graph.

prep = nw.prepare_shot(proj, "shot_01", upload=False)  # dry-run / cost preview
plan = nw.plan_render_shot(prep, quality="balanced")
print(plan.total_cost_usd, [c.tool for c in plan.calls])

prep = nw.prepare_shot(proj, "shot_01")  # upload=True for the real run
output = nw.execute_render(prep, plan, project=proj)

Plans built with upload=False are refused at execute time — they exist for inspection only.

A stored quote is not a current price

plan.total_cost_usd is true at the moment it is read and an as-of figure ever after: falaw's rate tables move — 0.0.46 re-quoted every premium LLM call tenfold upward — so a figure nw persisted before a table moves under-quotes the run it is later used to describe or gate. Under-quoting is the one direction a spend decision must never err in.

nw.pricing is the one place nw re-quotes. Give it a plan (or the calls stored in a render decision) and it answers with today's price, the stale one beside it, and which of the two you are allowed to show:

quote = nw.current_quote(plan)  # or nw.quote_render_decision(payload)
quote.total_usd  # today's price, or None
quote.status  # "unchanged" | "changed" | "unknown"
quote.as_of_total_usd  # what the plan said when it was written
quote.delta_usd  # the movement, or None if either side is unknown

None means unknown, never free. A call carrying no falaw.CostBasis — one hand-built outside a plan_*, or planned before falaw 0.0.49 — cannot be re-quoted at all, so it comes back unknown rather than repeating its frozen number. nw.jobs.estimate and nw.jobs.enqueue follow the same rule: when params["plan"] is supplied they price it, and a caller-supplied estimated_usd is ignored. Repricing is descriptive only — cost_basis never enters plan_hash, so a job's idempotency key and falaw's per-call cache key are byte-identical to what they were before, and a resumed render still dedups onto work already paid for.

Money already spent (project.total_spend_usd()) is deliberately not re-quoted: a receipt is not a quote.

A typed project on disk

nw.Project is a small facade over a project folder. The folder is the single source of truth: project.json holds project-level metadata; a per-project lacing graph (project.annot.sqlite) holds sections, shots, character / environment refs, and decisions.

proj = nw.Project.init("my_video", song="track.mp3")
proj.add_character("alex", description="warm, deadpan")
proj.set_character_anchor("alex", "characters/alex/refs/headshot.png")
proj.upsert_shot(
    nw.ShotSpec(
        id="shot_01",
        start_s=0.0,
        end_s=8.0,
        characters=("alex",),
        render_strategy="lipsync",
    )
)

proj.read_summary()  # typed ProjectSummary: title, counts, lifecycle stages
proj.read_spec()  # typed ProjectSpec
proj.log_decision("retry_shot", shot_id="shot_03", reason="lipsync drift")

# "Where did we leave off?" — decision tail, what the last *authored*
# change reaches downstream, recorded spend, unrendered shots, deterministic
# next actions. Offline.
brief = proj.resumption_brief()
brief.suggested_next
brief.caveats  # what the numbers above do NOT know — rendered next to them
proj.total_spend_usd()

my_video/
  project.json                  # project-level metadata (title, song, style)
  project.annot.sqlite          # lacing graph: sections, shots, refs, decisions
  storyboard.annot.sqlite       # storyboard panels (created on save_storyboard)
  song/                         # master audio
  lyrics/                       # lyrics + alignment (alignment.annot)
  characters/<name>/
    card.json                   # card with reference_image_path (the "anchor")
    refs/                       # candidate images
    selected/                   # curator-picked images
  environments/<name>/
    establishing.png            # the environment anchor
  shots/<shot_id>/
    audio.wav                   # the song over [start_s, end_s]
    shot.json                   # mirror of the shot spec
    output.mp4                  # the rendered shot
  output/
    final.mp4                   # composed timeline
  .nw/
    decisions.jsonl             # tail-grep-able decision audit
    migrated_to_graph           # migration sentinel

Pre-graph projects (and muvid fixtures) auto-migrate on first open; the migration is idempotent and writes a sentinel under .nw/.

Provenance and freshness

All sections, shots, refs, and decisions live in a lacing annotation graph with was_derived_from edges, so "what's downstream of this change?" is a query, not a heuristic:

downstream = nw.descendants_of(proj.root, character_annotation_id)
stale = nw.stale_after(proj.root, character_annotation_id)
upstream = nw.derived_from(proj.root, render_annotation_id)
shots = nw.annotations_at_tier(proj.root, "shot")

descendants_of and stale_after are different questions. The first is reachability — what is downstream of this? The second is freshness — what did this change actually invalidate? — and it cuts off early: every derived annotation records the content digests of its inputs at write time (a verifying trace), so a change that leaves a value untouched stops propagating there instead of invalidating the whole subtree. Edit a character and revert it, or regenerate a panel to identical content, and the stale set goes back to empty without anything downstream being recomputed.

for v in nw.stale_verdicts(proj.root, character_annotation_id):
    print(v.annotation.id, v.is_stale, v.reason)  # e.g. "upstream-changed"

There is also the snapshot form — what is stale in this project right now?, no changed_id needed — which is what a freshness indicator wants:

stale_now = nw.all_stale(proj.root)  # every currently-stale annotation
verdicts = nw.stale_verdicts_all(proj.root)  # ... with reasons

The rule is asymmetric on purpose: anything unverifiable — no trace, a deleted input, a trace that no longer covers the current parents, an annotation whose own generated_at_time is lacing's tick-0 UNKNOWN sentinel (generated-at-unknown, lacing#44) — counts as stale. Over-reporting costs a recompute; under-reporting serves a stale artifact. The unknown-stamp case is on the row's own stamp only: regenerating the row clears it. A tick-0 parent is not a verdict — freshness is digest-verified, and a legacy root's stamp is cleared only by the timestamp backfill, which nothing downstream waits on. For the scoped walk that also means no migration: annotations written before traces existed behave exactly as they did under pure reachability. The snapshot form is stricter: on a pre-trace project it reports every derived annotation stale until each is rewritten through the trace-writing path.

Scope: the annotation tier. Artifact-to-artifact lineage is still unrepresentable upstream (lacing#14). nw.stale_after compares upstream values, not the producing Transform, so bumping a Transform's implementation does not move a digest.

Async jobs

nw.jobs is a project-scoped facade over au for render work too long to sit inside an HTTP request. A job is one long, cancellable unit of work with a durable id, a persistent terminal state, live progress and a learned ETA, a cost, and a cancel entry keyed by that id:

gate = nw.jobs.estimate(proj, "panel.animate", params)  # cost gate, no enqueue
if not gate["requires_approval"]:
    job = nw.jobs.enqueue(proj, "panel.animate", params)
    nw.jobs.get_job(proj, job.job_id)
    nw.jobs.cancel_job(proj, job.job_id)
nw.jobs.list_jobs(proj)
nw.jobs.to_dict(job)  # the JSON a task tray renders

Unknown cost always requires approval. Resubmitting while a job with the same idempotency key is live returns the existing job rather than launching a duplicate. Every tunable is keyword-configurable via nw.jobs.JobsConfig. A caller's credential goes in secrets= (held in memory, offered to the dispatch callable only when it declares the keyword), never in params, which is what the job index persists.

Render strategies

Each shot carries an open-string render_strategy. nw.renderers ships five built-in strategies and lets apps register their own:

nw.list_strategies()  # ['composite_lipsync', 'image_to_video', ...]
nw.register_strategy("my_strategy", MyStrategy())

Each built-in strategy is also adapted into the Transform world as shot_to_render_result.fal.<strategy>, so the two registries stay one pipeline rather than two.

Storyboard layer

nw.storyboard bridges artful storyboards into an nw.Project — one panel per shot, seed-image generation planned as a falaw.Plan, then executed:

sb, intervals = nw.storyboard_from_shots(proj)
plan, panel_ids = nw.plan_render_panel_images(sb, quality="balanced")
sb = nw.execute_render_panel_images(proj, sb, plan, panel_ids)
nw.save_storyboard(proj, sb, panel_intervals=intervals)

QA reports

nw.inspect answers what a successful render can't: "did it come out the right length?", "is there a frozen-frame segment?", "are there gaps between shots?"

report = nw.shot_report(proj, "shot_01")
report.duration_within_tolerance  # False if the model returned a short clip
report.has_long_freeze  # True if a ≥1s frozen segment is detected

compose = nw.compose_report(proj)
compose.freeze_alerts  # tuple of suspicious shots
compose.gaps  # gaps between consecutive shots

Validation — a pluggable menu, placed where you want it

nw.inspect above answers one question about one shot. nw.validation is the general form: a registry of checks that anyone can add to, with dependencies resolved, independent checks run concurrently, and one report at the end.

report = nw.validate(film, checks=["media.encode_complete", "media.no_long_freeze"])
report.ok  # False if anything failed *or if any check could not run*
print(report.summary())
report.raise_if_failed()  # a hard gate before a publish

Three checks ship with nw (nw.menu()), each of which has caught a real defect in a finished film: a missing video or audio stream, an encode that stopped early (which duration alone cannot show — the container takes its duration from the audio stream), and a long frozen segment.

A check declares what it requires, what it costs, whether it is parallel_safe, which binaries it needs — and example_requests, the phrases a person actually says when they want it, which is what lets nw.suggest("the picture freezes") turn a request into a selection instead of exposing a forty-item enum to a model.

@nw.register_check(
    name="type.captions_complete",
    summary="no caption is cut",
    requires=("media.streams_present",),
    cost="dear",
    example_requests=("is the text cut off", "the caption is truncated"),
)
def _captions_complete(film, ctx): ...

Nothing calls validate for you, and that is the decision rather than an omission: where validation belongs — before a human sees a result, before a publish, or both — is a judgement about cost and consequence that only the caller can make. validate(x) with no selection runs nothing.

Two honesty rules worth knowing before you gate on it: a check that was skipped (missing binary) or that raised makes report.ok false — could-not-run is never a pass — and an unknown check name raises rather than being dropped.

Checks live in whichever package owns the knowledge they apply (type checks next to tituli, motion next to burns), and register themselves at import. Ideas for new ones accumulate as validation-idea issues on this repo. The full record is misc/docs/Validation — the seam, the menu, and where ideas go.md.

Sibling experiments

Comparing four interpretations of the same song is a first-class operation, not a shell loop:

nw.clone_project(
    "the_bells",
    "the_bells_v1_lipsync",
    preserve=("song", "lyrics", "characters"),
    reset=("script", "shots", "output", ".nw"),
)

summaries = nw.summarize_all(["the_bells_v1", "the_bells_v2", "the_bells_v3"])
nw.apply_to_projects(roots, lambda p: nw.compose_report(p), parallel=True)

API at a glance

# Genres — the reusable production specialization
(
    nw.Genre,
    nw.Template,
    nw.genres,
    nw.register_genre,
    nw.get_genre,
    nw.list_genres,
)
(
    nw.genre_catalog,
    nw.describe_genre,
    nw.recommend_genre,
    nw.resolve_defaults,
    nw.GENRE_STATUSES,
)
(
    nw.GenreResolver,
    nw.genre_resolvers,
    nw.register_genre_resolver,
    nw.resolve_genre,
)
(
    nw.GenreInitializer,
    nw.genre_initializers,
    nw.register_genre_initializer,
    nw.initialize_genre,
)
(
    nw.GenreProjectFactory,
    nw.genre_project_factories,
    nw.register_genre_project_factory,
    nw.has_genre_project_factory,
    nw.can_place_genre_project,
    nw.create_genre_project,
    nw.PLACEMENT_ARG,
)

# Transforms — the A -> B arrow
(
    nw.Transform,
    nw.BaseTransform,
    nw.TransformInputs,
    nw.TransformResult,
)
nw.transforms, nw.register_transform, nw.get_transform, nw.list_transforms

# Folder facade
nw.Project, nw.Project.init, nw.CharacterImage

# Schema
(
    nw.ProjectSpec,
    nw.ProjectSummary,
    nw.ResumptionBrief,
    nw.DecisionEntry,
    nw.SectionSpec,
    nw.ShotSpec,
)
nw.CharacterRef, nw.EnvironmentRef, nw.SongInfo, nw.SCHEMA_VERSION

# Render workflow
nw.prepare_shot, nw.plan_render_shot, nw.execute_render, nw.ShotPreparation

# Async jobs
nw.jobs.estimate, nw.jobs.enqueue, nw.jobs.list_jobs
nw.jobs.get_job, nw.jobs.cancel_job, nw.jobs.to_dict, nw.jobs.JobsConfig

# Strategies
(
    nw.Strategy,
    nw.get_strategy,
    nw.list_strategies,
)
nw.register_strategy, nw.strategies

# Storyboard
(
    nw.open_storyboard,
    nw.save_storyboard,
    nw.storyboard_from_shots,
)
(
    nw.plan_render_panel_images,
    nw.execute_render_panel_images,
)
nw.storyboard_db_path, nw.project_asset_id

# Inspect / QA
(
    nw.shot_report,
    nw.compose_report,
    nw.ShotReport,
    nw.ComposeReport,
)
nw.FrozenSegment, nw.Gap

# Validation — the pluggable menu
(
    nw.validate,
    nw.menu,
    nw.suggest,
    nw.plan_checks,
    nw.register_check,
    nw.checks,
)
nw.Check, nw.Finding, nw.CheckResult, nw.ValidationReport, nw.ValidationError

# Graph / provenance
(
    nw.ProjectGraph,
    nw.derived_from,
    nw.descendants_of,
    nw.stale_after,
    nw.stale_verdicts,
    nw.stale_verdicts_all,
    nw.all_stale,
    nw.FreshnessVerdict,
)
nw.annotations_at_tier, nw.iter_all_annotations, nw.open_project_stores
nw.ProjectGraph.unproduced_outputs  # why a blocked/failed output was never produced

# Experiments
nw.clone_project, nw.apply_to_projects, nw.summarize_all

# Migration
nw.migrate_to_graph, nw.is_migrated

Design notes

• SSOT on the folder. Every typed value comes from project.json plus the project graph. Tools never have to invent their own storage.
• Genres are declarations, not engines. Project, the prepare → plan → execute split, freshness, nw.jobs, and the cost gate are all genre-agnostic and serve every genre unchanged.
• Open registries throughout. Genres, Transforms, and render strategies are all xdol.Registry entries keyed by string, so apps extend nw without modifying it. on_conflict="error" keeps one plugin from silently shadowing another's.
• Plan-then-execute. Cost is computed and inspectable before anything bills.
• Provenance by default. Every render and every curator decision is written to the graph with was_derived_from, so freshness analysis is a graph walk, not a heuristic.

Longer rationale lives in misc/docs/ — Rendering Provenance and Partial Re-render and Execution Semantics and Fan-out.

Known limits

Places where the substrate currently promises less than it looks like it does:

• Early cutoff stops at the annotation tier. stale_after compares annotation value digests; artifact→artifact lineage is unrepresentable upstream, because lacing.Provenance.was_derived_from is list[UUID] and an asset_id is 64 hex chars (lacing#14). It also does not notice a changed Transform — a re-implemented or re-prompted Transform moves no upstream digest — and reads a hand-edited output as fresh, because relative to its inputs it is.
• resumption_brief still reports reachability, deliberately. downstream_of_last_authored_change is descendants_of, an explicit upper bound named for what it measures; nw.stale_after is the narrower answer if you want it (#7).
• A removed annotation leaves its verifying trace behind. Orphan traces are inert — they are indexed by a target id that no longer resolves — but nothing collects them yet (#36).
• BaseTransform.execute has no failure isolation — one failing call in a fan-out Plan raises, and no annotations reach the graph for the calls that did succeed (#25).

Dependencies

pydantic, falaw (fal-AI planner), lacing (annotation graph), xdol (registry), artful (storyboard), au (async job substrate), dol (storage).

Optional system tools: ffmpeg / ffprobe for audio slicing and QA reports.

License

MIT — see LICENSE.