curu-caption

What is this

curu-caption makes LoRA training sets from videos and photographs, on your own computer. It removes duplicate and blank frames, finds the items that you ask for, and captions only those items. The result is media and .txt caption pairs, with the configuration that AI-Toolkit trains from. It is for a person who trains LoRAs and does not want to extract, sort and caption frames by hand. Each step runs locally by default, and no step needs a cloud account.

What you can do

Install

You need Python 3.12 or later, uv, a Rust toolchain (rustup), ffmpeg and Ollama.

To install the released package from PyPI, you need only Python 3.12 or later, uv, ffmpeg and Ollama:

uv tool install "curu-caption[audio]"    # curu-caption and curu-caption-tui on the PATH

The wheels are for Linux x86_64 and macOS (arm64 and x86_64). They include the Rust extension and the review program. On a CUDA computer, use "curu-caption[audio,gpu]".

To develop, install from a clone of the repository. Then you also need the Rust toolchain, and on Linux a C compiler:

git clone https://github.com/darth-veitcher/curu-caption.git
cd curu-caption
uv sync --extra audio
cargo build --release -p curu-tui && scripts/stage-tui.sh   # the review program
source .venv/bin/activate                                  # puts curu-caption on the PATH

• uv sync also compiles the Rust extension curu_caption._core.
• Give --extra audio each time that you run uv sync. A plain uv sync removes it. It makes the sound, speech and music facts. ltx needs the sound fact, and h3 uses all three when it has them.
• The second line builds the review program. No other command needs it.
• No model downloads at install time. Each model downloads the first time that a command needs it (see Technical notes).

Then get a caption model for Ollama. The amoral-qwen3.5-9B project makes the three caption models. The default model needs about 32 GB of memory (unified memory or VRAM):

ollama pull saracen9/amoral-qwen3.5-9b

The full setup, with each platform, cache and fix, is in docs/setup.md.

Quickstart

Make a training set for a text-to-video LoRA from one video, episode.mp4:

curu-caption ingest --source episode.mp4 --out corpus/
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v \
    --want "a person smiling" --limit 60 --dry-run
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v \
    --want "a person smiling" --limit 60 --caption --trigger ohwx

1. ingest extracts one frame each second, removes the duplicate and blank frames, and indexes the frames that it keeps. It does not caption. The result is the corpus, corpus/.
2. cut --dry-run shows the 60 frames that match "a person smiling", and the facts that each frame does not have yet. It writes nothing and it costs nothing.
3. cut --caption captions those 60 frames only, and writes them to training/.

training/ then holds frame_<ms>.png and frame_<ms>.txt pairs and a dataset.yaml. Each caption starts with the trigger word ohwx. Copy training/ to the computer that trains, and give dataset.yaml to AI-Toolkit.

Make the corpus one time, and cut it many times. A second LoRA from the same video is a second cut, not a second ingest. Each command refuses an --out folder that is not empty, so no run writes over an earlier result.

Make a corpus from a video

ingest reads a video, or each video in a folder, and makes a corpus:

curu-caption ingest --source videos/ --out corpus/

It reads the videos in each subfolder too. At the end, it prints corpus ready with the number of items, and the next command to run.

• Frames. By default, ingest extracts one frame each second (--interval 1.0). Each frame is a PNG named for its time in milliseconds, for example frame_0000004000.png.
• The cull. ingest removes near-duplicate frames and blank frames. A blank frame has a sharpness (Laplacian variance) below --min-sharpness, 1.0 by default. Of each group of near-duplicates, it keeps the sharpest frame.
• The index. ingest then indexes each frame that it keeps, so that cut --want can find it. It also records a quality score for each item (see Index a corpus again).
• Many sources. In a corpus of many videos, each video has its own subfolder, with the path of the video under --source. In a cut, each file name starts with its source folder, so the clip_000.mp4 of each episode stays.
• Some videos of a folder. Give --source more than once, for example --source s01/ep01.mkv --source s01/ep02.mkv. A video that you name directly gets a subfolder with its file name, without the extension, for example ep01. If two videos have the same subfolder name, ingest stops before it reads a video. Then give their parent folder as --source.

The corpus is the result that you keep. Each item holds its facts in the media file itself, in PNG text chunks or MP4 metadata. Thus you can move or copy a corpus as one folder, and a single item that you copy out keeps its facts.

Clips for a video LoRA. Add --clips to cut the video into MP4 clips, one for each shot, with their audio. Use --interval 1 or less with it: ingest finds each shot from the gaps between the frames that it samples. With a wide interval, no clip is long enough, and ingest prints this reason. The frames that ingest used to find the shots stay in the corpus, marked detection-only, so a later detection costs nothing. A cut never selects them, and says N detection-only item(s) — kept as evidence, never selected.

Before it finds the shots of a video, ingest prints detecting shots in N frame(s). N is the number of frames that it extracted from that video. The next line, with the clip count, comes after it finds the shots and cuts the clips.

The audio of a clip is FLAC. It keeps each channel and the channel layout of the source, for example 5.1.

--phash-distance is the number of bits, from 0 to 64, that differ between the perceptual hashes of two frames. At 0, only frames that look the same are near-duplicates. On real TV footage, the default 8 removed 37% of the frames, and each group was one static shot.

Make a corpus from photographs

ingest also reads a folder of photographs, with its subfolders:

curu-caption ingest --source photos/ --out photo-corpus/

• Each photograph becomes one item. ingest converts it to PNG, because a PNG file can hold facts and a JPEG file cannot.
• The name of each item comes from its path under --source. Thus two photographs with the same file name do not write over each other.
• ingest ignores the files that are not media, for example Thumbs.db, .DS_Store and .zip files.
• ingest refuses --clips for photographs, because a photograph has no shots and no audio. It refuses before the ingest starts, not later at cut time.

ingest leaves out a folder name when the file name already starts with it, as for man_0001.jpg.

Index a corpus again

index runs the index step of ingest again on a corpus that exists. Use it after a change to an index model, or on a corpus that you made with --no-index or --no-quality:

curu-caption ingest --source photos/ --out photo-corpus/ --no-quality
curu-caption index --corpus photo-corpus/                 # adds the quality scores
curu-caption index --corpus photo-corpus/ --tags          # also adds WD14 tags and rating

index --no-quality makes only the index. On a corpus of 11 items, of which ingest kept 7, a second run of index prints:

index 7 kept item(s) in photo-corpus (4 culled, excluded or detection-only, not indexed)
  indexing with EmbeddingGemma 2 (curu-vision)
  judging quality with LiquidAI/d1-3B — a review pre-sort, never a gate
  embedding: 0 extracted, 7 already current
  quality: 0 extracted, 7 already current

index ready — photo-corpus

index reads only the kept items. It does not make a fact again when the fact is current for its model and parameters. Thus a second run does nothing, as in the example output.

Run index again on each corpus after a change to the index model. The embedding fact records the model and the version of curu-vision. After you update curu-vision, each vector is stale. cut --want does not use a stale vector, and index makes it again. curu-vision puts each transparent part of a picture onto white before the embedding.

The quality fact holds two probabilities from 0 to 1. The model d1-3B gives them from three frames of each clip, at 10%, 50% and 90%, or from the one picture of a still:

• p_keep is the probability that the item is good training data: sharp, one shot, and nothing broken.
• p_person is the probability that a person is visible.

A higher value is more likely to be true. The scores only sort items for review. No command removes, culls or selects an item on these numbers. On 101 labelled clips, d1-3B put a good clip above a rejected clip 72% of the time (AUROC 0.72; 0.5 is chance). That is useful to order a review, and too weak to make the decision alone. A cut records the scores of each item that it selects in cut-selection.json.

--tags adds two facts to each photograph, from one run of the WD14 tagger. A clip gets no tags yet (see Limits):

• The tags fact holds the descriptive tags, the tag with the highest score first, for example solo, 1girl, outdoors. A tag for a thing in the picture needs a score above 0.35, and the name of a character needs a score above 0.70. It never holds a rating label.
• The rating fact holds the four WD14 rating scores, each from 0 to 1, for example {"explicit": 0.002, "general": 0.736, "questionable": 0.011, "sensitive": 0.286}. general is safe content, and explicit is sexual content. A higher score means that the model finds that rating more probable. Each score is independent of the other three, so the four do not add up to 1.

No command reads the rating yet. It is a fact in each media file, for your own sort or filter. No command removes, culls or selects an item on it. WD14 learned from anime pictures, so its rating of live-action footage is not accurate.

On a Mac, use --no-quality and run index later on a computer with a GPU.

Find items by meaning

cut --want searches the index for the items that match a description in plain English. With --dry-run, it shows the result and writes nothing:

curu-caption ingest --source videos/ --out corpus/
curu-caption cut --corpus corpus/ --out training/ --want "a wide shot" --limit 60 --dry-run

The command prints the number of matches, and the highest and lowest score that it kept. The score of each item is the cosine similarity between the description and the item, from -1 to 1. A higher score is a closer match, but use the order, not the number. Over 377 clips, the scores of each query were between 0.64 and 0.79. The score where good matches stop changes with the query, so no fixed limit is correct for all queries.

• --limit keeps the best matches, 100 by default. --limit 0 keeps each match. This is the control to use.
• --threshold removes each match with a score below the value. It is off by default.
• Without --want, the cut takes all the items of the corpus. With a --limit above 0, it takes the first --limit items, in name order.
• cut-selection.json records the description and the lowest score that the cut kept.

Make a training set

cut selects items from a corpus and writes a training set (a cut) for one model. With --caption, it captions the selected items only, not the full corpus:

curu-caption ingest --source episode.mp4 --out corpus/
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v --caption --trigger ohwx

training/
├── frame_0000004000.png        the media, rendered for --projection
├── frame_0000004000.txt        "ohwx, a woman turns from the window as the door opens"
├── frame_0000004000.avdm.json  the scene document that the caption came from
├── dataset.yaml                the trainer config (job.yaml for h3 and h3-prose)
├── cut.json                    the projection, trainer, trigger and the source of each fact
└── cut-selection.json          the selected items, the lowest score kept and the quality scores

A projection is the form of caption that one model trains on. Select it for the model that you train. The default is ltx.

• Captions. The caption model fills in a fixed form of claims about the middle frame of the item. Then d1-3B checks each claim with a yes or no question. The caption keeps only the checked claims, one plain sentence for each.
• Missing facts. --dry-run names each fact that the projection needs and the corpus does not have. ltx needs a sound fact, and h3 uses one when it has it. --caption makes it when the audio extra is installed.
• The trainer. --trainer ai-toolkit (the default) writes dataset.yaml, or job.yaml for h3 and h3-prose. --trainer musubi writes a .wav file beside each clip, and no AI-Toolkit file. An ltx, wan-t2v or wan-i2v cut copies each clip with its FLAC audio. For LTX-2, AI-Toolkit mixes 6 or 8 channels down to stereo. musubi-tuner mixes the audio down to mono or stereo. WAN 2.2 reads no audio.
• Portable. folder_path in the config is ., the cut folder itself. Thus the cut works on the computer that trains, and the corpus stays on yours.

The result table of cut counts the items:

Each fact records its producer and parameters. A second cut does not make a fact again when its producer and parameters did not change. A second cut of the same two items finished in less than four seconds, after the first cut used minutes for the captions. The full reference is in docs/cut.md.

Make a training set for MiniMax H3

MiniMax H3 trains on clips. Make a corpus of clips, then cut it with --projection h3:

curu-caption ingest --source episode.mp4 --out clips-corpus/ --clips --interval 1
curu-caption cut --corpus clips-corpus/ --out h3-training/ --projection h3 --caption --trigger ohwx

The cut encodes each clip again to 24 fps, to a frame count of 17k + 5 (5, 22, ... 345), and to sides that are multiples of 32. It then measures each clip. A clip that does not agree with all three rules goes in the report, not in the cut. The audio becomes stereo, because H3 trains on stereo audio. The cut also writes a complete AI-Toolkit job.yaml. To train, run python <ai-toolkit>/run.py job.yaml in h3-training/.

These options apply to h3 and h3-prose only. The other projections copy each clip as it is.

Make a bundle for amoral

--trainer amoral-av writes a training bundle for the omni model of amoral, which hears and sees together. The bundle has clips and derived clips, and no captions:

curu-caption cut --corpus clips-corpus/ --out bundle/ --trainer amoral-av \
    --golden golden-v1/ --estate-root media/ --want "two people talking" --limit 200 \
    --derive 1 --seed 0

• The corpus must come from ingest --clips, which records the source file of each clip.
• The cut writes nothing if a source file is also a source file of golden-v1, the evaluation set. It lists each such file. To keep these files out of the corpus, give ingest the option --golden golden-v1/ too.
• --derive N makes N changed copies of each clip, for example with the audio of another clip.
• Open review.html in the bundle to look at and listen to each derived clip.

The bundle format, the five changes and the clips that the cut leaves out are in docs/amoral-av.md.

Review a corpus or a cut

review opens a corpus or a cut in the terminal. Look at each item, correct its caption, and mark the items that you do not want:

curu-caption review --corpus corpus/
curu-caption review --corpus training/ --source corpus/
curu-caption cut --prune training/

┌ Corpus (2/6) ⚠ 1 empty ────┐┌ Preview ────────────────────────────────────┐
│[!]   frame_0000001000      ││                                             │
│      frame_0000002000      ││        [ frame_0000002000.png ]             │
│[X]   frame_0000003000      ││                                             │
│[clip] clip_001             ││                                             │
└────────────────────────────┘└─────────────────────────────────────────────┘
                              ┌ Caption ────────────────────────────────────┐
                              │ a woman turns from the window as the door   │
                              │ opens                                       │
                              └─────────────────────────────────────────────┘
 j/k ↑↓ nav  e edit  V visual  R find-replace  P prefix  S set  x exclude  ? help  q quit

The title says Corpus or Cut. What you opened sets what x (exclude) means:

• Reject in the corpus from a cut. Open the cut with --source, which names its corpus. Then X rejects the item in the corpus too.
• Remove the rejected items. The trainer config names the full cut folder, so a marked file that is still in the folder still trains. cut --prune deletes the marked files and writes the config again.
• Safe to prune. Each file in a cut is a copy, and cut-selection.json records the selection. cut --prune refuses a folder with no cut.json, because a corpus holds the only copy of its media.
• ⚠ N empty in the title and [!] on a row show captions that are empty or duplicated.

Open a folder of corpora to work through each one in a queue. The keys, the batch editing and the queue are in docs/tui.md.

Caption a folder

caption captions each image and video in a folder, or one file, without a corpus. It writes a .txt file with the same name beside each file:

curu-caption caption photos/
curu-caption caption photos/ --dry-run --hint "a set of family birthday photos"

• caption reads the images (.png, .jpg, .jpeg) and videos (.mp4, .mkv, .mov, .avi, .webm) in the folder. It captions a video from one frame.
• It does not caption a file that has a caption of 10 characters or more. Use --force to caption it again.
• --hint gives context for each caption in the run. --dry-run prints the captions and writes no file.
• It shows the progress of each file. If one file fails, it continues. At the end, it exits with a code that is not 0.

Add WD14 tags, local and with no network, with --tags. With --no-prose, a caption has only the tags, and no caption model is necessary:

curu-caption caption photos/ --no-prose --tags

WD14 tags are short Danbooru-style tags, for example 1girl, outdoors, smile. With prose and tags, the .txt file has the prose, an empty line, and then the tags.

Find near-duplicates in a folder

curate reports the groups of near-duplicate images and clips in a folder. It does not change or delete a file:

curu-caption curate photos/

Found 1 near-duplicate group(s):
  Group 1: IMG_0001.jpg, IMG_0004.jpg

• Two files are near-duplicates when their perceptual hashes differ by 5 bits or fewer. Set a different value with --threshold. A higher value puts more files in each group.
• curate compares PNG and JPEG stills and clips in one pass. It compares a clip on its middle frame. A clip costs 150 to 200 times more than a still, so a few thousand clips take minutes.
• curate skips caption files and manifests, and does not report them.
• If all the files go in one group, the threshold is too high. Then curate says so, and reports no groups.

Judge a folder

judge asks the caption model to keep or reject each image against your criteria. It does not change a file:

curu-caption judge photos/ --criteria "no blurry shots, subject clearly visible"

Judged 4 image(s): 3 keep, 1 reject.

  KEEP   IMG_0001.jpg: <the reason of the model>
  REJECT IMG_0004.jpg: <the reason of the model>

Each line gives the verdict and the reason of the model. If the model rejects all the images, judge changes each verdict to keep and says so. Then the criteria are too strict or not clear.

Write dataset.yaml for a folder

dataset-config writes the dataset.yaml that AI-Toolkit trains from, for a folder of media and .txt captions:

curu-caption dataset-config photos/ --trigger-word ohwx

It writes photos/dataset.yaml:

datasets:
- folder_path: .
  caption_ext: txt
  num_frames: 1
  resolution:
  - 256
  - 256
  num_repeats: 1
  trigger_word: ohwx
  sample_prompts:
  - a test pattern, IMG_0001

folder_path is ., so run AI-Toolkit from the folder. --num-repeats sets the weight of the dataset, 1 by default. resolution is the size of the first image, or 512 × 512 when there is no image. sample_prompts holds up to five different captions from the folder.

Read a ComfyUI workflow

comfy inspect prints each node of the ComfyUI workflow in a generated PNG image:

curu-caption comfy inspect generated.png

Node 3 (KSampler)
  seed: 851616030078638
  steps: 20
  ...
Node 6 (CLIPTextEncode)
  text: masterpiece best quality girl
  clip: -> node 10 (output 1)

To use the text of a node as context for a caption, give its id to caption --comfy-node.

Caption on another computer

Captioning is the one step that needs a GPU. Keep the corpus on your computer, and send the captions to Ollama on a computer that has a GPU:

curu-caption caption photos/ --ollama-url http://gpu-box:11434
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v --caption \
    --ollama-url http://gpu-box:11434 --model saracen9/amoral-qwen3.8-27b
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v --caption \
    --pod curu-caption-vlm

• cut --caption checks the endpoint before it selects or writes anything. If the computer does not have the model, the error names the models that it has.
• --pod names a RunPod pod that curu-infra made. The command gets its address and its token from curu-infra.
• For an Ollama behind a token, set CURU_CAPTION_OLLAMA_TOKEN. Do not put a token on the command line.

SSH tunnels, pods and the error messages are in docs/remote-captioning.md.

Work on a network share

On a network share (NAS), add --stage-locally to ingest and cut. The command works on the local disk and copies the result to the share in one pass at the end:

curu-caption ingest --source videos/ --out corpus/ --stage-locally

• A network share is slow for thousands of small reads and writes. Staging changes them into one copy.
• Both commands show sync-back: <done>/<total> while they copy. If the counter stops, the mount has stopped.
• caption stages its writes by default. --no-stage-writes stops this for a local folder.

The rules, the free disk space and the fix for a stopped macOS mount are in docs/nas.md.

Set the caption model and other settings

Most settings are options. Set the caption model one time for each command:

export CURU_CAPTION_CAPTION_MODEL=saracen9/amoral-qwen3.8-27b

The command-line option comes first, then the environment variable, then the configuration file ~/.config/curu-caption/config.toml, then the default:

# ~/.config/curu-caption/config.toml
[caption]
model = "saracen9/amoral-qwen3.8-27b"

[ort]
providers = "cpu"

Each setting, environment variable and file on disk is in docs/configuration.md.

Handle errors

A command that cannot do its work stops before it writes, prints the reason, and exits with a code that is not 0:

curu-caption ingest --source photos/ --out photo-corpus/ --no-index
curu-caption ingest --source photos/ --out photo-corpus/ --no-index || echo "exit code $?"
curu-caption cut --prune photo-corpus/ || echo "exit code $?"

refusing: photo-corpus is not empty
exit code 1
Error: photo-corpus has no cut.json -- it is not a cut. Pruning a corpus would delete the only copy of its media, so this refuses rather than guesses.
exit code 1

• Before the cost. cut --caption checks the caption endpoint before it selects, copies or captions. With --stage-locally, a command checks the free disk space before it copies.
• One bad file. caption, curate and judge report a file that fails and continue. At the end, they exit with a code that is not 0.
• Missing facts. cut reports an item that it cannot caption. It does not write a caption that it cannot check.
• A failed encode. cut --trainer amoral-av writes each clip that it can encode. It shows the first 5 failures with the reason from ffmpeg, and the number of failures. Then it exits with 1.
• Disk space. ingest stops to wait when the disk of the corpus has less than 2 GB free. It writes a message on stderr when the wait starts, and again each 30 seconds. The message gives the free space and the minimum. After 10 minutes, it stops with an error.

Limits

• --tags does not work on clips. On a corpus made with --clips, the tags and rating facts fail for each clip (#170).
• The sound fact is not heard. The caption model writes it from the sound captioner's caption, the words, and one frame. On 40 test clips with the default 9B caption model, 9 of 30 claims beyond "a voice speaks" were correct and 15 were false. Most false claims are sources that the captioner names wrongly, for example a siren, a horse or a cow. For h3, this text becomes the soundscape (overall_soundscape). Read it before you train, or train video-only with h3-prose (#93).
• Dialogue does not say who speaks. Each line starts with Someone says:, so a clip with two speakers does not tell them apart. A fix needs speaker diarization in the captions.
• The picture is checked, not proven. On 40 test frames, 88 of the 898 claims that the check kept were false (precision 0.90, against 0.81 for the old prose captions). Most errors are upper garments, the direction that a person faces, and headwear. The check of helmets is poor, and an invented person keeps its claims.
• --provider openrouter ignores the corpus prompts. It does not get the claims form or the motion and sound prompts. Use Ollama for the captions of a cut. cut --caption has no cloud provider.
• review and clips. x on a clip in review does not stop cut from using the clip. X does not find a clip in a corpus of many sources (#87).
• review does not sort by quality, because it cannot read the facts of a clip yet (#87).
• review has no mouse (#101).
• d1-3B is slow without CUDA. On a Mac, the quality scores and the claim check run on the CPU, and take minutes for each item.
• It does not train. curu-caption makes datasets. AI-Toolkit trains.

Technical notes

• Version. 3.0.2, set in pyproject.toml (project.version) and in Cargo.toml (workspace.package.version). The changes are in CHANGELOG.md.
• Build. uv sync builds crates/curu-core into curu_caption._core with maturin. crates/curu-tui is the review program, and crates/curu-ondisk writes PNG text chunks. After a change to the Rust code of the extension, run uv sync --extra audio again. [tool.uv] cache-keys in pyproject.toml lists the files that cause a new build. On Linux, the extension contains its own copy of liblzma, a compression library, so it needs no system copy.
• Platforms. macOS and Linux. On Apple silicon, the models run on the CPU: CoreML was 1.6 to 5.6 times slower for WD14.
• ffmpeg. On x86_64, the Rust core downloads ffmpeg when it is not on the PATH. On other platforms, install it.
• Models. Each model downloads on first use, at a pinned revision. EmbeddingGemma 2 (1.8 GB) is the index, and d1-3B (6.2 GB) gives the quality scores and the claim check. WD14 (455 MB) gives the tags and the rating. EmbeddingGemma 2 and WD14 come from curu-vision, which checks their SHA-256.
• Audio models. The sound captioner (about 1 GB) and AST (346 MB) download into the cache of this project. AST is an audio tagger that gives a probability for each AudioSet sound label. Speech comes from curu-vision: MOSS (1.8 GB) on a computer with CUDA, which labels speakers, and Parakeet (2.4 GB) on other computers.
• Tutorials. The records of real training runs use the commands of version 2.0, which do not exist now. They are in project-management/archive/tutorials-v2/.
• Where it fits. curu-catalog finds media, curu-caption makes training data, curu generates, and curu-vision supplies shared models. See docs/ecosystem.md.
• Decisions. The decisions are in project-management/ADRs/. The main ones:
  • Facts in the media file: ADR-083
  • The corpus and the cut: ADR-084
  • H3 through AI-Toolkit: ADR-085
  • The cull thresholds: ADR-080
  • Search ranks and has no threshold: ADR-081
  • AVDM, the scene document: ADR-088
  • EmbeddingGemma 2, the index: ADR-089
  • The quality scores sort and do not decide: ADR-090
  • Audio facts of the trained window: ADR-091
  • Captions from checked claims: ADR-092
  • The bundle for amoral: ADR-093
  • Speech from curu-vision: ADR-094
  • The user files under XDG: ADR-064
  • The local quality gate, with no CI: ADR-059

Develop

Run the quality gate before you push. There is no CI, so this local gate is the only gate:

uv tool install beacon-framework   # BEACON 1.18 or later: the docs check and beacon doctor
uv sync --extra audio
scripts/check.sh            # ruff, ty, cargo, the tests, the docs check and beacon doctor
scripts/check.sh --full     # also a build from a new clone, before a release or a merge

• uv run pytest runs the default tests, with no model and no endpoint. uv run pytest -m model and uv run pytest -m vlm run the tests that need weights or Ollama.
• Write README.md, docs/ and AGENTS.md to the standard in docs/WRITING.md. beacon docs check checks them.
• The workflow, the tests and the release are in CONTRIBUTING.md. The rules for AI assistants are in AGENTS.md.
• The project uses the BEACON framework and Spec Kit. The problem statement, the architecture, the decisions and the roadmap are in project-management/.

Glossary

• AAC: a lossy audio format: the decoded audio is not the same as the audio before the encode. An MP4 file usually holds AAC audio.
• AI-Toolkit: the trainer that reads a dataset.yaml or a job.yaml (ostris/ai-toolkit).
• AST: an audio tagger that gives a probability for each AudioSet sound label. The music fact comes from it.
• AUROC: the probability that a score puts a good item above a bad item. 0.5 is chance, and 1.0 is always correct.
• AVDM: the scene document that each caption comes from (the avdm package).
• Channel layout: the position of each audio channel. For example, 5.1 is front left, front right, centre, LFE (the low bass channel) and two rear channels. In 5.1(side), the last two channels are at the sides.
• Claim check: d1-3B's yes or no answer to a question about each claim of the caption model.
• Clip: an MP4 of one shot, with its audio as FLAC, made by ingest --clips.
• Corpus: the folder that ingest makes. Each item holds its facts.
• Cosine similarity: a measure of how close two embeddings are, from -1 to 1. A higher value is closer.
• Cull: the step of ingest that removes near-duplicate and blank frames.
• Cut: a training set that cut writes from a corpus. Each file in it is a copy.
• d1-3B: a LiquidAI model that gives the quality scores and the claim check.
• dBFS: the level of a sound sample in decibels (dB), where 0 dBFS is full scale. A lower (more negative) value is quieter.
• Diarization: a step that finds who speaks when, in an audio track.
• Embedding: a list of numbers for a picture or a text. Items with a similar meaning have similar embeddings.
• Fact: a value that a step measured or made for one item, with its provenance.
• FLAC: a lossless audio format. The decoded audio is the same as the audio before the encode.
• Full scale: the loudest sample value that a digital audio format can hold. The format holds a louder sample at full scale, and you hear distortion.
• Item: one frame, clip or photograph in a corpus.
• Laplacian variance: a measure of the detail in a picture. A blank frame is near 0.
• LoRA: a small set of extra weights that trains a model on one subject or style.
• musubi-tuner: the trainer that reads the files of --trainer musubi (kohya-ss/musubi-tuner).
• Near-duplicate: a frame or file that looks almost the same as another.
• Ollama: the local program that runs the caption model.
• Perceptual hash (pHash): a 64-bit summary of how a picture looks. Similar pictures have hashes that differ in few bits.
• Projection: the form of caption and media that one model trains on.
• Provenance: the record of which producer made a fact, and with which parameters.
• Staging: work on the local disk, then one copy to the destination.
• Trigger word: the word at the start of each caption that calls the LoRA, for example ohwx.
• VLM: a vision-language model, here the caption model in Ollama.
• WD14 rating: four scores from the WD14 tagger, general, sensitive, questionable and explicit, each from 0 to 1. Each score is the probability that the picture has that content rating, from general (safe) to explicit.
• WD14 tags: short Danbooru-style tags from the local WD14 tagger.