caption-audit

Find the tokens that are fused with your LoRA trigger word — before you spend a training run.

pip install caption-audit
caption-audit ./dataset --trigger sks1

Works on any diffusion LoRA dataset: SD 1.5, SDXL, Flux, Qwen-Image, Z-Image, Wan, whatever comes next. It reads captions, not model weights, so it does not care which trainer or base model you use. Both caption dialects are handled — comma-separated Danbooru/booru tag lists and natural-language sentences — and they are analysed side by side in the same run.

The problem it detects

A token that appears in nearly every caption stops functioning as a describable attribute.

The model cannot tell it apart from the trigger word, so it bakes the concept into the identity itself. You can no longer prompt it in, and you can no longer prompt it out.

That is the actual cause of symptoms people usually blame on the optimiser:

• every generation comes out with the same colour cast
• the background texture is forced no matter what you prompt
• lighting is locked to one look
• the character always wears the same thing

It is a caption distribution problem, not a learning rate problem. Lowering the LR will not fix it. Training longer will not fix it. Nothing downstream of the dataset will fix it.

caption-audit measures the distribution and tells you which tokens are already fused, before you burn the GPU hours.

Install

pip install caption-audit          # pip
uv tool install caption-audit      # uv, installs the CLI globally
uvx caption-audit --trigger sks1   # uv, run without installing

Python 3.9+. Zero dependencies — standard library only.

Quick start

# audit the current directory, guess the trigger
cd my_lora_dataset && caption-audit

# the normal case
caption-audit ./dataset --trigger sks1 --class-word woman

# captions and images in separate folders
caption-audit --dir captions --images-dir images --trigger sks1

# an attribute you WANT fused into the character (see --fuse below)
caption-audit ./dataset --trigger sks1 --fuse "red scarf"

# A/B two captioning styles over the same images
caption-audit --compare captions_tags captions_prose --trigger sks1

# machine-readable, for diffing between runs
caption-audit ./dataset --trigger sks1 --format json --out audit.json

Use it as a pre-training gate. Exit code is 1 if anything is CRITICAL:

caption-audit ./dataset --trigger sks1 && python train.py

What it checks

1. Pairing integrity

Images with no caption file, captions with no image, empty captions, files that are not valid UTF-8, stray byte-order marks. If your images live elsewhere, point --images-dir at them; with no images present at all caption-audit runs in caption-only mode instead of screaming about it.

2. Trigger consistency

Coverage (is the trigger in 100% of captions?), position distribution (is it consistently first?), and near-miss spelling variants found by edit distance — the sksperson / sksperso typo that silently removes an image from your concept.

Every --class-word gets the same coverage line, measured and matched the same way. A class word you declared but never wrote into the captions is a WARNING:

   coverage      : 100.0% (35/35)
   position      : idx 0: 35 (100%)
   first token   : 100% of captions
   class woman   : 0.0% (0/35)
   [WARNING] declared class word 'woman' appears in 0/35 captions - either add it to
             the captions or drop --class-word woman

WARNING does not change the exit code — the gate stays CRITICAL-only — but the declaration is never silently ignored.

3. Document frequency — the primary check

For every word, 2-word phrase, 3-word phrase and whole comma segment, caption-audit computes document frequency: in how many captions the term appears at least once.

Not raw occurrence count. A word repeated five times inside a single caption is not the same risk as a word present in every caption, and raw counts cannot tell those apart.

Thresholds are configurable with --critical-threshold, --warn-threshold, --info-threshold.

Declared terms ignore the info threshold. Every --trigger, --class-word and --fuse term gets a row at its real document frequency, including 0.0%, and is never dropped by --max-rows. A term that is absent from the captions entirely is shown in the declared view:

SEVERITY  TERM              VIEW        DF    DF%  P(t|trg)
INTENDED  red scarf         declared     0   0.0%      0.00  (declared --fuse)
EXPECTED  woman             declared     0   0.0%      0.00

Otherwise a declaration you forgot to write into the captions leaves no trace anywhere in the report, and the run passes clean.

4. Trigger co-occurrence

For every flagged token, P(token | trigger present). Tokens sitting at 1.00 are the dangerous ones: they never appear independently of the trigger, so the loss has no example anywhere in the dataset that separates the two concepts.

5. Caption redundancy

Byte-identical captions, and near-duplicates by token-set Jaccard similarity (default >= 0.9). Identical captions on visually different images give the model no signal to disambiguate them, so it averages them. Unless the set is a bare-trigger set — then they are identical on purpose and reported as INFO.

6. Distribution health

Caption length min/median/max/stdev with outlier detection, plus vocabulary diversity — type-token ratio and hapax count. See Reading the numbers.

The interpretation caption-audit refuses to guess

Every CRITICAL and WARNING term has two possible causes, and they need opposite fixes:

(a) Caption hygiene — the concept genuinely varies across your images, but every caption mentions it anyway. The variation exists in pixels and is invisible to the loss because the text never distinguishes it. → Fix: delete the word from the captions where it is not the point.

(b) Dataset composition — the concept really is present in every image. Deleting the word changes nothing; the model still learns it from pixels, it just loses the handle you would have used to name it. → Fix: shoot or collect contrast data.

caption-audit reads captions. It cannot see your images. So it will not guess. For every flagged term it prints the question you have to answer yourself, and the exact files to go and look at:

CRITICAL  red scarf           df 100%  P(t|trigger) 1.00  [bigram]
    - P(term | trigger) = 1.00 - this term never appears independently of the
      trigger. The model has zero signal to separate the two.
    - At 100% document frequency this is fused with the trigger and will be
      unpromptable at inference.
    ? Is 'red scarf' actually visually present in all 35 of those images?
      If yes -> dataset problem (b). If no -> caption problem (a).
    look at: img_001, img_002, img_004, img_005, ... (+31 more)

Doing the wrong one of those two fixes wastes an entire run. That is why it asks instead of telling.

--class-word: what the trigger is a kind of

caption-audit ./dataset --trigger sks1 --class-word woman

A class word is supposed to be in every caption, so it is reported as EXPECTED and never flagged. Declaring it is what stops woman at 100% from being read as accidental fusion.

Repeatable, matched exactly the way the trigger is — case-insensitive, and multi-token class words (--class-word "young woman") are matched as a phrase, so they land on the bigram rather than on two loose words. sks1 woman at 100% is reported as the declared identity bigram, with sks1 and woman shown underneath it as (part of 'sks1 woman').

The declaration is checked, not just believed: coverage below 100% is a WARNING, and the term keeps its table row at whatever its real frequency turns out to be.

--fuse: when fusion is the goal

Sometimes you want an attribute welded to the trigger. A character's signature scarf is part of who they are; a product's logo is not optional. Declaring it says so explicitly:

caption-audit ./dataset --trigger sks1 --fuse "red scarf"

The phrase and all of its fragments (red, scarf, bright red silk scarf) move to INTENDED, stay visible in the table, and stop affecting the exit code — so the gate now enforces your intent rather than a generic default. The declared phrase keeps its row even at 0%, so a --fuse that no longer matches your captions is visible rather than silently inert.

It still reminds you what you signed up for: once trained, you will not be able to prompt that attribute away or vary it.

--compare: A/B your captioning

Same images, different caption sets. The only thing that moves between columns is the caption distribution:

caption-audit --compare tags/ prose/ prose-no-hair/ --trigger sks1

                              tags             prose   prose-no-hair
captions                        35                35              35
trigger coverage              100%              100%            100%
trigger first token           100%              100%            100%
type-token ratio             0.284             0.265           0.310
CRITICAL / WARNING           6 / 0             4 / 5           0 / 0

TERM                          tags             prose   prose-no-hair
red scarf                    100%!             100%!               -
blurred                       89%!               43%               -
camera                           -              63%*             54%

! = CRITICAL, * = WARNING, - = below the info threshold in that set.

Fewest findings does not automatically mean best LoRA — it means the most spread caption distribution. caption-audit says so in the verdict rather than letting you read it as a score.

Reading the numbers

Document frequency (DF) — the fraction of captions containing a term at least once. This is the whole idea. A word in 100% of captions carries no information: it cannot explain any difference between two images, because it is identical in both. Information lives in variation.

P(token | trigger) — among the captions that contain the trigger, the fraction that also contain this token. At 1.00 the two are inseparable in the data.

Type-token ratio (TTR) — unique words divided by total words. 1000 words of caption using 300 distinct ones gives 0.30.

• high TTR (~0.5+) → you describe each image in its own words. The model sees many ways of phrasing things and generalises to phrasings it has not seen.
• low TTR (~0.2) → you reused the same sentence skeleton everywhere. The model over-fits the phrasing, and paraphrases at inference land off-distribution. Prompts get brittle: the exact wording from your captions works, anything else degrades.

There is no universal target — it depends on caption length and dataset size — but a big gap between two caption sets over the same images is meaningful, which is what --compare is for.

Hapax count — words appearing exactly once in the whole dataset. High hapax with high TTR is healthy specificity. High hapax with low TTR usually means a rigid template plus a few unique nouns bolted on.

N-grams and the segment view — caption-audit analyses single words, pairs, triples, and whole comma segments separately, because a phrase is a stronger signal than any of its words. soft paper texture locked at 93% matters more than soft at 93%. N-grams never cross a comma, so tag lists do not produce phantom phrases like hair wearing.

Subsumption — when soft, paper, texture, soft paper and soft paper texture all sit at the same DF, that is one finding seen at five resolutions, not five findings. caption-audit attributes it to the longest phrase and marks the fragments (part of '...'). They stay in the table; they do not get their own interpretation block or inflate the count.

Function words are counted but kept out of the flag list, since a, the and with being everywhere tells you nothing. --no-stopwords turns the filter off.

Relational words — wearing, holding, carrying, showing, featuring and their inflections — are treated the same way, and so is any phrase ending in one (woman wearing).

wearing in 60% of captions is not a bias. There is no other word for it, and every clothed subject is wearing something; deleting it is not a fix and neither is shooting contrast data. What can be biased is the object it introduces, and that is already measured on its own — red scarf and wearing a red scarf are flagged normally. So the connector is reported as a footnote under the table with its real DF, and left out of the flag list, the interpretation section and the exit code:

   4 term(s) above the info threshold are not flagged: function and relational words
   relational, so the bias would be in what follows, not in the word itself: wearing 70%

--no-stopwords puts them back.

Bare-trigger sets

Writing only the trigger in every caption and letting the images carry everything is a real captioning style. It also produces byte-identical captions and a type-token ratio near zero, which is what the duplicate and diversity checks exist to catch — so the style used to fail its own audit with a CRITICAL and exit 1.

When every caption is the trigger plus any declared --class-word and nothing else, caption-audit recognises the shape and says so:

mode     : bare-trigger set - every caption is the trigger and nothing else.
           Duplicate and diversity checks report INFO.

Both findings drop to INFO and the run exits 0. Nothing is hidden — the duplicate group and the ratio are still printed, with the trade spelled out: nothing in those images is separately promptable, which is what you chose. It is detected, not declared: one descriptive caption in the set and the exemption is gone, because a half-captioned set is not a style.

A --class-word you declared but never wrote still warns. The style explains the duplicates; it does not excuse a word that is not there.

Options

input
  DIR                       directory to audit (default: current directory)
  --dir DIR                 same thing, explicit
  --images-dir DIR          where the images live, if kept apart from captions
  --recursive               descend into subdirectories
  --compare DIR [DIR ...]   two or more caption sets, side by side

what the trigger is supposed to own
  --trigger TRIGGER         the trigger word; inferred and marked [INFERRED] if omitted
  --class-word WORD         e.g. 'woman', 'car'. Repeatable. Shown as EXPECTED, never flagged.
                            Coverage below 100% is a WARNING
  --fuse TERM               an attribute you WANT fused. Repeatable. Shown as INTENDED,
                            excluded from the exit code

thresholds
  --critical-threshold F    default 0.85
  --warn-threshold F        default 0.60
  --info-threshold F        default 0.35, below this nothing is reported
  --ngram-max N             longest phrase analysed, default 3
  --jaccard F               near-duplicate threshold, default 0.9
  --no-stopwords            stop filtering function and relational words out of the flag list

output
  --format {term,md,json}   term (default), markdown, or JSON for diffing runs
  --out FILE                write to a file instead of stdout
  --list-files N            name up to N caption files per flagged term, default 10, 0 to hide
  --max-rows N              table rows, default 60
  --max-interpret N         terms expanded with full interpretation, default 12
  --no-color                disable ANSI colour (also honours NO_COLOR)

Supported image extensions: .png .jpg .jpeg .webp. Captions are .txt sidecars sharing the image's basename.

Exit codes: 0 clean, 1 at least one CRITICAL finding, 2 bad input. WARNING findings are reported but never change the exit code.

Use in CI

- name: Audit caption distribution
  run: |
    pip install caption-audit
    caption-audit dataset/ --trigger sks1 --fuse "red scarf" --format md --out audit.md

The run fails on CRITICAL findings, and audit.md is a readable artifact. With --format json you can diff two runs and see exactly which term moved.

Library use

from caption_audit.loader import scan_dir
from caption_audit.checks import check_document_frequency, check_distribution

captions, images, orphans = scan_dir("dataset/")
stats = check_distribution(captions)
print(stats["type_token_ratio"], stats["hapax"])

Every check returns plain dicts. caption_audit.cli.analyze(directory, args) gives you the whole report object, which is exactly what --format json serialises.

Development

git clone https://github.com/0xBeycan/caption-audit
cd caption-audit

python -m unittest discover -s tests   # no dependencies at all
uv run --extra dev pytest -q           # same 24 tests, nicer output
uv run --extra dev ruff check .
uv build

CI runs the suite on Python 3.9–3.14, lints, then installs the built wheel into a clean environment and re-runs the tests plus an end-to-end exit-code check against the console script.

The test suite builds a synthetic fixture containing one of each defect — a fused adjective phrase at 100%, a duplicate caption pair, a near-duplicate pair, an orphan image, an empty caption, a non-UTF8 file and a trigger typo — and asserts that every check fires. A second fixture covers declared terms as a differential pair: the same captions with and without the class word actually written into them, so a declaration at 0% and the same declaration at 100% are both pinned.

License

MIT