scourgify 1.0.0

Normalize and consolidate tags/fandoms/characters/genres in a FanFicFare-imported Calibre library

scourgify

The tag-wrangler / canonizer for your fanfiction library — normalize and consolidate tags, fandoms, characters, relationships and genres in a FanFicFare-imported Calibre library. Data-driven from ~1,700 bundled generic defaults, fully customizable, audit-first and reversible.

pipx install scourgify                                # or: uv tool install scourgify  (one dependency: rich)

export CALIBRE_LIBRARY="$HOME/Calibre/fanfiction"     # folder containing metadata.db
scourgify                                             # ← the wizard: everything below, menu-driven

Requires Calibre installed — the tool reads via read-only sqlite and shells out to Calibre's own calibre-debug for writes. From a checkout, uv run scourgify (or uvx --from . scourgify) runs it without installing; uv handles the environment (one dependency: rich).

The wizard (no arguments) is the intended way in — and it steers you: on a fresh library it detects missing columns/config and offers to run setup immediately; afterwards the status header shows book count, column health, how many books are new/changed since the last run, and any pending proposal, and the menu defaults to whatever the library needs next. Menu item 0 — maintenance walks the whole loop in the right order (wrangle → staleness → classify → review), each step explained, previewed, and skippable. Classify runs show a live dashboard (progress + tagged/failed/rate + throughput sparkline + rising tag candidates). Every write previews first, asks for confirmation, and auto-backs-up metadata.db.

Each step is also a plain scriptable subcommand:

scourgify setup                                      # interactive health check + setup (FanFicFare, columns, config)
scourgify audit                                      # read-only dry-run report of every pass
scourgify apply --apply                              # write changes (Calibre CLOSED for this step)

Everything runs under plain python3. The tool reads via read-only sqlite and, for the actual writes, shells out once to calibre-debug -e _writer.py (Calibre's API is the only fast batch-write path) — so any command that writes (apply --apply, setup creating columns, classify.py --apply) needs Calibre closed, and refuses to run while it's open. You never invoke calibre-debug yourself.

setup is the first-run wizard + re-runnable health check. It verifies, with ✓/⚠/✗ status and Y/n prompts (default-yes; --yes to auto-accept): the library; that the FanFicFare plugin is installed and configured, flagging + offering to fix the known gotchas (fandom-vs-series mapping, include_in_series:category, unprotected #genres); that every needed column exists (#fandoms, #characters, #relationships, #genres, #status, plus #updated and #wrangled for staleness / incremental classification), creating any that are missing; and writes config.toml (preserving your behavior toggles). Safe to re-run anytime.

rich is required for the wizard and powers the live dashboards/tables everywhere else; the plain subcommands still degrade to text without it (rich is try/except-imported in the core tools, so scripting/CI without rich keeps working, and _writer.py under Calibre's bundled Python needs none).

The engine reads:

• defaults/ — bundled, generic fanfic knowledge (FFN Harry P.→Harry Potter, JP→English fandom titles, no beta we die like… = junk, …). Ships with the tool; not specific to anyone.
• config.toml — your column mapping + opinionated behavior toggles (generated by setup).
• overrides/ (optional) — your own files (same formats as defaults/) that extend and win over the defaults.

---

FanFicFare → Calibre columns (how the linking works)

FanFicFare scrapes metadata fields from each story and writes them into Calibre columns. The mapping lives in the FFF Calibre-plugin config (stored per-library in the metadata.db preference namespaced:FanFicFarePlugin:settings → key custom_cols). scourgify's setup reads that mapping, and creates any recommended columns you're missing.

Recommended mapping (FFF metadata field → Calibre column):

FanFicFare field	Calibre column	Type	Holds
category	#fandoms	text, multiple	fandom(s) — map this to category, not series (see gotcha)
characters	#characters	text, multiple	characters
ships	#relationships	text, multiple	pairings
genre	#genres	text, multiple	genres
status	#status	text	In-Progress / Completed / …
series	Series (built-in)	series	the real site/AO3 series
numWords	#words	int	word count
numChapters	#chapters	int	chapter count
dateUpdated	#updated	datetime	last-updated date
storyUrl	#storyurl	text	source URL (also stored as the url identifier)
subject tags	tags (built-in)	—	freeform tags (what this tool normalizes most)

⚠️ The fandom-vs-series gotcha

FanFicFare's personal.ini setting include_in_series:category stuffs the fandom into FFF's series field. If your custom_cols then maps #fandoms ← series, two things break: your Series column fills with fandom names (not real series), and #fandoms is fed from that fandom-stuffed series field. Fix:

1. Remove include_in_series:category from personal.ini → series becomes the real (e.g. AO3) series.
2. Map #fandoms ← category (the true fandom field).

wrangle.py setup detects and offers to fix both, plus the protection below (the original attic/apply_fff_config.py does the same standalone). After that, real series fills in going forward and fandoms come from category.

Why fandom-as-series is especially bad: Calibre's Series is a numbered field — every book gets a series_index (A Fandom Name [1], [2], …). So fandom-as-series doesn't just duplicate the fandom, it invents a bogus ordered hierarchy: dozens of unrelated stories become "book 1, book 2… of Harry Potter," a sequence that reflects nothing real. Clearing it (see attic/apply_other.py) and mapping #fandoms ← category removes the fake ordering; real series (where the index is meaningful, e.g. a genuine 3-part AO3 series) then populate correctly.

Franchise unification (fandom granularity)

Related works in one universe (e.g. Fate/stay night, Fate/Zero, Fate/Grand Order) are distinct titles but one fandom. The bundled defaults/fandoms.csv unifies the obvious franchises to a single canonical — the Fate/Nasuverse works all map to Type-Moon (the studio/umbrella name the Nasuverse fandom uses). Prefer an English title as canonical wherever one exists (e.g. The Saga of Tanya the Evil, not Youjo Senki; Puella Magi Madoka Magica, not the romaji). This is a granularity preference: if you'd rather keep Fate/Zero separate from Fate/stay night, remove those rows from your overrides/fandoms.csv (or leave them unmapped). Note some franchises should stay split — Disney works are mostly standalone worlds (keep DuckTales, don't fold to a Disney mega-fandom), and Overlord (Game) vs Overlord (Anime) are unrelated. Curated unifications live in build_defaults.py's CURATED_FAN.

Protecting your cleanup from re-pollution

FFF's custom_cols_newonly ({column: bool}) controls overwrite-on-update: when true, FFF only writes that column if it's empty, so a metadata refresh won't clobber your normalized values. Recommended: newonly:true for #genres; leave #status writable so FanFicFare refreshes it on fetch (staleness.py re-derives the activity inference). The built-in tags column is never protected by this — so new downloads/updates re-add raw tag junk, and you re-run wrangle.py to clean it (see Maintenance).

---

Customizing

config.toml — column map + behavior toggles (all have sane, opinionated defaults):

Toggle	Default	Effect
fold_characters	true	apply abbreviation→full-name defaults
ascii_only_tags	true	transliterate non-ASCII tags to plain ASCII
au_as / crossover_as / reincarnation_as / time_travel_as	genre	put these tropes in #genres (tag to keep in tags)
fold_ratings	false	fold Erotica→Smut, Adult→Mature
keep_categories	true	keep Multi/Gen/F/M tags (false drops them)

overrides/ — drop in characters.csv, fandoms.csv, tropes.csv, junk.txt, genres_allow.txt, … (same formats as defaults/). Anything here is merged on top of the bundled defaults and wins on conflicts. This is where your preferences live — the code stays generic.

defaults/ formats:

• characters.csv — variant,canonical,fandom (blank fandom = global; set = homonym-scoped, e.g. Luke C. differs in Marvel vs PJO)
• fandoms.csv — alias,canonical
• tropes.csv — variant,canonical,route (route = tag|genre|character|fandom)
• genres_split.csv (combined,atoms), genres_canon.csv (variant,canonical), genres_allow.txt (the genre vocabulary)
• junk.txt — drop list (plain line = case-insensitive exact; re:<regex> = regex)
• ratings.txt — content-rating/warning vocabulary

---

Safety model

audit and apply compute the full new state in memory and assert no book loses its last fandom or character (backfill-before-strip), aborting without writing if that fails. A second guardrail aborts if tags would mass-shrink (>25% of assignments and >200 lost — the signature of an over-broad junk.txt rule; --force overrides after you've checked). A redundant tag is only stripped when the concept already lives in that book's structured column. audit is read-only (plain python3, fine with Calibre open); apply/setup use the Calibre API (Calibre closed). Every write automatically snapshots metadata.db to /tmp/ff_<timestamp>.db first and prints the path — that's your instant rollback (master rollback = a full "Export all Calibre data" backup).

Maintenance — after new downloads / updates

New stories arrive raw (junky subject tags, unfolded names). Order matters — deterministic cleanup first, content tagging second, because raw junk tags inflate a book's tag count and would hide it from the classifier's "sparsely tagged" targeting:

scourgify apply --apply                # 1. wrangle FIRST: junk-drop/canonicalize the new raw tags (idempotent)
scourgify staleness --apply            # 2. free: re-derive #status from #updated age (independent, any time)
scourgify classify --incremental       # 3. cheap: content-tag only new/changed books -> proposal
                                       # 4. review data/classify_proposal.csv
scourgify classify --apply             # 5. apply the reviewed tags + stamp #wrangled

Or just scourgify and walk the wizard menu in order: 3 wrangle → 4 staleness → 5 classify → 6 review. Re-running wrangle is always safe — it's idempotent and won't regress curated genres (it uses the full genres_allow.txt).

---

Content-based tagging — scourgify classify

Reads each book's description (#comments) and produces two outputs: (1) added_tags — tags chosen from the controlled vocabulary (defaults/classify_vocab.txt), which get applied; and (2) proposed_new — short novel tags not in the vocab, aggregated by frequency into classify_newtags_ranked.csv so you can review and promote the recurring ones into the vocab. Grows the tag set deliberately, without freeform noise.

scourgify classify --engine apple --limit 50        # propose -> data/classify_proposal.csv (dry-run, read-only)
scourgify classify --apply                          # add the proposed tags (Calibre CLOSED)

• --engine apple — on-device Apple Foundation Models via afm.swift (free, private; macOS 26+, Apple Intelligence). Ships as source; a swift toolchain runs it as-is, or from a checkout build the faster binary once: swiftc -O src/scourgify/afm.swift -o src/scourgify/afm. Lower quality — prone to over-tagging, so the prompt caps at --max-tags 6 and dumps (>2× cap) are rejected.
• --engine claude|openai|gemini — cloud APIs (ANTHROPIC_API_KEY / OPENAI_API_KEY / GEMINI_API_KEY); defaults claude-haiku-4-5 / gpt-4o-mini / gemini-2.5-flash, override with --model. Sharper; cheap.
• Only books with < --min-tags (default 2) tags and a description are touched. Always dry-run until --apply. Edit defaults/classify_vocab.txt to shape the allowed tag set.
• Long runs save incrementally and resume on re-run (skip books already in the proposal; --fresh to restart). --batch N processes only N new books per run — handy for pacing API spend/rate limits. A spend gate asks for confirmation (or --yes) before sending more than 200 books to a cloud engine.
• Proposals/outputs live in data/ (gitignored): classify_proposal.csv, classify_newtags_ranked.csv, classify_failures.csv. On --apply the proposal is archived to classify_proposal_applied_<ts>.csv, so a later apply can never re-add tags you've since hand-removed in Calibre.
• Incremental maintenance (--incremental): after new FanFicFare downloads, classify.py --incremental (re)tags only books whose #updated is newer than their own #wrangled marker (a datetime column auto-created and stamped on --apply), plus any still untagged — cents instead of a full pass. State lives in the library (travels with metadata.db, no external file). A full cloud --fresh run is expensive; reserve it for vocab changes.

Custom maps from your library (overrides/)

The bundled defaults/ are generic. Two helper workflows mined library-specific maps into overrides/ (gitignored): AO3-style tag clustering (overrides/tropes.csv) and fandom universe-unification (overrides/fandoms.csv, e.g. Avengers/Captain America (Movies) → Marvel, Game of Thrones (TV) → A Song of Ice and Fire). The engine loads these on top of the defaults automatically.

Repo layout

The package lives in src/scourgify/; the single scourgify command (cli.py) dispatches bare → wizard, setup/audit/apply → wrangle, classify, and staleness.

• cli.py — the scourgify entry point (argv dispatcher over the tools below)
• wrangle.py — the engine: setup / audit / apply; with no command it launches the wizard
• wizard.py + ui.py — the interactive wizard and its rich terminal helpers (the one rich-required surface)
• classify.py — content-based tagging (LLM engines) · staleness.py — #status re-derivation
• common.py — shared core: library resolution, read-only sqlite + custom-column reading, config, and run_writer() (the single write funnel, with automatic backup)
• _writer.py — the only file that imports Calibre; a generic ops executor invoked under calibre-debug by run_writer(), never by hand
• defaults/ — bundled generic maps, shipped inside the package (read-only at runtime)
• Per-user files resolve against the working directory: config.toml, overrides/ (your maps, gitignored), and data/ (proposals/intermediates, gitignored)
• build_defaults.py (repo root) — maintainer tool: regenerates defaults/ from the review maps in data/
• tests/ — uv run tests/test_core.py, no library needed
• attic/ — the original single-purpose pipeline, kept as provenance (see attic/README.md); scourgify supersedes it.

The per-library review-map CSVs in data/ are gitignored (they contain your library's actual data); only the generic defaults/ ship with the repo.