myMend repairs OCR'd books. mend.pipeline is the authority: it keeps complete document revisions in PostgreSQL and runs explicit Actions that change them. rinse and clean are the deterministic tools around it: rinse builds an edition from raw OCR, and the pipeline's cleaning Action runs clean's rules. Nothing in the package calls a model. INTAKE's eight verbs on the Store are in docs/intake.md, PROOF's generic runner in docs/proof.md, and SQL-backed work-level reading in docs/native-proof.md.

Left-rule blocks mark artificial prose; quotations retain their own attribution.

Repair history can also develop executable rules through finite, resumable batches. The symbolic cleanup guide connects preparation, learning, qualification, independent feedback and public CLEAN application. Search checkpoints preserve completed trials and costs; learned artifacts remain explicit, with their qualification and source-evidence grades.

Installation

Python 3.13 or later is required.

From a checkout, in a virtual environment:

$ uv pip install .
# or
$ python -m pip install .

Commands

The package installs three console scripts. mend-pipeline is the pipeline and the one to learn first; docs/pipeline.md is its manual. rinse and clean are the deterministic tools around it (a cleaning Action runs clean's rules), and they stay usable on their own. None of them calls a model.

$ clean works/                       # repair in place
$ clean --check works/               # write nothing; exit 1 if anything would change
$ clean --diff works/hegel/chapter-003.md
$ clean --check --report edits.jsonl works/   # every edit and detector hit, with context
$ clean --no-corpus --summary convergence.jsonl draft/ # isolated context; explicit fixed-point receipt
$ clean --check -j 12 --rule spacing works/   # one concern (or `concern.rule`), 12 works at a time
$ clean --check --corpus ~/my/lit raw/book.md  # weigh rare tokens against all of lit's English text
$ clean --family legal --skip legal.page_footer docs/   # a document family; leave one rule out
$ clean --list-rules --family legal              # what the family runs, rule by rule

Rinse

rinse turns the raw OCR of a book into a lit edition, structure first. It reads a work's source (replay/sources/<hash>.md, read only) and, when the raw shelf holds a byte-identical copy, Marker's .metadata.json beside it. It writes index.md and chapter-NNN.md and nothing else: the pictures under assets/ and at the root of an edition are never moved or deleted, and only stale chapter-*.md files are removed. The build is deterministic and idempotent: a second run changes nothing. It makes no model call.

--intake takes a candidate ledger in the schema of lit's retired intake.yaml, which lit deleted on 2026-10-19 (MEMY-2113). The rinse Action on the Store replaces that admission (docs/intake.md, section 4); until then the command runs only against an explicit ledger file.

$ rinse hegel-g__1807__phenomenology-of-spirit --intake intake.yaml   # rebuild a lit work in place
$ rinse --check --diff WORK... --intake intake.yaml   # write nothing; print the diff; exit 1 if any edition would change
$ rinse WORK... --intake intake.yaml --out /tmp/sample   # write /tmp/sample/<slug>/ and leave lit alone
$ rinse --source book.md --intake intake.yaml --out ~/ai/scratch/new   # admit a new source by the ledger

Exit codes follow clean: 0 when nothing is wrong, 1 when --check or --diff finds an edition that would change, 2 when a work could not be rinsed.

--intake FILE is required. It names lit's generated lit.intake.v1 ledger, which is the only admission policy. A new source must match both the recorded path and full lowercase SHA-256, have status verified or promoted, and have a supported tier (core, primary, secondary, tertiary, auxiliary, nonacademic, or engineering). The ledger supplies the slug, title, author and year, and a source's own YAML header never overrides them. Output goes to --out/<slug>/. Relative ledger paths resolve beside the ledger. Candidate, held and withdrawn rows may retain null hashes for missing evidence with an explicit verdicts.reason; verified and promoted rows require full hashes. source.metadata may name a Marker metadata file; otherwise the source's .metadata.json sibling is used, only while the recorded source bytes match. Missing or malformed ledgers, duplicate slugs/source paths/source hashes, held or withdrawn candidates, unknown tiers and changed bytes fail with exit 2 before an edition is written. --json includes the refusal in an error field. The ledger reader does not scan the shelf or fall back to ai.

An edition is index.md (the curated front matter kept key for key, a title, a byline and a toctree), chapter-000.md (# [Front matter], each front item an H2, the printed contents kept as a nested list of titles), and one chapter-NNN.md per top-level section of the work, cut at H2 boundaries only above about 250 KB. Square brackets mark a heading rinse added ([Front matter], [Text] for a work with no headings, [Title, continued]); no other heading is ever invented. The back-of-book index is dropped, and it ends where the index does: at the heading of the notes behind it, or of the errata, abbreviations, bibliography or books referred to that follow it, which stay with their pictures. An index set in long paragraphs of entries is still an index: a paragraph in which a tenth of the words are page numbers or ranges (223, 290-291; xxix-xxxi;) is no running text, and a headword with no number of its own (Greece) does not end an index whose entries follow it. A line that opens with a note number is a note, never a running head, however often its words recur. Inherited nonempty image alt survives an empty or missing target with a visible limitation; available images retain access text separately from printed captions (see image access). % page N comments mark the first block of each PDF page the source lets rinse place, strictly increasing, never guessed.

Heading inference

Marker's ATX levels are noisy (a book's chapters may come out at #, ## and ### by turns), so rinse reads each heading's numbering (Chapter IV, II., b., 3.2.1, § 12, (AA.), Teil 2) as a set of candidate readings and searches, over the whole sequence, for the nesting that breaks the fewest habits of books: a number that continues its series is a sibling, one that restarts it (A. after IV.) opens a deeper level, Part outranks Chapter, a dotted number sits under its parent, and an unnumbered Proof or Remark is a leaf. Marker's own level breaks ties. A last pass puts the headings that follow a Volume, Part or Book inside it, up to the next one of its size or the notes and bibliography behind the work, whatever level Marker gave them; the Parts of a series that runs through the whole work (no place comes twice) are raised when Marker set them below their chapters, and a Part and a Book do not hold one another, since works put either first. A heading is never added except from evidence: a short line that Marker's table of contents lists, a line that is nothing but Preface or CHAPTER 3, or (in a source with no headings) a title the printed contents lists. Paragraph-length "headings", running heads, contents and index entries are turned back into text. Depth runs to six.

Whatever rinse decides that a rerun must repeat is written under one rinse: key of index.md: body_start (the heading that opens the work), demote and promote (titles that are, or are not, headings), levels (a title's forced depth) and keep_sup (letters whose superscripts are notation in this work). A curator edits that key; rinse reads it and never overwrites it.

The pattern library

The CLEAN families guide groups the current repairs, explains existing note support and distinguishes lexical guards from embeddings or POS tagging.

clean runs contextual OCR rules. Its actual output then uses the shared canonical reading policy: ordinary searchable accents, quotes, fraction characters, ellipses, dashes, ligatures and spaces. The policy is independent of language guessing and family selection; --rule and --skip select contextual repairs while canonical output still applies. Human-readable bibliographic fields and editorial access text follow the same policy.

A rule names one pattern in a concern's RegexStore and says what becomes of each match: a template string, a niche function (match, context) -> str for decisions a template cannot make, or nothing, which makes it a detector that reports hits for reviewers. Rules run in order, one whole-text pass each, and never touch a fence: front matter, code, math, inline code (idx tags), roles, comments (page labels), HTML tags, link destinations, footnote labels and URLs (mend/fences.py). A niche function sees the whole work, not just its file, so a word split in one chapter can be rejoined as another chapter spells it. A rule that is right in some languages only names them (languages=('en',)), and the rulebook skips it for a work in another: a work's language is guessed from its function words (mend/language.py), and a short text that cannot be told counts as other.

Two more gates keep a rule to the text it was written for. A rule marked prose=True runs only in English running prose, judged by the share of function words, so a table, a bibliography or a page of notation is left whole. A rule marked corpus=True weighs a token against the whole library: Context.corpus counts every token of every English chapter in works/ (mend/corpus.py, cached under $XDG_CACHE_HOME/mend), because damage is rare and the word it hides is common (rnight once, might ten thousand times). Without a corpus such a rule makes no edit. The lexicon (mend/lexicon.py, mend/data/english.txt) answers the other question, whether a string is a word at all.

For its reviewed default rulebook, the CLI keeps its newly loaded corpus counts in a read-only snapshot shared across the run. Custom rulebooks and caller-supplied corpora keep their existing live mapping behavior.

A repair can make another one visible, so clean repeats the rulebook over a work until a round leaves final canonical text unchanged (five rounds at most), and rebuilds the evidence between rounds. A final round without a text change establishes convergence, including when it is the fifth round. Text still changing at the cap means convergence was not established: the command reports cap exhaustion and exits 2. In ordinary writing mode it still writes the capped result; callers must inspect the exit code before accepting it. One limit remains: cleaning lit itself changes the counts the corpus gives, so a first run over lit can leave a few dozen splits that a second run finds, and a third finds none.

--summary FILE writes one JSON-lines convergence record per selected file, even when nothing changes: file, kind: "summary", changed, rounds, converged and final_round_edits. Convergence and the final edit count describe the whole work because sibling files supply evidence. This is separate from the unchanged --report edit/hit format. Summary and edit reports must be distinct files outside cleaning inputs. clean writes by default; --check and --diff leave inputs intact. --check exits 1 on changes; legacy --diff alone exits 0 even when it prints changes, so an empty-diff check must require both success and empty stdout. Cap exhaustion exits 2 in either mode.

--no-corpus explicitly skips corpus token context and catalog title context, including automatic ancestor discovery. Rules requiring that context make no corresponding edits; diagnostics name skipped context. Work sibling evidence still runs. Record this selection with a preparation recipe rather than treating skipped context as a passed check. The option is mutually exclusive with --corpus ROOT; ordinary discovery defaults remain unchanged.

clean --rule prep --corpus /path/to/lit repairs mixed existing ellipses and reports Greek. Inferring an omission from two dots requires reviewed opt-in (--rule prep.two_dots --force); a dominant ellipsis style alone does not prove that the dots denote an omission. Reviewed title styling requires clean --rule prep.title --force --corpus /path/to/lit; catalog membership alone does not establish a quoted title's style. The catalog supplies full titles and pre-colon short forms; capital-started quoted matches become italics, except common one-word titles, nested quotations and existing emphasis. Missing or malformed catalogs make no title edit. Damaged ellipses use the unique dominant intact form in the work's unprotected text; ties, missing evidence and a lone adjacent sentence stop stay for the reader. Greek words are reported without editing; near-miss catalog titles still need a reader.

source.exact uses the finite readings and scope policy in mend/data/source_repairs.yaml. Missing structural evidence refuses a repair; code and markup cannot supply corroboration. This is not general running-head removal or junk-suffix stripping. source.attached_link reports forms such as DT[theorem](problem) without escaping them: a deliberately attached local Markdown link can have the same syntax. Use its report for source judgment; it is not a strict-build gate.

Frequency-based spelling repair preserves all-uppercase technical acronyms, named literals, apostrophes/primes and known lexicon words. Diagram separators remain intact. The reviewed compound policy in mend/data/lexicon.yaml keeps well-known when the work supplies no explicit closed spelling; rebuilding the dictionary retains that exclusion.

Document families

One library repairs more than one kind of document. A family is a profile of the rules (mend/data/families.yaml), not a fork: for each family, the rules it runs, the rules it only reports (a repair listed under detect runs as a detector), and the rules it switches off. --family NAME picks one; the default is academic, an explicit profile that leaves catalog-title styling and frequency-based OCR substitutions off until reviewed opt-in (--rule prep.title --force or --rule words.ocr_swap --force). Frequency alone cannot establish the intended word: valid rare words, page continuations and numbered references can resemble more common words. Exact misread and source-proved repairs remain enabled. The default and explicit family still agree (tests/test_families.py holds that against the existing cases and frozen long texts). legal is for court papers, transcripts, mail and records: the 21 rules that the APPS-497 pilot found safe on 398 such documents, no rule that reads lit, and the legal concern, which adds a rule only this family has (Rule.families). A family places every rule of its library exactly once, so a rule added later fails the tests until someone decides where it goes.

The academic profile also runs words.split_lexical for guarded split-word repair. It protects explicit spelling/specimen/identifier contexts, code, math and meaningful differance distinctions; ordinary quoted prose remains eligible. Use --skip words.split_lexical to omit it. Unique repeated same-work lost-letter recovery remains available explicitly through --rule glyphs.work_diacritic --force or --rule glyphs.work_ascii --force; the legal profile keeps all three off. Canonical accent folding still applies to resulting prose.

--skip ID leaves out a rule or a concern, and may be repeated. --rule stays include-only, and in a profiled family it may name only rules the family runs; --force-rule lets it name a rule the family has switched off, and run it as the library registers it. --list-rules prints what the family does with each rule and exits. --stage published is for text that has already been split into pages: a rule limited to a stage (Rule.stages) runs only there, and the default stage is source. The family, skip list and stage reach every worker of -j N.

The legal concern ports ten of the twelve accepted epstein_legal fixes. It repairs a redaction bar set as block characters ([REDACTED]), a Page X of Y line and, at the published stage only, a bare EFTA Bates line; it reports a bare run of six digits or more and the stamps CONFIDENTIAL, PRIVILEGED, EXHIBIT and RELEASED PURSUANT TO FOIA; and it holds two repairs, for FILED: and RECEIVED: date lines, off until a corpus shows them. The disclaimer that a mail system appends to a page is a bounded repair (legal.disclaimer): it removes the notice, from please note at the head of a line to its last sentence or, when the page broke inside it, to the Bates stamp that follows, and never the stamp, a signature or any text after it. A repair that removes a line leaves the line break behind, as the original fixes did, except page_footer and legal.disclaimer: when one of them takes a whole line it takes the line break too, so that a paragraph cut by a page break is whole again, and when text or a stamp shares the line it leaves the line as it was. The legal profile leaves legal.disclaimer off, with four of the pilot's repairs (glyphs.glyph, punctuation.double_period, punctuation.space_colon and punctuation.space_stop), because a review of 40 sampled edits per rule found more than one wrong edit in each (D-22); --force-rule still runs one, as the library registers it, and the academic family is unchanged. It runs one hyphen join, hyphen.line_break, which the pilot read at 6 of 26 wrong and which a fresh review of its edits decides (D-26); hyphen.paragraph_break, hyphen.broken_word and hyphen.dot_break stay off. The cases are tests/examples/clean.legal.yaml.

$ clean --family legal --check --report edits.jsonl docs/
$ clean --family legal --stage published pages/       # also removes bare EFTA lines
$ clean --family legal --rule hyphen.broken_word --force-rule note.md   # a rule the family switched off

Adding a rule

The library grows by turning accepted proofreading experience into tested symbolic capability. The operator's vision explains the purpose; the learning specification defines finite learning cycles, narrowly scoped cases, uncertainty-based review and the evaluation needed to measure useful growth. The source-scoped learned-case contract defines explicit source/member applicability, stable case attribution and Store-backed identity propagation. Standalone cleaning with unknown identity leaves those cases inactive. Contributors should be able to extend another agent's work without reconstructing its session. The tests hold every rule to the same contract.

1. Put the pattern in the concern's store (mend/stores/<concern>.py), named, using the regex module's syntax through RegexStore (subroutines with (?P>name), atomic alternations with the '<|>' mark); never stdlib re. A new concern is a new module with NAME, STORE and RULES, listed in mend/stores/__init__.py.
2. Add a Rule to the module's RULES: a template, a niche function, or no replacement for a detector; add languages= when the repair is right in some languages only, prose=True when it is right in running English prose only, corpus=True when it weighs a token against all of lit, encloses=True when its match may take whole fences with it, and families=('name',) when the rule belongs to one document family only (a concern of its own, listed under that family in mend/data/families.yaml; stages=('published',) limits it to a stage). Literal swaps belong in a YAML table under mend/data/.
3. Add pattern cases to tests/examples/regexes.yaml (what matches, what must not).
4. Add rule cases to tests/examples/clean.yaml (clean.legal.yaml for a rule only the legal family has): at least one input the rule changes (or reports) and one it must leave exactly alone. A case may carry lang: en to fix the language a short text cannot show, and lit: {token: uses} to give a corpus rule the counts it weighs (every other token of the text counts as common). Every case also runs twice; the second pass must change nothing.
5. Run task test -- tests/test_clean_snapshots.py. It compares frozen long texts, work evidence, edits, and detector hits. Read every changed passage before accepting an intentional update with task test:ssu -- tests/test_clean_snapshots.py, then inspect git diff -- tests/__snapshots__/test_clean_snapshots.ambr. Unexpected changes become negative cases. See tests/fixtures/clean/README.md for provenance and the update contract.
6. Run clean --check --report edits.jsonl --rule <your rule> over lit's works/ and over the raw sources in replay/sources/ (lit holds what was already repaired, the sources hold the damage), read a sample of the rule's edits yourself. Read 40 sampled edits per rule, with a seed you record. A repair lands when at most one of the 40 reads wrong and you have looked at that one. A misfire becomes a leave-alone case, and so does any clean edit a reviewer later reverses.

Outside the library the same rule holds: a module that needs a pattern keeps it in a RegexStore called STORE beside its code, with a NAME, and lists the module in mend/stores/support.py (rinse's modules in mend/rinse/stores.py), so that tests/test_support_regex.py asks for cases in tests/examples/support_regexes.yaml. Write a space or a word as the module's old engine read it when a pattern moves from stdlib re (RE_SPACE and RE_WORD in mend/stores/common.py), and hold the move to the old pattern in tests/test_legacy_regex.py. The one pattern that stays raw is smoke's bytes pattern for PDF streams: a RegexStore composes str only.

Downstream pins

Corpus pins this package by git revision and moves its pin deliberately. apps/epstein-index pins an older revision (1cc7b170) that still carries the retired mend.proof harness, the model-review tooling and rinse's canon mode; its migration to mend.pipeline is APPS-566 (brief: ~/ai/campaigns/canon-mymend-2026-10-18/handoffs/EPST.md). Do not move that pin to a revision from 0.2.0 on until the migration lands: its proof-drive lane and its read-through imports would fail.

Development

GitLab is the canonical development forge. Any GitHub repository is a read-only source mirror; report issues and propose changes at https://gitlab.com/doering-ai/libs/mend.

Versioning

This project follows semantic versioning. A 1.0.0 release requires the maintainer's stability promise in RELEASE.md.