Skip to main content

rag-hygiene

Find and repair broken RAG chunks before you embed them.

Most RAG failures are decided before retrieval ever runs. A chunk that opens in the middle of a sentence embeds two unrelated claims. A table split from its header embeds a grid of numbers with no column names. A PDF that broke a word across lines embeds docu and ment as one token. None of this is visible until answers start being subtly wrong months later.

rag-hygiene sits between your splitter and your vector database. It diagnoses nine classes of chunk damage and repairs seven of them automatically — no LLM calls, no embeddings, no eval set, no network, no dependencies.

pip install rag-hygiene

Quick start

rag-hygiene scan chunks.json          # what is wrong?
rag-hygiene fix  chunks.json          # repair it → chunks.clean.json
rag-hygiene fix  chunks.json --dry-run -v   # preview the diff first
2 chunks scanned

mid-sentence-start  Chunk resumes a cut sentence
  [high] c2  opens mid-sentence ('40% year over year, beat'); its embedding mixes two unrelated claims ·fixable

2 finding(s)  1 high  1 low  · 1 auto-fixable
run `rag-hygiene fix` to repair, or add --dry-run to preview

The repair that matters

before                                    after
────────────────────────────────────      ────────────────────────────────────
[chunk 7]  …因此公司营收同比增长            [chunk 7]  …因此公司营收同比增长 40%,
[chunk 8]  40%,超出市场预期。                        超出市场预期。
                                          ✓ merged  ✓ heading "Q3 财报" injected

Every repair is lossless or a rejoin of text that was already adjacent. Nothing is summarised, paraphrased, or invented — which is what lets it run unattended in a build step.

Rules

Rule Severity Auto-fix What it catches
empty-chunk high ✅ removed Whitespace-only chunks; wasted embedding slots
mid-sentence-start high ✅ merged Chunk resumes a sentence the previous one cut
tiny-chunk medium ✅ merged Under 20 chars of prose — no meaning to embed
unclosed-fence medium ✅ closed Code fence opens, never closes — rest of prompt becomes code
orphan-table medium ✅ header restored Markdown table rows whose header landed in another chunk
near-duplicate medium — ≥72% word-sequence overlap; dupes split ranking mass
oversized-chunk low — Over 4000 chars — retrieval dilutes
pdf-noise low ✅ normalised Hyphenated line breaks, column padding, page gaps
missing-heading low ✅ injected A heading exists in the document but the chunk can't see it

near-duplicate and oversized-chunk are deliberately not auto-repaired: dropping a duplicate or truncating text changes what the retriever can find, and that is an editorial decision, not a mechanical one. The tool reports them and stays out of the way.

Python API

from rag_hygiene import check, repair

chunks = [
    {"id": "a", "text": "Revenue grew by",        "source": "q3.md", "order": 0},
    {"id": "b", "text": "40% year over year.",    "source": "q3.md", "order": 1},
    {"id": "c", "text": "   ",                    "source": "q3.md", "order": 2},
]

report = check(chunks)
report.count("high")          # 2
report.by_rule()              # {'empty-chunk': 1, 'mid-sentence-start': 1, ...}
report.fails("medium")        # True

clean, report = repair(chunks)
clean[0].text                 # 'Revenue grew by 40% year over year.'
clean[0].metadata['merged_ids']   # ['b']  ← provenance is never lost

Accepts Chunk objects, dicts (text / page_content / content), or bare strings. Works with LangChain and LlamaIndex document exports unchanged.

CI gate

The exit code follows the linter convention, so it drops into any pipeline:

- run: pip install rag-hygiene
- run: rag-hygiene scan chunks.json --fail-on high
exit meaning
0 clean, or the --fail-on gate was not tripped
1 a finding at or above --fail-on remains
2 bad input (missing file, unsupported format, unknown rule)
3 internal error

Formats

Reads .json, .jsonl, and .txt / .md. A JSON payload may be a bare list or an object with a chunks / documents / data / items key. Output preserves the input format.

Design notes

Three decisions shape the behaviour:

It joins only definite continuations. A chunk opening with 40% …, , and then, or a lowercase word is provably a continuation, so it gets merged. A chunk opening with Revenue grew might have lost its predecessor — that is reported at high severity but not auto-merged, because guessing would rewrite evidence. rag-hygiene scan --fail-on high still catches it; you decide.

Repairs are idempotent. Running fix twice produces the same bytes as running it once. That is asserted in the test suite (test_repair_converges), because a formatter that keeps drifting is worse than no formatter.

missing-heading converges. A document that carries no heading anywhere gets one info-level finding, not one per chunk — there is nothing to repair, and flooding the report would make the gate useless. A document with a heading but unlabelled chunks gets per-chunk findings that fix actually resolves.

Comparison

chunklint (May 2026) performs overlapping static checks and depends on pydantic, rich, typer, rapidfuzz and pyyaml. rag-hygiene differs in three ways: it repairs rather than only reporting, it has zero dependencies, and it ships CJK sentence segmentation (Chinese/Japanese/Korean text has no spaces after 。, so the usual whitespace heuristics silently do nothing on it).

License

MIT

Changelog

0.1.1

  • Fix: Chinese chunks resuming a clause after a numeral were reported as auto-fixable but never repaired. is_definite_continuation now recognises CJK measure words (3 个工作日, 2024 年), which carry the same "this resumed a sentence" signal that capitalisation carries in Latin text.
  • Test: added test_fixable_findings_are_actually_fixed, asserting the contract that every finding marked fixable is gone after repair().

0.1.0

  • Initial release: 9 rules, 7 auto-repairing, zero dependencies, CJK segmentation.

Metadata

Release files for rag-hygiene 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for rag-hygiene 0.1.1
File Size Uploaded Provenance Actions
rag_hygiene-0.1.1.tar.gz 29.7 kB Without provenance Details for rag_hygiene-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for rag-hygiene 0.1.1
File Interpreter ABI Platform Actions
rag_hygiene-0.1.1-py3-none-any.whl Python 3 none any Details for rag_hygiene-0.1.1-py3-none-any.whl

Total release size: 55.1 kB

Release files / rag_hygiene-0.1.1.tar.gz

Download URL rag_hygiene-0.1.1.tar.gz
Size 29.7 kB
Tags Source
SHA-256 checksum
How to use checksums
249b30655794768c8affb4b74c7a18d692df3915912a4747f340d20b92801cce
BLAKE2b-256 checksum
How to use checksums
fbdf9be2130bf730bdc6673b29d4d5070a7b5d7a2bfaa7ef48de08a1a4a9a0a2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Release files / rag_hygiene-0.1.1-py3-none-any.whl

Download URL rag_hygiene-0.1.1-py3-none-any.whl
Size 25.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
63fb505ce7bd50acde6c39c18541faa9ba02774af42b8d7501d6b8bbada6d878
BLAKE2b-256 checksum
How to use checksums
5cd2079a65489c2e2b6aaef4a78888993f6f0b82f3e6477069f7653309531c5f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Security for rag-hygiene 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page