rag-hygiene
Find and repair broken RAG chunks before you embed them.
Most RAG failures are decided before retrieval ever runs. A chunk that opens in
the middle of a sentence embeds two unrelated claims. A table split from its
header embeds a grid of numbers with no column names. A PDF that broke a word
across lines embeds docu and ment as one token. None of this is visible until
answers start being subtly wrong months later.
rag-hygiene sits between your splitter and your vector database. It diagnoses
nine classes of chunk damage and repairs seven of them automatically — no
LLM calls, no embeddings, no eval set, no network, no dependencies.
pip install rag-hygiene
Quick start
rag-hygiene scan chunks.json # what is wrong?
rag-hygiene fix chunks.json # repair it → chunks.clean.json
rag-hygiene fix chunks.json --dry-run -v # preview the diff first
2 chunks scanned
mid-sentence-start Chunk resumes a cut sentence
[high] c2 opens mid-sentence ('40% year over year, beat'); its embedding mixes two unrelated claims ·fixable
2 finding(s) 1 high 1 low · 1 auto-fixable
run `rag-hygiene fix` to repair, or add --dry-run to preview
The repair that matters
before after
──────────────────────────────────── ────────────────────────────────────
[chunk 7] …因此公司营收同比增长 [chunk 7] …因此公司营收同比增长 40%,
[chunk 8] 40%,超出市场预期。 超出市场预期。
✓ merged ✓ heading "Q3 财报" injected
Every repair is lossless or a rejoin of text that was already adjacent. Nothing is summarised, paraphrased, or invented — which is what lets it run unattended in a build step.
Rules
| Rule | Severity | Auto-fix | What it catches |
|---|---|---|---|
empty-chunk |
high | ✅ removed | Whitespace-only chunks; wasted embedding slots |
mid-sentence-start |
high | ✅ merged | Chunk resumes a sentence the previous one cut |
tiny-chunk |
medium | ✅ merged | Under 20 chars of prose — no meaning to embed |
unclosed-fence |
medium | ✅ closed | Code fence opens, never closes — rest of prompt becomes code |
orphan-table |
medium | ✅ header restored | Markdown table rows whose header landed in another chunk |
near-duplicate |
medium | — | ≥72% word-sequence overlap; dupes split ranking mass |
oversized-chunk |
low | — | Over 4000 chars — retrieval dilutes |
pdf-noise |
low | ✅ normalised | Hyphenated line breaks, column padding, page gaps |
missing-heading |
low | ✅ injected | A heading exists in the document but the chunk can't see it |
near-duplicate and oversized-chunk are deliberately not auto-repaired:
dropping a duplicate or truncating text changes what the retriever can find, and
that is an editorial decision, not a mechanical one. The tool reports them and
stays out of the way.
Python API
from rag_hygiene import check, repair
chunks = [
{"id": "a", "text": "Revenue grew by", "source": "q3.md", "order": 0},
{"id": "b", "text": "40% year over year.", "source": "q3.md", "order": 1},
{"id": "c", "text": " ", "source": "q3.md", "order": 2},
]
report = check(chunks)
report.count("high") # 2
report.by_rule() # {'empty-chunk': 1, 'mid-sentence-start': 1, ...}
report.fails("medium") # True
clean, report = repair(chunks)
clean[0].text # 'Revenue grew by 40% year over year.'
clean[0].metadata['merged_ids'] # ['b'] ← provenance is never lost
Accepts Chunk objects, dicts (text / page_content / content), or bare
strings. Works with LangChain and LlamaIndex document exports unchanged.
CI gate
The exit code follows the linter convention, so it drops into any pipeline:
- run: pip install rag-hygiene
- run: rag-hygiene scan chunks.json --fail-on high
| exit | meaning |
|---|---|
| 0 | clean, or the --fail-on gate was not tripped |
| 1 | a finding at or above --fail-on remains |
| 2 | bad input (missing file, unsupported format, unknown rule) |
| 3 | internal error |
Formats
Reads .json, .jsonl, and .txt / .md. A JSON payload may be a bare list or
an object with a chunks / documents / data / items key. Output preserves
the input format.
Design notes
Three decisions shape the behaviour:
It joins only definite continuations. A chunk opening with 40% …, , and
then, or a lowercase word is provably a continuation, so it gets merged. A chunk
opening with Revenue grew might have lost its predecessor — that is reported
at high severity but not auto-merged, because guessing would rewrite evidence.
rag-hygiene scan --fail-on high still catches it; you decide.
Repairs are idempotent. Running fix twice produces the same bytes as running
it once. That is asserted in the test suite (test_repair_converges), because a
formatter that keeps drifting is worse than no formatter.
missing-heading converges. A document that carries no heading anywhere gets
one info-level finding, not one per chunk — there is nothing to repair, and
flooding the report would make the gate useless. A document with a heading but
unlabelled chunks gets per-chunk findings that fix actually resolves.
Comparison
chunklint (May 2026) performs overlapping static checks and depends on
pydantic, rich, typer, rapidfuzz and pyyaml. rag-hygiene differs in three ways:
it repairs rather than only reporting, it has zero dependencies, and it
ships CJK sentence segmentation (Chinese/Japanese/Korean text has no spaces after
。, so the usual whitespace heuristics silently do nothing on it).
License
MIT
Changelog
0.1.1
- Fix: Chinese chunks resuming a clause after a numeral were reported as
auto-fixable but never repaired.
is_definite_continuationnow recognises CJK measure words (3 个工作日,2024 年), which carry the same "this resumed a sentence" signal that capitalisation carries in Latin text. - Test: added
test_fixable_findings_are_actually_fixed, asserting the contract that every finding markedfixableis gone afterrepair().
0.1.0
- Initial release: 9 rules, 7 auto-repairing, zero dependencies, CJK segmentation.
Metadata
Release files for rag-hygiene 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| rag_hygiene-0.1.1.tar.gz | 29.7 kB | Without provenance | Details for rag_hygiene-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| rag_hygiene-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for rag_hygiene-0.1.1-py3-none-any.whl |
Total release size: 55.1 kB
Release files / rag_hygiene-0.1.1.tar.gz
| Download URL | rag_hygiene-0.1.1.tar.gz |
|---|---|
| Size | 29.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
249b30655794768c8affb4b74c7a18d692df3915912a4747f340d20b92801cce
|
|
BLAKE2b-256 checksum How to use checksums |
fbdf9be2130bf730bdc6673b29d4d5070a7b5d7a2bfaa7ef48de08a1a4a9a0a2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.10
|
Release files / rag_hygiene-0.1.1-py3-none-any.whl
| Download URL | rag_hygiene-0.1.1-py3-none-any.whl |
|---|---|
| Size | 25.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
63fb505ce7bd50acde6c39c18541faa9ba02774af42b8d7501d6b8bbada6d878
|
|
BLAKE2b-256 checksum How to use checksums |
5cd2079a65489c2e2b6aaef4a78888993f6f0b82f3e6477069f7653309531c5f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.10
|
Security for rag-hygiene 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.