Skip to main content

rag-hygiene

Find and repair broken RAG chunks before you embed them.

Most RAG failures are decided before retrieval ever runs. A chunk that opens in the middle of a sentence embeds two unrelated claims. A table split from its header embeds a grid of numbers with no column names. A PDF that broke a word across lines embeds docu and ment as one token. None of this is visible until answers start being subtly wrong months later.

rag-hygiene sits between your splitter and your vector database. It diagnoses nine classes of chunk damage and repairs seven of them automatically — no LLM calls, no embeddings, no eval set, no network, no dependencies.

pip install rag-hygiene

Quick start

rag-hygiene scan chunks.json          # what is wrong?
rag-hygiene fix  chunks.json          # repair it → chunks.clean.json
rag-hygiene fix  chunks.json --dry-run -v   # preview the diff first
2 chunks scanned

mid-sentence-start  Chunk resumes a cut sentence
  [high] c2  opens mid-sentence ('40% year over year, beat'); its embedding mixes two unrelated claims ·fixable

2 finding(s)  1 high  1 low  · 1 auto-fixable
run `rag-hygiene fix` to repair, or add --dry-run to preview

The repair that matters

before                                    after
────────────────────────────────────      ────────────────────────────────────
[chunk 7]  …因此公司营收同比增长            [chunk 7]  …因此公司营收同比增长 40%,
[chunk 8]  40%,超出市场预期。                        超出市场预期。
                                          ✓ merged  ✓ heading "Q3 财报" injected

Every repair is lossless or a rejoin of text that was already adjacent. Nothing is summarised, paraphrased, or invented — which is what lets it run unattended in a build step.

Rules

Rule Severity Auto-fix What it catches
empty-chunk high ✅ removed Whitespace-only chunks; wasted embedding slots
mid-sentence-start high ✅ merged Chunk resumes a sentence the previous one cut
tiny-chunk medium ✅ merged Under 20 chars of prose — no meaning to embed
unclosed-fence medium ✅ closed Code fence opens, never closes — rest of prompt becomes code
orphan-table medium ✅ header restored Markdown table rows whose header landed in another chunk
near-duplicate medium — ≥72% word-sequence overlap; dupes split ranking mass
oversized-chunk low — Over 4000 chars — retrieval dilutes
pdf-noise low ✅ normalised Hyphenated line breaks, column padding, page gaps
missing-heading low ✅ injected A heading exists in the document but the chunk can't see it

near-duplicate and oversized-chunk are deliberately not auto-repaired: dropping a duplicate or truncating text changes what the retriever can find, and that is an editorial decision, not a mechanical one. The tool reports them and stays out of the way.

Python API

from rag_hygiene import check, repair

chunks = [
    {"id": "a", "text": "Revenue grew by",        "source": "q3.md", "order": 0},
    {"id": "b", "text": "40% year over year.",    "source": "q3.md", "order": 1},
    {"id": "c", "text": "   ",                    "source": "q3.md", "order": 2},
]

report = check(chunks)
report.count("high")          # 2
report.by_rule()              # {'empty-chunk': 1, 'mid-sentence-start': 1, ...}
report.fails("medium")        # True

clean, report = repair(chunks)
clean[0].text                 # 'Revenue grew by 40% year over year.'
clean[0].metadata['merged_ids']   # ['b']  ← provenance is never lost

Accepts Chunk objects, dicts (text / page_content / content), or bare strings. Works with LangChain and LlamaIndex document exports unchanged.

CI gate

The exit code follows the linter convention, so it drops into any pipeline:

- run: pip install rag-hygiene
- run: rag-hygiene scan chunks.json --fail-on high
exit meaning
0 clean, or the --fail-on gate was not tripped
1 a finding at or above --fail-on remains
2 bad input (missing file, unsupported format, unknown rule)
3 internal error

Formats

Reads .json, .jsonl, and .txt / .md. A JSON payload may be a bare list or an object with a chunks / documents / data / items key. Output preserves the input format.

Design notes

Three decisions shape the behaviour:

It joins only definite continuations. A chunk opening with 40% …, , and then, or a lowercase word is provably a continuation, so it gets merged. A chunk opening with Revenue grew might have lost its predecessor — that is reported at high severity but not auto-merged, because guessing would rewrite evidence. rag-hygiene scan --fail-on high still catches it; you decide.

Repairs are idempotent. Running fix twice produces the same bytes as running it once. That is asserted in the test suite (test_repair_converges), because a formatter that keeps drifting is worse than no formatter.

missing-heading converges. A document that carries no heading anywhere gets one info-level finding, not one per chunk — there is nothing to repair, and flooding the report would make the gate useless. A document with a heading but unlabelled chunks gets per-chunk findings that fix actually resolves.

Comparison

chunklint (May 2026) performs overlapping static checks and depends on pydantic, rich, typer, rapidfuzz and pyyaml. rag-hygiene differs in three ways: it repairs rather than only reporting, it has zero dependencies, and it ships CJK sentence segmentation (Chinese/Japanese/Korean text has no spaces after 。, so the usual whitespace heuristics silently do nothing on it).

License

MIT

Metadata

Release files for rag-hygiene 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for rag-hygiene 0.1.0
File Size Uploaded Provenance Actions
rag_hygiene-0.1.0.tar.gz 28.5 kB Without provenance Details for rag_hygiene-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for rag-hygiene 0.1.0
File Interpreter ABI Platform Actions
rag_hygiene-0.1.0-py3-none-any.whl Python 3 none any Details for rag_hygiene-0.1.0-py3-none-any.whl

Total release size: 53.3 kB

Release files / rag_hygiene-0.1.0.tar.gz

Download URL rag_hygiene-0.1.0.tar.gz
Size 28.5 kB
Tags Source
SHA-256 checksum
How to use checksums
ba334b47ebdf1caa665a4bd5e45f2e85fac0d83e649056aba855c3f46353d720
BLAKE2b-256 checksum
How to use checksums
c08cc9687566ebb3396e90308166b8fc7c3f5e3e83d66803e9794a35c6fc5a43
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Release files / rag_hygiene-0.1.0-py3-none-any.whl

Download URL rag_hygiene-0.1.0-py3-none-any.whl
Size 24.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
75709866f35d000be1c6540e7c65040fa6f043f8600a9d50b63129724893fd79
BLAKE2b-256 checksum
How to use checksums
236f40ead5aad0bb60abaa897186d6e3e25069777ffdcfcae923dfd5aa26cef2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Security for rag-hygiene 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page