corpus-quality 1.0.0

Document quality profiler for ML pipelines. Score, deduplicate, and validate your corpus before embedding. Zero mandatory dependencies.

doc-quality

Document quality profiler for ML pipelines. Score, deduplicate, and validate your corpus before embedding. Zero mandatory dependencies.

Install

pip install corpus-quality

The problem

Most RAG failures start at ingestion -- when nobody checked whether the documents were actually clean before embedding them. Teams spend days tuning chunk sizes and prompt templates when the real problem is that 20% of their corpus is failed PDF extractions, near-duplicates, or wall-of-text that will split badly at any chunk boundary.

doc-quality runs before any of that.

Quick start

from doc_quality import CorpusProfiler

profiler = CorpusProfiler(corpus_name="my_rag_corpus")
report = profiler.profile_directory("./documents/")
print(report.summary())

=== DOC-QUALITY -- CORPUS QUALITY REPORT ===
  Documents:      47
  Pass:           34
  Warn:           9
  Fail:           4
  Pass rate:      72%
  Avg score:      74.2/100
  Duplicates:     6 pairs
  Recommended chunking: recursive

Single document

from doc_quality import DocumentProfiler

profiler = DocumentProfiler()
profile = profiler.profile(text, name="annual_report.pdf")

print(profile.quality_score)      # 83.4
print(profile.quality_level)      # QualityLevel.PASS
print(profile.boilerplate_ratio)  # 0.18
print(profile.chunk_risk_score)   # 0.22
print(profile.issues)

Near-duplicate detection (no embeddings needed)

from doc_quality import find_duplicates

pairs = find_duplicates({"doc_a.txt": text_a, "doc_b.txt": text_b}, threshold=0.85)
for pair in pairs:
    print(pair)
# DuplicatePair('policy_v1.txt' <-> 'policy_v1_copy.txt', EXACT)

Data card for model cards and papers

print(report.data_card.to_markdown())

CLI

doc-quality profile report.txt
doc-quality corpus ./documents/ --name "my_corpus"
doc-quality corpus ./documents/ --data-card
doc-quality deduplicate ./documents/
doc-quality corpus ./documents/ --json

Quality dimensions scored

Dimension	Weight	Detects
Encoding	25%	Replacement chars, mojibake
Text density	20%	Whitespace-heavy extractions
Uniqueness	20%	Repeated lines, template noise
Sentences	15%	List dumps, failed extractions
Boilerplate	10%	Headers, footers, disclaimers
Table integrity	5%	Malformed table structure
Chunk boundary risk	5%	Wall-of-text, no split points

Pipeline position

doc-quality (profile) -> chunk-bench (benchmark) -> rag-eval-kit (evaluate)

---

Linda Oraegbunam | LinkedIn | GitHub