rag-scrubber

Clean your "dirty" PDF text before feeding it to LLMs.

rag-scrubber is a lightweight Python library designed for RAG (Retrieval-Augmented Generation) pipelines. It automatically detects and removes recurring headers, footers, and page numbers from extracted text, and fixes broken hyphenation.

😟 The Problem

When extracting text from PDFs (especially corporate reports or scanned docs), you often get artifacts that confuse LLMs:

"The Q3 revenue was CONFIDENTIAL REPORT 2024 higher than expected..."

These interruptions waste tokens and degrade model performance. rag-scrubber cleans this mess up.

✨ Features

• Auto-Header/Footer Removal: Uses positional statistics to detect lines that repeat across pages (e.g., "Page 1 of 50", "Confidential").
• Smart De-Hyphenation: Fixes words split across lines (e.g., com- + puter computer) without merging proper nouns.
• Zero-Shot Configuration: Works out of the box with sensible defaults, or tune the sensitivity yourself.
• Lightweight: No heavy ML dependencies. pure Python logic.

📦 Installation

pip install rag-scrubber

🚀 Quick Start

from rag_scrubber import RAGScrubber

# 1. Simulate dirty text (usually coming from PyPDF2 or PDFMiner)
pages = [
    "Q3 FINANCIAL REPORT - CONFIDENTIAL\nThe revenue increased by 20%.\nPage 1",
    "Q3 FINANCIAL REPORT - CONFIDENTIAL\nOperating costs were lowered.\nPage 2",
    "Q3 FINANCIAL REPORT - CONFIDENTIAL\nNew user acquisition is sta-\nble.\nPage 3"
]

# 2. Initialize the scrubber
# threshold=0.4 means if a line appears in >40% of pages, it's garbage.
scrubber = RAGScrubber(threshold=0.4)

# 3. Clean the text
clean_text = scrubber.clean(pages)

print(clean_text)

Output:

The revenue increased by 20%.

Operating costs were lowered.

New user acquisition is stable.

⚙️ Advanced Usage

You can adjust the sensitivity if it is deleting too much (or too little).

# Low threshold (0.1) = Very aggressive (removes anything that repeats even slightly)
# High threshold (0.9) = Very safe (only removes exact matches on almost every page)
scrubber = RAGScrubber(threshold=0.2) 

🤝 Contributing

1. Fork the repository.
2. Create a feature branch (git checkout -b feature/new-logic).
3. Commit your changes.
4. Push to the branch.
5. Open a Pull Request.

📄 License

MIT License. Free to use for commercial and personal projects.