RagPrep

Document preprocessing toolkit for RAG (retrieval-augmented generation) and LLM pipelines. ragprep handles the unglamorous but high-leverage work that sits between "raw document" and "ready to embed": cleaning noisy text, splitting it into retrieval-sized chunks, pulling out lightweight structural metadata, and estimating token counts before you ever call a model.

It has zero required dependencies. Everything works out of the box on Python 3.9+; installing the optional tiktoken extra upgrades token counting from a heuristic estimate to an exact count.

Why this exists

Most RAG bugs aren't in the retrieval or the prompt — they're in the preprocessing step nobody looked at closely: boilerplate leaking into embeddings, sentences getting cut in half at chunk boundaries, or token budgets blowing up because nobody counted before sending. ragprep is a small, inspectable, dependency-light layer for that step, not a framework that owns your whole pipeline.

Features

• Text cleaning — unicode normalization, control-character stripping, configurable boilerplate-line removal (cookie notices, copyright footers, newsletter prompts), and whitespace normalization.
• Three chunking strategies — fixed-size character windows, sentence-safe chunking that never splits mid-sentence, and a recursive paragraph → sentence → fixed-size fallback for mixed long-form documents.
• Lightweight metadata extraction — word/sentence counts, estimated reading time, markdown headings, extracted URLs, and a coarse script-based language guess.
• Token counting — exact counts via an optional tiktoken integration, with a dependency-free heuristic fallback so the library always works.
• Zero required dependencies — the core package has no runtime dependencies; tiktoken is opt-in via an extra.
• Fully typed — type hints throughout, with a py.typed marker for PEP 561 compatibility with type checkers.

Project structure

ragprep/
├── src/
│   └── ragprep/
│       ├── __init__.py      # public API exports, __version__
│       ├── cleaning.py      # clean_text, normalize_whitespace, strip_boilerplate
│       ├── chunking.py      # Chunk, fixed_size_chunks, sentence_chunks, recursive_chunks
│       ├── metadata.py      # DocumentMetadata, extract_metadata
│       ├── tokens.py        # count_tokens, estimate_cost
│       └── py.typed         # PEP 561 typed-package marker
├── tests/                   # pytest test suite (mirrors src/ragprep modules)
├── pyproject.toml           # build config, metadata, dependencies
├── LICENSE
└── README.md

Installation

pip install ragprepkit

Note: The package is published on PyPI as ragprepkit, while the Python import name is ragprep.

from ragprep import clean_text

Requires Python 3.9 or later.

Quick start

from ragprep import clean_text, recursive_chunks, extract_metadata, count_tokens

raw = """
# Quarterly Report

Cookie Policy applies to this site.

Revenue grew 12% year over year, driven primarily by expansion
in the enterprise segment. Customer churn declined for the third
consecutive quarter.

## Outlook

Management expects continued growth into next year, though macro
headwinds remain a risk. See https://example.com/full-report for
the full filing.

All rights reserved 2026.
"""

text = clean_text(raw)
chunks = recursive_chunks(text, chunk_size=300, overlap=30)
meta = extract_metadata(text)

print(f"{len(chunks)} chunks, {meta.word_count} words, {meta.estimated_reading_time_minutes} min read")
for chunk in chunks:
    print(f"[chunk {chunk.index}] ({count_tokens(chunk.text)} tokens) {chunk.text[:60]}...")

Usage examples

1. Cleaning scraped or exported documents

from ragprep import clean_text

raw_html_text = extracted_text_from_your_scraper  # already stripped of tags
cleaned = clean_text(
    raw_html_text,
    remove_boilerplate=True,
    extra_boilerplate_patterns=[r"^sign up for our newsletter.*$"],
)

clean_text runs unicode normalization, control-character stripping, boilerplate-line removal, and whitespace normalization in one pass. Each step is also exposed individually (normalize_whitespace, strip_boilerplate) if you want to compose your own pipeline.

2. Choosing a chunking strategy

from ragprep import fixed_size_chunks, sentence_chunks, recursive_chunks

# Uniform windows — fastest, ignores structure. Good for short, dense text.
fixed = fixed_size_chunks(text, chunk_size=800, overlap=80)

# Never splits a sentence — good when exact quotes/citations matter.
by_sentence = sentence_chunks(text, max_chars=800, overlap_sentences=1)

# Paragraph-first, falling back to sentence- then fixed-size splitting.
# The best default for mixed long-form documents (reports, articles, docs).
by_structure = recursive_chunks(text, chunk_size=800, overlap=80)

Each strategy returns a list of Chunk objects carrying text, start_char, end_char, index, and an open metadata dict you can populate with your own fields (source document ID, page number, etc.) before handing chunks to your embedding step.

for chunk in by_structure:
    chunk.metadata["source"] = "quarterly_report.pdf"
    chunk.metadata["page"] = estimate_page_number(chunk.start_char)

3. Extracting metadata for filtering and routing

from ragprep import extract_metadata

meta = extract_metadata(document_text)

print(meta.word_count)
print(meta.headings)                        # e.g. ["Quarterly Report", "Outlook"]
print(meta.urls)                             # links found in the body
print(meta.estimated_reading_time_minutes)
print(meta.likely_language)                  # coarse latin/non-latin script guess

Useful for building filters ("only index docs over 200 words"), surfacing a table of contents from headings, or routing documents to a language-specific pipeline before deeper processing.

4. Token counting and cost estimation

from ragprep import count_tokens, estimate_cost

tokens = count_tokens(chunk.text)  # exact if tiktoken is installed, else an estimate

# Pass in your current provider's price per 1k tokens — pricing changes
# often, so this is deliberately not hardcoded into the library.
cost = estimate_cost(chunk.text, price_per_1k_tokens=0.003)

5. End-to-end pipeline sketch

from ragprep import clean_text, recursive_chunks, count_tokens

def prepare_for_embedding(raw_text: str, source_id: str, max_tokens_per_chunk: int = 300):
    cleaned = clean_text(raw_text)
    chunks = recursive_chunks(cleaned, chunk_size=1200, overlap=120)

    prepared = []
    for chunk in chunks:
        if count_tokens(chunk.text) > max_tokens_per_chunk:
            # Fall back to a tighter split for oversized chunks.
            chunk_pieces = recursive_chunks(chunk.text, chunk_size=600, overlap=60)
        else:
            chunk_pieces = [chunk]

        for piece in chunk_pieces:
            prepared.append({
                "text": piece.text,
                "source_id": source_id,
                "tokens": count_tokens(piece.text),
            })
    return prepared

API reference

Development

git clone https://github.com/mindropsai/ragprep.git
cd ragprep
pip install -e ".[dev,tokenizers]"
pytest
ruff check .

Design notes / limitations

• extract_metadata's language detection is a coarse latin vs. non-latin script heuristic, not real language identification — pair with a proper library (e.g. langdetect, fasttext) if you need accurate ISO codes.
• extract_metadata's sentence count is a simple punctuation-based heuristic and can overcount on things like decimal numbers or abbreviations — treat it as an estimate, not an exact linguistic count.
• count_tokens falls back to a len(text) // 4 heuristic without tiktoken installed. This is a commonly cited rough average for English text, not an exact count for every tokenizer or language — install the tokenizers extra for precision.
• estimate_cost intentionally takes price as an argument rather than embedding a pricing table, since provider pricing changes frequently.

License

MIT — see LICENSE.

About Mindrops

Mindrops is a global technology company that helps organizations accelerate digital transformation through intelligent software solutions. With expertise spanning Artificial Intelligence, cloud technologies, custom software development, mobile applications, and digital platforms, Mindrops partners with businesses to build scalable, secure, and high-performance technology solutions.

Driven by its philosophy of "Thinking Together, Creating Together," Mindrops combines deep technical expertise with a collaborative, process-driven approach to solve complex business challenges. The company serves clients across North America, Europe, and Asia through its presence in India, the United Kingdom, Belgium, and the United States.

ragprep is one of the open-source initiatives developed by Mindrops to support modern AI development. It reflects our commitment to building practical, production-ready tools that simplify document preprocessing for Retrieval-Augmented Generation (RAG), semantic search, and enterprise LLM applications.

Learn more about Mindrops at https://www.mindrops.com.

Links

• 🌐 Homepage: https://www.mindrops.com
• 💻 Repository: https://github.com/mindropsai/ragprep