RagPrepKit

Document preprocessing toolkit for RAG (retrieval-augmented generation) and LLM pipelines. ragprepkit handles the unglamorous but high-leverage work that sits between "raw document" and "ready to embed": cleaning noisy text, splitting it into retrieval-sized chunks, pulling out lightweight structural metadata, and estimating token counts before you ever call a model.

It has zero required dependencies. Everything works out of the box on Python 3.9+; installing the optional tiktoken extra upgrades token counting from a heuristic estimate to an exact count.

Why this exists

Most RAG bugs aren't in the retrieval or the prompt — they're in the preprocessing step nobody looked at closely: boilerplate leaking into embeddings, sentences getting cut in half at chunk boundaries, or token budgets blowing up because nobody counted before sending. ragprepkit is a small, inspectable, dependency-light layer for that step, not a framework that owns your whole pipeline.

Features

• Text cleaning — unicode normalization, control-character stripping, configurable boilerplate-line removal (cookie notices, copyright footers, newsletter prompts), and whitespace normalization.
• Three chunking strategies — fixed-size character windows, sentence-safe chunking that never splits mid-sentence, and a recursive paragraph → sentence → fixed-size fallback for mixed long-form documents.
• Lightweight metadata extraction — word/sentence counts, estimated reading time, markdown headings, extracted URLs, and a coarse script-based language guess.
• Token counting — exact counts via an optional tiktoken integration, with a dependency-free heuristic fallback so the library always works.
• Zero required dependencies — the core package has no runtime dependencies; tiktoken is opt-in via an extra.
• Fully typed — type hints throughout, with a py.typed marker for PEP 561 compatibility with type checkers.

Project structure

ragprepkit/
├── src/
│   ├── ragprepkit/
│   │    ├── __init__.py     # public API exports, __version__
│   │    ├── cleaning.py     # clean_text, normalize_whitespace, strip_boilerplate
│   │    ├── chunking.py     # Chunk, fixed_size_chunks, sentence_chunks, recursive_chunks
│   │    ├── metadata.py     # DocumentMetadata, extract_metadata
│   │    ├── tokens.py       # count_tokens, estimate_cost
│   │    └── py.typed 
│   └──tests                 # pytest test suite (mirrors src/ragprepkit modules)                 
├── pyproject.toml           # build config, metadata, dependencies
├── LICENSE
└── README.md

Installation

  pip install ragprepkit

from ragprepkit import clean_text

Requires Python 3.9 or later.

Quick start

from ragprepkit import clean_text, recursive_chunks, extract_metadata, count_tokens

raw = """
# Quarterly Report

Cookie Policy applies to this site.

Revenue grew 12% year over year, driven primarily by expansion
in the enterprise segment. Customer churn declined for the third
consecutive quarter.

## Outlook

Management expects continued growth into next year, though macro
headwinds remain a risk. See https://example.com/full-report for
the full filing.

All rights reserved 2026.
"""

text = clean_text(raw)
chunks = recursive_chunks(text, chunk_size=300, overlap=30)
meta = extract_metadata(text)

print(f"{len(chunks)} chunks, {meta.word_count} words, {meta.estimated_reading_time_minutes} min read")
for chunk in chunks:
  print(f"[chunk {chunk.index}] ({count_tokens(chunk.text)} tokens) {chunk.text[:60]}...")

Usage examples

1. Cleaning scraped or exported documents

from ragprepkit import clean_text

raw_html_text = """
Sign up for our newsletter
This is the main content of the article.

It contains    extra      whitespace.
"""

cleaned = clean_text(
    raw_html_text,
    remove_boilerplate=True,
    extra_boilerplate_patterns=[r"^Sign up for our newsletter.*$"],
)

print(cleaned)

clean_text runs unicode normalization, control-character stripping, boilerplate-line removal, and whitespace normalization in one pass. Each step is also exposed individually (normalize_whitespace, strip_boilerplate) if you want to compose your own pipeline.

2. Choosing a chunking strategy

from ragprepkit import fixed_size_chunks, sentence_chunks, recursive_chunks

text = """
# Quarterly Report

Revenue grew 12% year over year, driven by strong enterprise demand.

The company expanded into three new markets during the quarter.

Management expects continued growth into next year, though macroeconomic
conditions remain uncertain.
"""

# Uniform windows — fastest, ignores structure. Good for short, dense text.
fixed = fixed_size_chunks(text, chunk_size=80, overlap=10)

# Never splits a sentence — good when exact quotes/citations matter.
by_sentence = sentence_chunks(text, max_chars=80, overlap_sentences=1)

# Paragraph-first, falling back to sentence- then fixed-size splitting.
# The best default for mixed long-form documents (reports, articles, docs).
by_structure = recursive_chunks(text, chunk_size=80, overlap=10)

print(f"Fixed: {len(fixed)} chunks")
print(f"Sentence: {len(by_sentence)} chunks")
print(f"Recursive: {len(by_structure)} chunks")

Each strategy returns a list of Chunk objects carrying text, start_char, end_char, index, and an open metadata dict you can populate with your own fields (source document ID, page number, etc.) before handing chunks to your embedding step.

for chunk in by_structure:
    chunk.metadata["source"] = "quarterly_report.pdf"
    chunk.metadata["page"] = estimate_page_number(chunk.start_char)

3. Extracting metadata for filtering and routing

from ragprepkit import extract_metadata

document_text = """
# Quarterly Report

Revenue grew 12% year over year.

For more information visit:
https://example.com

## Outlook

Management expects continued growth into next year.
"""

meta = extract_metadata(document_text)

print(meta.word_count)
print(meta.headings)  # ['Quarterly Report', 'Outlook']
print(meta.urls)      # ['https://example.com']
print(meta.estimated_reading_time_minutes)
print(meta.likely_language)

Useful for building filters ("only index docs over 200 words"), surfacing a table of contents from headings, or routing documents to a language-specific pipeline before deeper processing.

4. Token counting and cost estimation

from ragprepkit import count_tokens, estimate_cost

text = """
Revenue grew 12% year over year, driven by strong enterprise demand.
Management expects continued growth into next year.
"""

# Count tokens in the text.
tokens = count_tokens(text)
print(tokens)

# Estimate processing cost.
cost = estimate_cost(text, price_per_1k_tokens=0.003)
print(cost)

5. End-to-end pipeline sketch

from ragprepkit import clean_text, recursive_chunks, count_tokens


def prepare_for_embedding(raw_text: str, source_id: str, max_tokens_per_chunk: int = 300):
    cleaned = clean_text(raw_text)
    chunks = recursive_chunks(cleaned, chunk_size=1200, overlap=120)

    prepared = []
    for chunk in chunks:
        if count_tokens(chunk.text) > max_tokens_per_chunk:
            # Fall back to a tighter split for oversized chunks.
            chunk_pieces = recursive_chunks(chunk.text, chunk_size=600, overlap=60)
        else:
            chunk_pieces = [chunk]

        for piece in chunk_pieces:
            prepared.append({
                "text": piece.text,
                "source_id": source_id,
                "tokens": count_tokens(piece.text),
            })

    return prepared


text = """
# Quarterly Report

Revenue grew 12% year over year, driven by strong enterprise demand.
Management expects continued growth into next year.
"""

prepared = prepare_for_embedding(text, source_id="quarterly_report.pdf")
print(prepared[0])

API reference

Development

 git clone https://github.com/mindropsrepo/ragprepkit.git
 cd ragprepkit
 pip install -e ".[dev,tokenizers]"
 pytest
 ruff check .

Design notes / limitations

• extract_metadata's language detection is a coarse latin vs. non-latin script heuristic, not real language identification — pair with a proper library (e.g. langdetect, fasttext) if you need accurate ISO codes.
• extract_metadata's sentence count is a simple punctuation-based heuristic and can overcount on things like decimal numbers or abbreviations — treat it as an estimate, not an exact linguistic count.
• count_tokens falls back to a len(text) // 4 heuristic without tiktoken installed. This is a commonly cited rough average for English text, not an exact count for every tokenizer or language — install the tokenizers extra for precision.
• estimate_cost intentionally takes price as an argument rather than embedding a pricing table, since provider pricing changes frequently.

About Mindrops

Mindrops is a software company that develops AI, cloud, web, mobile, and enterprise software solutions.

ragprepkit is an open-source library developed and maintained by Mindrops for document preprocessing in Retrieval-Augmented Generation (RAG) and large language model (LLM) workflows. The project focuses on practical utilities for text cleaning, chunking, metadata extraction, and token counting.

For more information, visit Mindrops.

Links

• 🌐 Mindrops Open Source: mindrops.com/open-source
• 📦 RagPrepKit: Project page
• 🐙 GitHub: Repository
• 📧 Contact: info@mindrops.com

License

Ragprepkit is released under the MIT License.

MIT — see LICENSE.