ragchunker

A modular Python package for Retrieval-Augmented Generation (RAG) preprocessing. It supports:

• File Ingestion: Load documents from various formats (TXT, PDF, DOCX, PPTX, images with OCR, audio with Whisper, CSV/Excel/Parquet, HTML, JSON, ZIP/TAR).
• Chunking: Split documents into chunks using fixed-length, semantic, or recursive strategies.
• Provenance: Track metadata, checksums, and token counts for chunks.
• Embeddings: Generate embeddings using Sentence-Transformers or OpenAI.
• Storage: Save chunks and embeddings to JSONL, Parquet, SQLite, NumPy, or FAISS; optional integration with Pinecone or ChromaDB.

Installation

Install the core package:

pip install ragchunker



For Testing:

from ragchunker.rag_pipeline import run_rag_pipeline
from ragchunker.storage import store_from_result

VECTOR_DB = "chroma"  # "faiss" or "chroma"


## For Open Source Embeddings
result = run_rag_pipeline(
    data_dir="data",                # Folder with your PDFs, docs, or txt files
    output_dir="output",            # Where JSONL, Chunks and embeddings will be saved
    chunk_strategy="semantic",      # or "fixed", "recursive"
    chunk_size=800,
    overlap=100,
    embed_model="all-MiniLM-L6-v2", # or any other
    embed_provider="sentence-transformers",
    openai_api_key=None                    # or "sk-your-openai-key" if using OpenAI
)

print("\n✅ Pipeline executed successfully!")


# For Open AI Embeddings
result = run_rag_pipeline(
    data_dir="data",                # Folder with your PDFs, docs, or txt files
    output_dir="output",            # Where JSONL, Chunks and embeddings will be saved
    chunk_strategy="semantic",      # or "fixed", "recursive"
    chunk_size=800,
    overlap=100,
    embed_model="text-embedding-3-small", # or any other
    embed_provider="openai",
    openai_api_key=""                   # or "sk-your-openai-key" if using OpenAI
)

print("\n✅ Pipeline executed successfully!")


print(f"\nStoring embeddings + chunks to: {VECTOR_DB}")
store_info = store_from_result(result, VECTOR_DB)
print("Store info:", store_info)
print("Done.")