ragtoolbox 0.1.0

Modular toolkit for building Retrieval-Augmented Generation (RAG) pipelines.

RAGToolBox

RAGToolBox is a modular, extensible Python package for building Retrieval-Augmented Generation (RAG) pipelines. It provides end-to-end components for:

• Loading content (local files, web pages, PubMed/PMC articles)
• Chunking text (paragraphs, sentences, sliding windows, section‑aware, hierarchical)
• Storing embeddings (SQLite and optional ChromaVectorStore backends)
• Indexing content with parallel embedding and metadata parsing
• Retrieval via FastEmbed or (optional) openai embeddings and similarity search
• Augmentation using customizable prompts and LLMs (local or Hugging Face)

---

Table of Contents

1. Installation
2. Quickstart
   • Loading Documents
   • Indexing Pipeline
   • Retrieval & Augmentation
3. CLI Usage
4. Configuration
5. Testing
6. Contributing
7. License

---

Installation

Install the latest release from PyPI:

pip install ragtoolbox

# with optional extras
pip install "ragtoolbox[transformers,chromadb,openai,ncbi]"

Or install from source:

git clone https://github.com/Nick-Nunley/RAGToolBox.git
cd RAGToolBox
pip install .

Prerequisites

• Python 3.10+
• Core dependencies (see also pyproject.toml):
  • numpy
  • requests
  • pyyaml
  • pdfplumber
  • html2text
  • pytest
  • beautifulsoup4
  • readability-lxml
  • nltk
  • fastembed
  • huggingface_hub
• Optional dependencies (for extended capabilities):
  • openai
  • chromadb
  • torch
  • transformers
  • biopython

---

Quickstart

Loading Documents

Use the built‑in factory to detect and process various formats:

from RAGToolBox.loader import BaseLoader

# Raw bytes fetched externally (e.g., via requests or open file)
raw_bytes = open("example.pdf", "rb").read()
LoaderClass = BaseLoader.detect_loader("example.pdf", raw_bytes)
loader = LoaderClass("example.pdf", "assets/kb")
loader.raw_content = raw_bytes
loader.process()

Indexing Pipeline

Chunk, embed, and store your KB in one pipeline:

from pathlib import Path
from RAGToolBox.chunk import SectionAwareChunker, SlidingWindowChunker, HierarchicalChunker
from RAGToolBox.index import Indexer, IndexerConfig

chunker = HierarchicalChunker([
    SectionAwareChunker(max_chunk_size=1000, overlap=200),
    SlidingWindowChunker(window_size=1000, overlap=200)
])
indexer = Indexer(
    chunker=chunker,
    embedding_model="fastembed",
    config = IndexerConfig(
        vector_store_backend="sqlite",
        vector_store_config={"db_path": Path("assets/kb/embeddings/embeddings.db")}
    )
)
indexer.index(chunked_results)

Retrieval & Augmentation

Retrieve relevant chunks and generate an LLM response:

from pathlib import Path
from RAGToolBox.retriever import Retriever, RetrievalConfig
from RAGToolBox.augmenter import Augmenter, GenerationConfig

# Example query
user_query = "What is RAG?"

# Initialize retriever
retriever = Retriever(
    embedding_model="fastembed",
    db_path=Path("assets/kb/embeddings/embeddings.db")
)

# Perform retrieval
contexts = retriever.retrieve(user_query, RetrievalConfig(top_k=5))

# Initialize augmenter
augmenter = Augmenter(
    model_name="google/gemma-2-2b-it",
    prompt_type="default",
    api_key="${HUGGINGFACE_API_KEY}",
    use_local=False
)

# Generate a response with sources
result = augmenter.generate_response_with_sources(
    query=user_query,
    retrieved_chunks=contexts,
    gen_config=GenerationConfig(temperature=0.25, max_new_tokens=200)
)
print(result["response"])

---

CLI Usage

Each module exposes a CLI entrypoint. Use -h for details:

# Load documents from URLs or files
python -m RAGToolBox.loader https://example.com/article.pdf --output-dir assets/kb

# Build index (chunk + embed)
python -m RAGToolBox.index --kb-dir assets/kb --embedding-model fastembed --vector-store sqlite

# Retrieve top-10 chunks\ npython -m RAGToolBox.retriever --query "Explain RAG" --embedding-model openai

# Augment with LLM
python -m RAGToolBox.augmenter "What is RAG?" --sources

# For a concise one-liner answer
python -m RAGToolBox.augmenter "What is RAG?" --prompt-type concise

# Interactively chat with your knowledgebase
python -m RAGToolBox.augmenter --chat

---

Testing

Run the full test suite:

bash tests/Run_tests.sh

Continuous integration is configured via GitHub Actions (see .github/workflows/ci.yml).

---

Note

RAGToolBox is in active development. The current release (v0.1.0) is an unstable preview. APIs and behavior may change, and some features may not yet be fully tested.

---

Contributing

1. Fork the repo
2. Create a feature branch: git checkout -b feature/your-feature
3. Commit changes: git commit -m "Add new feature"
4. Push to your fork and open a PR

Please follow the Contributing Guidelines.

---

License

This project is licensed under the MIT License. See LICENSE for details.