ragkit

A modular, production-oriented Retrieval-Augmented Generation (RAG) library for Python.

ragkit ingests documents, indexes them for both dense (vector) and keyword (BM25) search, retrieves and reranks the most relevant chunks for a question, and generates a grounded, citation-backed answer - all through a small public API:

from ragkit import RAG

rag = RAG()
rag.ingest("documents/example.pdf")

response = rag.query("What is the refund policy?")
print(response.answer)
for source in response.sources:
    print(source.document, source.page, source.retrieval_score)

It is built as a standalone, installable library (not an application) so it can be pip installed into any Python project.

---

Table of contents

1. Architecture
2. Installation
3. Environment configuration
4. Vector storage
5. Basic usage
6. Document ingestion
7. Querying
8. Sources & citations
9. Configuration reference
10. Running ragkit with a local Ollama LLM
11. Evaluation
12. Advanced usage
13. Project structure
14. Development & testing
15. Packaging / PyPI
16. Design decisions & constraints

---

Architecture

Each responsibility is its own module behind a small interface, so any one stage can be swapped without touching the others:

Document
   |
   v
Loader (ingestion/loaders.py)          .pdf / .txt / .md -> Document
   |
   v
Cleaner (ingestion/cleaner.py)         whitespace/control-char normalization
   |
   v
Chunker (ingestion/chunker.py)         RecursiveCharacterChunker -> list[Chunk]
   |
   v
EmbeddingProvider (embeddings/)        SentenceTransformerEmbeddingProvider -> vectors
   |
   v
VectorStore (vectorstore/)             QdrantVectorStore
   |
   v
   +----------------------+
   |                      |
   v                      v
VectorRetriever      BM25Retriever (keyword/)
   |                      |
   +----------+-----------+
              v
       HybridRetriever (fusion: RRF or weighted)
              |
              v
       QueryRewriter (optional, LLM-based)
              |
              v
       Reranker (optional, cross-encoder)
              |
              v
       ContextBuilder (dedup, limit, cite)
              |
              v
       LLMProvider (generation/llm.py)   OpenAIProvider (OpenAI-compatible)
              |
              v
          RAGResponse (answer + sources + metadata)

pipeline/rag.py's RAG class is the only module that knows about every other module - it wires concrete implementations together (constructed from Settings by default, or injected directly for tests/customization) and exposes ingest() / query() as the public surface.

Baseline vs. advanced is a configuration difference, not two implementations. Query rewriting, hybrid retrieval, and reranking are each gated by a feature flag (ENABLE_QUERY_REWRITING, ENABLE_HYBRID_RETRIEVAL, ENABLE_RERANKING). There is one pipeline; the evaluation harness (evaluation/evaluate.py) runs it under different flag combinations - baseline, hybrid, hybrid_rerank, final - and records results for comparison across them. See Evaluation.

Installation

Requires Python >= 3.10.

git clone <this-repo>
cd rag
python -m venv .venv
# Windows: .venv\Scripts\activate      macOS/Linux: source .venv/bin/activate

pip install -e ".[dev]"

The default EmbeddingProvider and Reranker run locally via sentence-transformers (downloads a small model to the local HF cache on first use - no API key needed for embeddings). The vector store (Qdrant) also runs embedded, in-process - nothing separate to start (see Vector storage). By default the LLMProvider talks to an OpenAI-compatible Chat Completions API (OpenAI itself, or any self-hosted server - vLLM, LM Studio, ... - that speaks the same API) and needs API_KEY. To run generation locally with zero API cost via Ollama instead (pip install -e ".[dev,ollama]"), see Running ragkit with a local Ollama LLM.

Environment configuration

Copy the example file and fill in real values:

cp .env.example .env

No API keys, model names, ports, or other configuration are hardcoded anywhere in the source code. Everything is read once, centrally, in src/ragkit/config/settings.py via pydantic-settings, and every other module receives a Settings object rather than calling os.getenv itself. See Configuration reference for every variable.

Vector storage

No separate service required by default. ragkit uses Qdrant's embedded local mode: the vector database runs in-process and persists to disk at VECTOR_DB_PATH (default: ./.ragkit_data/qdrant). pip install ragkit-rag, configure .env, and rag.ingest(...) works immediately - nothing to start beforehand. Collections are created automatically on first ingest() call, sized to match the active embedding model's dimension.

Embedded mode is a single-process store, not built for concurrent multi-process access or large- scale production workloads. For those, run Qdrant as a real server instead (self-hosted or managed) and point ragkit at it by setting VECTOR_DB_HOST (and optionally VECTOR_DB_PORT) in .env - this switches QdrantVectorStore from local mode to server mode, with no other code changes:

VECTOR_DB_HOST=localhost
VECTOR_DB_PORT=6333

Verify a server is reachable:

curl http://localhost:6333/collections
# or open http://localhost:6333/dashboard in a browser

Basic usage

python examples/basic_usage.py

from pathlib import Path
from ragkit import RAG

rag = RAG()
result = rag.ingest("documents/example.pdf")
print(f"Indexed {result.chunk_count} chunks from {result.document_name}")

response = rag.query("What is the refund policy?")
print(response.answer)

Document ingestion

rag.ingest(path) runs: load -> clean -> chunk -> embed -> upsert into Qdrant -> index into BM25.

Supported formats out of the box: .pdf, .txt, .md (registered in ingestion/loaders.py; adding a format means adding one loader function, nothing else changes).

Every Chunk carries full provenance: document_id, document_name, source, chunk_id, chunk_index, and page_number (populated for PDFs, None for plain text/markdown).

Chunking is structure-aware and recursive (ingestion/chunker.py): it prefers to split on paragraph boundaries, falling back to sentence and then word boundaries only for oversized pieces, and carries CHUNK_OVERLAP characters of trailing context from one chunk into the next. Configurable via CHUNK_SIZE / CHUNK_OVERLAP.

Re-ingesting the same file (by path) replaces its previous chunks rather than duplicating them - document_id is derived deterministically from the file's absolute path.

Querying

rag.query(question, top_k=None) runs: query rewriting (optional) -> retrieval (dense or hybrid, per config) -> reranking (optional) -> context construction -> grounded generation.

The system prompt (generation/prompts.py) instructs the model to answer only from retrieved context, cite sources with [n] markers, explicitly say when the context is insufficient rather than guess, and distinguish stated facts from inference.

Sources & citations

RAGResponse is a structured object, not a bare string:

response.answer     # str
response.sources    # list[Source]: document, page, chunk_id, retrieval_score, text
response.metadata    # dict: latency per stage, rewritten query, active feature flags

ContextBuilder (generation/context_builder.py) deduplicates retrieved chunks, orders them by score, stops once MAX_CONTEXT_CHARS is reached, and assigns each included chunk a citation number ([1], [2], ...) matching response.sources, so every claim the model cites can be traced back to the exact chunk that supports it.

Configuration reference

All variables, with defaults, live in .env.example. Summary:

Variable	Default	Purpose
LLM_PROVIDER	openai	openai (any OpenAI-compatible endpoint) or ollama (local, see next section).
API_KEY	(none)	LLM API key. Required only when LLM_PROVIDER=openai, and only for query()/generation, not ingest().
MODEL	gpt-4o-mini	Chat model name, OpenAI-compatible. Used when LLM_PROVIDER=openai.
OPENAI_BASE_URL	(none)	Point at a self-hosted OpenAI-compatible endpoint instead of api.openai.com.
OLLAMA_BASE_URL	http://localhost:11434	Ollama server address. Used when LLM_PROVIDER=ollama.
OLLAMA_MODEL	llama3.2:3b	Local model name (must already be ollama pulled). Used when LLM_PROVIDER=ollama.
EMBEDDING_MODEL	sentence-transformers/all-MiniLM-L6-v2	Local embedding model.
RERANKER_MODEL	cross-encoder/ms-marco-MiniLM-L-6-v2	Local cross-encoder reranker model.
VECTOR_DB_PATH	./.ragkit_data/qdrant	On-disk path for Qdrant's embedded local mode (default, no server needed).
VECTOR_DB_HOST / VECTOR_DB_PORT	(none) / 6333	Set VECTOR_DB_HOST to use a real Qdrant server instead of local mode.
VECTOR_DB_COLLECTION	documents	Qdrant collection name.
CHUNK_SIZE / CHUNK_OVERLAP	500 / 100	Chunking parameters (characters).
TOP_K	5	Chunks included in the final context/answer.
RETRIEVAL_CANDIDATE_K	20	Candidates pulled before fusion/reranking narrows to TOP_K.
ENABLE_QUERY_REWRITING	false	Turn on LLM-based query rewriting.
ENABLE_HYBRID_RETRIEVAL	true	Combine vector + BM25 retrieval.
ENABLE_RERANKING	true	Cross-encoder rerank of candidates.
HYBRID_FUSION_STRATEGY	rrf	rrf or weighted - see retrieval/hybrid.py.
HYBRID_ALPHA	0.5	Vector-vs-keyword weight, weighted strategy only.
MAX_CONTEXT_CHARS	6000	Hard cap on context sent to the LLM.
LOG_LEVEL	INFO	Python logging level for the ragkit logger namespace.

Settings can also be overridden programmatically (bypassing .env entirely), which is how evaluation/evaluate.py runs the same pipeline under different presets:

from ragkit import RAG, Settings
rag = RAG(settings=Settings(enable_hybrid_retrieval=False, enable_reranking=False))

Running ragkit with a local Ollama LLM

LLM_PROVIDER=ollama runs the entire pipeline - including evaluation - with zero paid API calls: generation happens against a small model running locally via Ollama, on the same machine, fully offline after the model is pulled once. Retrieval already runs locally (Qdrant + local embeddings + local reranker); this is what makes generation local too.

1. Install Ollama

• Windows: winget install --id Ollama.Ollama -e, or download the installer from ollama.com/download.
• macOS / Linux: see ollama.com/download.

The installer sets up a background server listening on http://localhost:11434 and normally starts it automatically. Verify it's running:

curl http://localhost:11434/api/version
# {"version":"0.32.15"}  (or similar)

If that fails, start it manually with ollama serve (or launch the Ollama app).

2. Pick and pull a model sized for your hardware

This project defaults to llama3.2:3b (~2GB download), chosen specifically for a CPU-only machine with 16GB RAM - it's small enough to leave headroom for the local embedding model (~90MB), the local reranker (~90MB), Qdrant, and everything else running on the machine, while still following instructions well enough to produce grounded, citation-following answers. Do not default to a 7B+ model on hardware like this - it will be slow and can crowd out RAM needed by the rest of the pipeline.

ollama pull llama3.2:3b

Model	Approx. download	When to use it
llama3.2:1b	~1.3GB	Tightest memory budget, or noticeably faster answers; weaker instruction-following.
llama3.2:3b (default)	~2.0GB	Recommended default for 16GB RAM / CPU-only.
qwen2.5:3b	~1.9GB	Comparable alternative to llama3.2:3b, worth trying if you want a second opinion.

Do not pull a 7B/8B/14B+ model on this kind of hardware unless you know you have the RAM and patience for it - this project intentionally does not default to one.

3. Configure ragkit to use it

In .env:

LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=llama3.2:3b

No API_KEY needed for this path.

4. Run ragkit

Nothing else changes - RAG() reads LLM_PROVIDER from .env and picks OllamaProvider automatically:

python examples/basic_usage.py

If Ollama isn't running or the model hasn't been pulled, you get an actionable ConfigurationError (not a generic connection error) telling you exactly what to run:

from ragkit import RAG
from ragkit.generation.llm import OllamaProvider

OllamaProvider(model="llama3.2:3b", base_url="http://localhost:11434").check_availability()
# raises ConfigurationError with a concrete fix if Ollama or the model isn't ready

evaluation/evaluate.py runs this same check automatically before doing any work.

5. Run the evaluation harness locally

pip install -e ".[dev,ollama]"

python evaluation/evaluate.py --all

This runs every preset against the fixed corpus/question set and records raw results (answers, retrieved contexts, per-question latency) - see Evaluation. Quality scoring (faithfulness, relevancy, precision/recall) is not currently wired up; it will be reintroduced via LangSmith.

Resource considerations

• CPU inference is slower than GPU or a hosted API - expect single answers to take several seconds to tens of seconds depending on context length and machine speed, and a full evaluation run (one generation call per question, per preset) to take meaningfully longer than the equivalent OpenAI run. This is expected, not a bug.
• Don't run multiple large local models at once, and don't add a second Ollama model into the mix unless you've confirmed you have the RAM for both models plus everything else.
• Qdrant runs embedded (see Vector storage) and Ollama runs natively on the host - neither needs a separate service or virtualization overhead, keeping the whole local setup lightweight.

Evaluation

Evaluation is reproducible and dataset-driven, not a one-off manual check. By default this uses whatever LLM_PROVIDER is set to in .env - see the previous section to run it entirely locally via Ollama with no API key, or configure LLM_PROVIDER=openai to use a hosted model instead:

pip install -e ".[dev]"          # add ",ollama" too if using LLM_PROVIDER=ollama

python evaluation/evaluate.py --preset baseline
python evaluation/evaluate.py --preset hybrid
python evaluation/evaluate.py --preset hybrid_rerank
python evaluation/evaluate.py --preset final
# or: python evaluation/evaluate.py --all

Each run: (re-)ingests the fixed corpus in evaluation/datasets/corpus/ (a synthetic "Acme Gadgets" customer-policy knowledge base - refunds, shipping, warranty, privacy, subscriptions), answers every question in evaluation/datasets/eval_dataset.json, and writes:

• evaluation/results/<preset>_<timestamp>.json - full detail: every question, answer, retrieved contexts, ground truth, and per-stage latency.
• evaluation/results/summary.json - aggregate latency appended across every run, for comparing presets over time.

Quality scoring (faithfulness, relevancy, precision/recall) is not currently wired up in this harness - it will be reintroduced via LangSmith. Until then, evaluation/offline_analysis.py provides a zero-LLM-cost proxy: it exercises the real retrieval pipeline for all four presets and scores what was retrieved against a hand-authored answer key (see the script's docstring).

eval_dataset.json deliberately spans six question categories (simple factual, multi-sentence, requiring multiple chunks of context, confusingly similar policies, unanswerable from the corpus, and exact numeric/date facts) so results reflect genuine retrieval difficulty rather than trivially-answerable questions.

No results are manufactured or hardcoded, and the dataset is not modified to inflate metrics - this repository does not commit a results file with claimed numbers; run the evaluation yourself to get honest numbers for your machine, models, and API access, and compare summary.json across presets to see what each retrieval upgrade (hybrid retrieval, reranking, query rewriting) actually changed for this corpus.

Advanced usage

python examples/advanced_usage.py

Covers: programmatic Settings overrides for A/B-ing retrieval configurations, and dropping in a fully custom Reranker (or any other interface: EmbeddingProvider, VectorStore, Retriever, QueryRewriter, LLMProvider) via constructor injection on RAG(...) without modifying ragkit itself:

from ragkit import RAG
from ragkit.retrieval.reranker import Reranker

class MyReranker(Reranker):
    def rerank(self, query, results):
        ...  # your logic

rag = RAG(reranker=MyReranker())

Project structure

rag/
├── src/ragkit/
│   ├── config/          # Settings (pydantic-settings) - the only place reading env vars
│   ├── ingestion/        # loaders, cleaner, chunker
│   ├── embeddings/       # EmbeddingProvider interface + local (sentence-transformers) impl
│   ├── vectorstore/      # VectorStore interface + Qdrant impl
│   ├── retrieval/        # vector / keyword / hybrid retrievers, query rewriter, reranker
│   ├── generation/       # LLMProvider interface + OpenAI/Ollama impls, prompts, context builder
│   ├── pipeline/         # RAG - the public orchestrator
│   ├── models/           # Document, Chunk, RAGResponse, Source
│   ├── exceptions.py      # RAGKitError hierarchy
│   ├── logging_config.py
│   └── utils/            # Timer
├── tests/
│   ├── unit/              # hermetic, mocked - no network/API required
│   └── integration/       # full pipeline wiring (fakes) + optional live Qdrant/LLM test
├── evaluation/
│   ├── datasets/          # eval_dataset.json + corpus/*.txt
│   ├── evaluate.py        # preset-driven evaluation harness
│   └── results/           # generated JSON/CSV run output (gitignored)
├── examples/
│   ├── basic_usage.py
│   └── advanced_usage.py
├── .env.example
├── pyproject.toml
└── README.md

Development & testing

pip install -e ".[dev]"

# Unit tests: fast, no external services
pytest tests/unit -q

# Integration tests (fakes-based tier runs by default; live tiers are opt-in)
pytest tests/integration -q
RAGKIT_RUN_LIVE_INTEGRATION_TESTS=1 pytest tests/integration -q   # needs Qdrant + real API_KEY
RAGKIT_RUN_LIVE_OLLAMA_TESTS=1 pytest tests/integration -q        # needs Qdrant + Ollama + pulled model

# Everything with coverage
pytest --cov=ragkit --cov-report=term-missing

Unit tests cover configuration validation, document loading, cleaning, chunking (size limits, overlap, page boundaries, provenance), embeddings (via a fake model), the vector store (via a fake Qdrant client), retrieval (vector, BM25, hybrid fusion - both RRF and weighted), query rewriting (including graceful fallback on LLM failure), reranking, context construction (dedup/limit/ citations), response model serialization, LLM provider selection (OllamaProvider vs. OpenAIProvider, based on LLM_PROVIDER), and OllamaProvider itself (via a fake ollama.Client

• unreachable server, model not pulled, model ready, generation success/failure). No unit test requires network access, an API key, or a running Ollama server.

Packaging / PyPI

The package uses a modern src/ layout and pyproject.toml (PEP 621 metadata, setuptools.build_meta backend):

pip install build twine
python -m build                 # produces dist/*.whl and dist/*.tar.gz
python -m twine check dist/*     # validate metadata before publishing
python -m twine upload dist/*    # publish to PyPI (needs PyPI credentials)

Once published:

pip install ragkit-rag

from ragkit import RAG

The PyPI distribution is named ragkit-rag (the name ragkit was already taken by an unrelated project), but the importable package stays ragkit - pip install ragkit-rag and from ragkit import RAG are both correct, together.

Only the public surface (RAG, RAGResponse, Source, IngestResult, Settings, get_settings, and the RAGKitError hierarchy) is exported from the top-level ragkit package; internal modules remain importable for advanced customization but are not part of the promised stable API.

Design decisions & constraints

• No hardcoded secrets, models, or endpoints. Every configurable value flows through ragkit.config.settings.Settings, loaded once from .env / environment variables.
• Every external dependency sits behind an interface: EmbeddingProvider, VectorStore, Retriever, Reranker, QueryRewriter, LLMProvider. Swapping an implementation never requires touching the pipeline or public API - LLMProvider ships two implementations (OpenAIProvider, OllamaProvider), selected purely via LLM_PROVIDER in .env.
• Vector database runs embedded by default, no separate service needed - pip install ragkit-rag works standalone. A real Qdrant server (self-hosted or managed) is opt-in via VECTOR_DB_HOST, for production workloads that need concurrent access or scale past a single process.
• Query rewriting is not assumed to help - it ships disabled by default and is measured, not assumed, in evaluation.
• Errors are typed (RAGKitError subclasses per stage) and carry actionable messages; library users are never handed a raw, unexplained stack trace for common failure modes (missing API key, unreachable Qdrant, empty/unsupported document, embedding/LLM failure).
• No evaluation data is fabricated. The evaluation harness produces honest, reproducible numbers from a fixed dataset; comparing presets shows what each retrieval upgrade actually changed, for better or worse.