RAGforge

Local-first RAG pipeline — PDF & Markdown ingestion, Qdrant retrieval, bge reranking, and an answer-quality eval harness.
Pairs with turboquant-ml for quantized LLM serving.

Why RAGforge?

Most "RAG starter" repos are a 30-line glue between LangChain and OpenAI that nobody can reproduce because it hides retrieval quality, reranking, latency, and cost behind a single .invoke() call. RAGforge is the opposite: a small, readable, local-first pipeline that you can run end-to-end on your own laptop with open-source models, and that ships with an answer-quality eval harness so you can actually measure what changing a knob does.

Three opinions:

1. Local-first. Default everywhere is open-source: BAAI/bge-small-en-v1.5 for embeddings, BAAI/bge-reranker-base for reranking, Qdrant in embedded mode (no Docker required), and any HuggingFace causal LM for generation. No OpenAI key required to try the project.
2. Measurable. Every change should answer the question "did the answer get better?". RAGforge ships ragforge eval with built-in context_recall, answer_relevance, and faithfulness metrics — no RAGAS dependency required, but RAGAS-compatible.
3. Composable, not framework-y. Each stage (ingest, embed, retrieve, rerank, generate, evaluate) is one short module behind a small interface. Swap the encoder, swap the vector store, swap the LLM — no Runnable.invoke() magic to debug.

Features

Installation

The PyPI distribution is named ragforge-ml (the unsuffixed ragforge name was taken by an unrelated project). Python import and CLI are just ragforge / rf:

pip install ragforge-ml                       # core
pip install "ragforge-ml[serve]"              # + FastAPI
pip install "ragforge-ml[quantized]"          # + turboquant-ml NF4 path
pip install "ragforge-ml[all]"                # everything

60-second tour

from ragforge import Pipeline

rag = Pipeline.from_defaults(model_id="Qwen/Qwen2.5-3B-Instruct")
rag.ingest(["docs/policy.pdf", "notes/onboarding.md"])

answer = rag.ask("What is the maximum reimbursable amount for client lunches?")
print(answer.text)
for src in answer.sources:
    print(f"  {src.score:.3f}  {src.metadata['path']}#chunk{src.metadata['chunk']}")

CLI

rf ingest docs/ --collection company
rf ask "How do I rotate an API key?" --collection company --k 5
rf eval datasets/qa.jsonl --collection company --metrics context_recall,faithfulness
rf serve --collection company --host 0.0.0.0 --port 8080

Quantized LLM via TurboQuant

from ragforge import Pipeline
from ragforge.llm import QuantizedHFLLM

llm = QuantizedHFLLM("meta-llama/Llama-3.2-3B-Instruct", method="bnb-nf4")
rag = Pipeline.from_defaults(llm=llm)

Architecture

ragforge/
├── ingest/        # PDF + Markdown loaders, chunking
├── embed/         # sentence-transformers wrapper
├── vectorstore/   # Qdrant embedded + remote
├── rerank/        # bge-reranker-base
├── llm/           # HF causal LM + turboquant-ml integration
├── pipeline.py    # The end-to-end orchestrator
├── eval/          # context_recall, answer_relevance, faithfulness
├── serve/         # FastAPI app
└── cli.py         # ragforge / rf

Each module is short, readable, and replaceable through a small interface (Encoder, VectorStore, Reranker, LLM). The pipeline calls them in order — no DAG, no runnables, no callbacks.

Eval harness

The reason RAGforge exists. Most RAG projects ship without measuring whether their retrieval is any good. RAGforge ships three metrics in pure Python (no external API), all RAGAS-compatible:

rf eval datasets/qa.jsonl --collection company

                            +---------------+--------+
                            |  metric       |  mean  |
                            +---------------+--------+
                            | context_recall|  0.84  |
                            | answer_rel    |  0.78  |
                            | faithfulness  |  0.91  |
                            +---------------+--------+
                            n=120  ·  latency_p50=620ms  ·  latency_p95=1.4s

Roadmap

• PDF + Markdown ingestion
• Recursive char chunker with overlap
• BGE embeddings + BGE reranker
• Qdrant embedded + remote
• FastAPI serve
• CLI: ingest, ask, eval, serve
• Eval: context_recall, answer_relevance, faithfulness
• turboquant-ml integration for NF4 LLM serving
• Hybrid retrieval (BM25 + dense)
• Streaming generation in /ask
• Notion / Confluence loaders (community PRs welcome)
• SQL agent for structured-data questions

Contributing

See docs/CONTRIBUTING.md.

git clone https://github.com/Ademo93/ragforge
cd ragforge
pip install -e ".[dev,serve,eval]"
pytest

License

MIT.