---

The Problem

Building RAG pipelines today means writing bespoke scripts that:

• Re-embed everything on every run — slow and expensive
• Have no tests — you only find out your retrieval is broken in production
• Are impossible to diff or review in PRs

ragbt brings the dbt workflow — declarative configs, incremental builds, tests, and lineage — to embedding pipelines.

---

✨ Features

Feature	What it means
🔄 Incremental runs	Content-hash based; only changed docs get re-embedded
🧪 Retrieval tests	Assert that queries return the right docs before shipping
📊 Lineage & manifest	Lineage from source → chunks → vectors
🎯 Multi-target	Isolated dev / prod environments
🔌 Pluggable	Swappable chunkers, embedders, and vector stores
⚡ Fast	HNSW index on pgvector for sub-second search

---

🚀 Quick Start

1. Install

pip install ragbt

Requires Python 3.10+ and a Postgres database with the pgvector extension.

2. Set environment variables

export RAGBT_DATABASE_URL="postgresql+asyncpg://user:pass@host:5432/db"
export VOYAGE_API_KEY="your-api-key"
# or OPENAI_API_KEY="your-key"

3. Scaffold a project

ragbt init my_rag_project
cd my_rag_project

4. Add your docs

Drop Markdown or text files into data/product_docs/.

5. Run incrementally

ragbt run

Only changed documents are re-chunked and re-embedded. Subsequent runs are near-instant.

6. Test retrieval quality

ragbt test

Fails CI if your expected docs don't surface for key queries.

7. Generate lineage

ragbt docs generate
ragbt ls

---

📁 Project Structure

my_rag_project/
├── ragbt_project.yml          # Project metadata
├── sources.yml                # Where raw content lives
├── models/
│   └── product_docs.yml       # Chunking + embedding strategy
├── tests/
│   └── retrieval_tests.yml    # Quality assertions
├── data/
│   └── product_docs/          # Your raw docs
└── manifest.json              # Generated lineage graph

---

🛠️ Configuration

sources.yml — define content sources

sources:
  - name: product_docs
    type: directory
    path: ./data/product_docs
    file_types: [md, txt]

models/product_docs.yml — declare chunking & embedding

models:
  - name: product_docs_chunks
    source: product_docs
    description: "Customer-facing product documentation"

    chunking:
      strategy: recursive
      chunk_size: 512
      chunk_overlap: 50

    embedding:
      provider: voyage
      model: voyage-3
      dimensions: 1024
      batch_size: 100

    materialization: incremental
    index:
      backend: pgvector
      table: product_docs_vectors
      distance: cosine

tests/retrieval_tests.yml — gate your pipeline

tests:
  - name: refund_policy_findable
    model: product_docs_chunks
    query: "what is your refund policy"
    expect:
      top_k: 5
      must_contain_doc: "refund-policy.md"
      min_score: 0.75

  - name: no_orphaned_chunks
    model: product_docs_chunks
    type: integrity

---

🖥️ CLI Reference

Command	Description
ragbt init [name]	Scaffold a new project
ragbt run	Incremental run (default)
ragbt run --full-refresh	Force re-process everything
ragbt run --target prod	Run against the prod index
ragbt test	Run retrieval assertions
ragbt docs generate	Build manifest.json + lineage graph
ragbt ls	List models and last-run status

---

✅ Supported in v0.1

Component	Supported
Sources	directory (local files)
Chunkers	recursive, markdown
Embedders	voyage, openai
Vector Stores	pgvector (Postgres + pgvector)
Distance Metrics	cosine, euclidean, dot
Tests	retrieval, integrity, freshness

---

🗺️ Roadmap

Feature	Status	Target
semantic chunker	Planned	v0.2
cohere embedder	Planned	v0.2
pinecone / weaviate vector stores	Planned	v0.2
ragbt plan — dry-run preview	Planned	v0.2
ragbt diff — changed files report	Planned	v0.2
DAG with ref() between models	Planned	v0.3
Retrieval metrics (MRR, HitRate)	Planned	v0.3
Automatic RAGBT PR checks	Planned	v0.3
GitHub Actions CI	✓ Available	v0.1
RAGBT Cloud dashboard	Future	v1.0

---

🏗️ Architecture

sources.yml ──▶ config loader (Pydantic)
                      │
                      ▼
              ┌───────────────┐
              │  state store   │  (Postgres: doc hashes, run history)
              └───────┬───────┘
                      │ diff: what changed?
                      ▼
              ┌───────────────┐
              │  chunker       │  recursive | markdown
              └───────┬───────┘
                      ▼
              ┌───────────────┐
              │  embedder      │  voyage | openai
              └───────┬───────┘
                      ▼
              ┌───────────────┐
              │  vector store  │  pgvector
              └───────┬───────┘
                      ▼
              ┌───────────────┐
              │  test runner   │  golden query → expected chunk assertions
              └───────┬───────┘
                      ▼
              ┌───────────────┐
              │  manifest/docs │  lineage graph, freshness, run history
              └───────────────┘

Core Concepts vs dbt

dbt concept	ragbt equivalent
sources.yml	sources.yml — where raw content lives
models/*.sql	models/*.yml — declared chunking + embedding strategy
Incremental models	Content-hash based incremental runs
dbt test	ragbt test — retrieval assertions
dbt docs generate	ragbt docs generate — lineage graph
target: dev / target: prod	--target flag — separate indexes
manifest.json	manifest.json — lineage metadata + run state
Jinja macros	Python plugin functions for custom chunkers/embedders

---

🧪 Development

git clone https://github.com/shivamim/ragbt.git
cd ragbt
pip install -e ".[dev]"
pytest

---

🐳 Docker

docker-compose up -d postgres
ragbt init demo && cd demo
ragbt run
ragbt test

---

📜 License

Apache License 2.0 — see LICENSE.

---