rag-audit 0.1.0

CLI + library to audit and benchmark RAG pipelines

rag-audit

CLI + library to audit and benchmark RAG pipelines. Detects hallucinations, measures retrieval quality, compares chunking strategies, and generates structured reports.

Documentation

Installation

pip install rag-audit

Or with uv:

uv add rag-audit

Quickstart

1. Create a pipeline config file (pipeline.json):

{
  "pipeline_id": "my-pipeline",
  "question": "What is the capital of France?",
  "answer": "Paris is the capital of France.",
  "contexts": [
    "Paris is the capital and largest city of France.",
    "France is a country in Western Europe."
  ],
  "relevant": [
    "Paris is the capital and largest city of France."
  ],
  "k": 2,
  "llm": {
    "provider": "openai",
    "model": "gpt-4o-mini"
  }
}

2. Run the audit:

export OPENAI_API_KEY=sk-...
rag-audit run pipeline.json -o result.json

3. Generate a report:

# Markdown (default)
rag-audit report result.json

# JSON
rag-audit report result.json --format json

Config reference

Field	Type	Description
pipeline_id	string	Identifier for the pipeline being audited
question	string	The question posed to the RAG pipeline
answer	string	The answer generated by the pipeline
contexts	string[]	Retrieved chunks, in rank order
relevant	string[]	Ground-truth relevant chunks (for retrieval metrics)
k	int	Number of top chunks to evaluate (default: 5)
llm.provider	"openai" | "anthropic"	LLM provider for the faithfulness judge
llm.model	string	Model name (e.g. "gpt-4o-mini", "claude-3-5-haiku-20241022")

Metrics

Retrieval

Metric	Description
Precision@k	Fraction of the top-k retrieved chunks that are relevant
Recall@k	Fraction of all relevant chunks that appear in the top-k
MRR	Mean Reciprocal Rank — how high the first relevant chunk ranks

Faithfulness

Metric	Description
Score	0.0–1.0 — how well the answer is grounded in the retrieved contexts
Verdict	FAITHFUL if score ≥ threshold (default 0.5), otherwise HALLUCINATION

Python API

Audit a pipeline

from rag_audit.core.config import PipelineConfig, LLMConfig
from rag_audit.core.runner import AuditRunner
from rag_audit.report.renderer import ReportRenderer

config = PipelineConfig(
    pipeline_id="my-pipeline",
    question="What is the capital of France?",
    answer="Paris is the capital of France.",
    contexts=["Paris is the capital and largest city of France."],
    relevant=["Paris is the capital and largest city of France."],
    k=1,
    llm=LLMConfig(provider="openai", model="gpt-4o-mini"),
)

report = AuditRunner(config).run()
print(ReportRenderer().to_markdown(report))

Compare chunking strategies

from langchain_openai import OpenAIEmbeddings
from rag_audit.chunker import ChunkingEvaluator, FixedSizeChunker, RecursiveChunker, SemanticChunker

embeddings = OpenAIEmbeddings()
evaluator = ChunkingEvaluator(embeddings)

report = evaluator.evaluate(
    "Your long document text here...",
    {
        "fixed": FixedSizeChunker(chunk_size=500, overlap=50),
        "recursive": RecursiveChunker(chunk_size=500),
        "semantic": SemanticChunker(embeddings, similarity_threshold=0.8),
    },
)

print(f"Best strategy: {report.best_strategy}")
for s in report.strategies:
    print(f"  {s.strategy}: avg_cohesion={s.avg_cohesion:.3f}, chunks={s.chunk_count}")

Use a vectorstore adapter

from rag_audit.adapters import ChromaDBAdapter

adapter = ChromaDBAdapter("my-collection")
adapter.add(ids=["doc1"], texts=["Paris is in France."], embeddings=[[...]])
results = adapter.query(embedding=[...], k=1)

Roadmap

• CLI (rag-audit run, rag-audit report)
• Hallucination detection (LLM-as-judge)
• Retrieval metrics (Precision@k, Recall@k, MRR)
• Structured audit reports (JSON + Markdown)
• Chunking strategy benchmark (fixed-size vs recursive vs semantic)
• Vectorstore adapters (ChromaDB — Pinecone and Qdrant coming soon)
• Documentation (GitHub Pages)
• PyPI release

Development

# Install dependencies
uv sync --group dev

# Run tests
uv run pytest

# Lint + format
uv run ruff check src/ tests/
uv run ruff format src/ tests/

# Type check
uv run mypy src/rag_audit

# Build docs locally
uv sync --group docs
uv run mkdocs serve