ragnarok-ai 1.11.0

Local-first RAG evaluation framework for LLM applications

Evaluate, benchmark, and monitor your RAG pipelines — 100% locally, no API keys required.

Problem • Solution • Quick Start • Installation • Roadmap

The Problem

Building RAG systems is easy. Knowing if they actually work is hard.

A pipeline can have a good retriever but a bad generator, great faithfulness but poor relevance, or a beautiful score that quietly degrades in production. And when the answer matters — medical, legal, internal data — you often can't send your documents to an external API just to evaluate them.

ragnarok-ai is designed from the ground up for local, private, reproducible RAG evaluation:

• Runs 100% locally (Ollama, local models) — your data never leaves your machine
• Evaluates with checkpointing — crash mid-run, resume exactly where you left off
• Tells you why quality is low (root-cause diagnosis), not just that it is
• Lets you calibrate the judge against your own labels — trust the scores, or know you can't
• Gates your CI, monitors production, detects drift, A/B tests configurations
• Integrates with your existing stack (LangChain, LangGraph, LlamaIndex...)

The Solution

ragnarok-ai is a lightweight, local-first framework to evaluate RAG pipelines.

from ragnarok_ai import evaluate, generate_testset

# Generate test questions from your knowledge base
testset = await generate_testset(
    knowledge_base="./docs/",
    num_questions=50,
    types=["simple", "multi_hop", "adversarial"],
    llm="ollama/mistral",
    checkpoint=True,  # Resume if interrupted
)

# Evaluate your RAG pipeline
results = await evaluate(
    rag_pipeline=my_rag,
    testset=testset,
    metrics=["retrieval", "faithfulness", "relevance"],
    llm="ollama/mistral",
)

# Get actionable insights
results.summary()
# ┌─────────────────┬───────┬────────┐
# │ Metric          │ Score │ Status │
# ├─────────────────┼───────┼────────┤
# │ Retrieval P@10  │ 0.82  │ PASS   │
# │ Faithfulness    │ 0.74  │ WARN   │
# │ Relevance       │ 0.89  │ PASS   │
# │ Hallucination   │ 0.12  │ PASS   │
# └─────────────────┴───────┴────────┘

results.export("report.html")

v1.11.0 is now available! Developer Experience: evaluate your own pipeline from the CLI with live streaming results, and browse them in an interactive TUI. Install with pip install ragnarok-ai

Key Features

Design Principles

Excellent evaluation frameworks exist (RAGAS, DeepEval, TruLens...). ragnarok-ai doesn't try to have more metrics than them — it makes a different set of guarantees:

Performance

Benchmarked on Apple M2 16GB, Python 3.10:

Retrieval Metrics: ~24,000 queries/sec

LLM-as-Judge (Prometheus 2):

Retrieval is pure computation — instant. LLM-as-Judge is the bottleneck (~25s/eval), but runs 100% local.

Full benchmarks →

Quick Start

Try it now: Open in Google Colab

Prerequisites

• Python 3.10+
• Ollama running locally
• uv (recommended) or pip

Install

pip install ragnarok-ai

With optional dependencies:

pip install ragnarok-ai[ollama,qdrant]

Run your first evaluation

# CLI demo
ragnarok evaluate --demo

# With options
ragnarok evaluate --demo --output results.json --fail-under 0.7

# Or in Python
python examples/basic_evaluation.py

Installation

Using pip

pip install ragnarok-ai

Optional dependencies

# LLM providers
pip install ragnarok-ai[ollama]      # Ollama support
pip install ragnarok-ai[openai]      # OpenAI support
pip install ragnarok-ai[anthropic]   # Anthropic support

# Vector stores
pip install ragnarok-ai[qdrant]      # Qdrant support
pip install ragnarok-ai[chroma]      # ChromaDB support
pip install ragnarok-ai[faiss]       # FAISS support

# RAG frameworks
pip install ragnarok-ai[langchain]   # LangChain/LangGraph support
pip install ragnarok-ai[llamaindex]  # LlamaIndex support
pip install ragnarok-ai[dspy]        # DSPy support

# Observability
pip install ragnarok-ai[telemetry]   # OpenTelemetry tracing

# Everything
pip install ragnarok-ai[all]

Development

git clone https://github.com/2501Pr0ject/RAGnarok-AI.git
cd RAGnarok-AI
pip install -e ".[dev]"
pre-commit install

Use Cases

Continuous RAG Testing in CI/CD

# .github/workflows/rag-tests.yml
- uses: 2501Pr0ject/ragnarok-evaluate-action@v1
  with:
    config: ragnarok.yaml
    threshold: 0.8
    # fail-on-threshold: false (default - advisory only)
    # comment-on-pr: true (default - posts PR comment)

The action posts a PR comment distinguishing deterministic retrieval metrics from advisory LLM-as-Judge scores.

Compare Embedding Models

configs = [
    {"embedder": "nomic-embed-text", "chunk_size": 512},
    {"embedder": "mxbai-embed-large", "chunk_size": 256},
]

results = await benchmark(
    rag_factory=create_rag,
    configs=configs,
    testset=testset,
)
results.compare()  # Side-by-side comparison

Monitor Production Quality

# Track quality drift over time
metrics = await evaluate(rag, production_queries)
metrics.log_to("./metrics/")  # Time-series storage

Metrics

Retrieval Metrics

• Precision@K — Relevant docs in top K results
• Recall@K — Coverage of relevant docs
• MRR — Mean Reciprocal Rank
• NDCG — Normalized Discounted Cumulative Gain

Generation Metrics

• Faithfulness — Is the answer grounded in retrieved context?
• Relevance — Does the answer address the question?
• Hallucination — Does the answer contain fabricated info?
• Completeness — Are all aspects of the question covered?

LLM-as-Judge (v1.2+)

Use Prometheus 2 for comprehensive, local evaluation:

from ragnarok_ai import LLMJudge

# Initialize judge (uses Prometheus 2 by default)
judge = LLMJudge()

# Evaluate a single response
result = await judge.evaluate_all(
    context="Python was created by Guido van Rossum in 1991.",
    question="Who created Python?",
    answer="Guido van Rossum created Python.",
)

print(f"Overall: {result.overall_verdict} ({result.overall_score:.2f})")
# Overall: PASS (0.85)

print(f"Faithfulness: {result.faithfulness.verdict}")
print(f"Hallucination: {result.hallucination.verdict}")

Performance:

• ~20-30s per evaluation on Apple M2 16GB
• Prometheus 2 Q5_K_M: ~5GB RAM usage
• keep_alive enabled by default (prevents model unloading between requests)

Installation:

# Install Prometheus 2 (~5GB, runs on 16GB RAM)
ollama pull hf.co/RichardErkhov/prometheus-eval_-_prometheus-7b-v2.0-gguf:Q5_K_M

Medical Mode

Reduce false positives in healthcare RAG evaluation with automatic medical abbreviation normalization.

from ragnarok_ai import LLMJudge

# Enable medical mode
judge = LLMJudge(medical_mode=True)

# "CHF" and "congestive heart failure" are now treated as equivalent
result = await judge.evaluate_faithfulness(
    context="Patient diagnosed with CHF.",
    question="What condition does the patient have?",
    answer="Patient has congestive heart failure.",
)
# Without medical_mode: may flag as unfaithful (text mismatch)
# With medical_mode: correctly identifies as faithful

Features:

• 350+ medical abbreviations (CHF, MI, COPD, DVT...)
• Context-aware disambiguation (MS = multiple sclerosis vs mitral stenosis)
• Multiple formats: dotted (q.d.), slash (s/p), mixed-case (SpO2)
• False positive filtering (OR, US, IT stay unchanged)

Also works with FaithfulnessEvaluator(llm, medical_mode=True).

Contributed by @harish1120

System Metrics

• Latency — End-to-end response time
• Token usage — Cost tracking for LLM calls

Cost Tracking (v1.3+)

Track exactly what your evaluations cost:

results = await evaluate(rag, testset, track_cost=True)
print(results.cost)
# +--------------------+------------+----------+
# | Provider           |     Tokens |     Cost |
# +--------------------+------------+----------+
# | ollama (local)     |     45,230 |    $0.00 |
# | openai             |     12,500 |    $0.38 |
# +--------------------+------------+----------+

Local-first advantage: Ollama evaluations cost $0.00.

Jupyter Notebook (v1.3.1+)

Rich HTML display for evaluation results:

from ragnarok_ai.notebook import display, display_comparison

# Full dashboard with metrics, cost, latency
display(results)

# Compare multiple pipelines side-by-side
display_comparison([
    ("Baseline", baseline_results),
    ("Improved", improved_results),
])

Roadmap

Completed

Planned

Next — Evaluation Study (feature freeze)

• Public reproducible benchmark: human-labeled sample set, multiple local judges
• Judge reliability study using the calibration module (human agreement per judge model)
• Technical write-up of methodology and results

On-demand

• Discord & email alert adapters

v2.0.0 — Visualization

• Basic Web UI dashboard

Future (on-demand)

• VS Code extension
• Multi-modal evaluation
• Enterprise (SSO, RBAC, Audit)

Architecture

ragnarok-ai/
├── src/ragnarok_ai/
│   ├── core/           # Types, protocols, exceptions
│   ├── evaluators/     # Metric implementations
│   ├── generators/     # Test set generation
│   ├── adapters/       # LLM, vector store, framework adapters
│   ├── reporters/      # Output formatters (JSON, HTML, console)
│   └── cli/            # Command-line interface
├── tests/              # Test suite (pytest)
├── examples/           # Usage examples
├── benchmarks/         # Performance benchmarks
└── docs/               # Documentation

Development

# Setup
uv pip install -e ".[dev]"
pre-commit install

# Run checks
pytest                    # Tests
pytest --cov=ragnarok_ai  # With coverage
ruff check . --fix        # Lint
ruff format .             # Format
mypy src/                 # Type check

Advanced Usage

Importing Types

For advanced use cases (custom RAG implementations, type hints), import types directly from submodules:

# Core types
from ragnarok_ai.core.types import Document, Query, RAGResponse, TestSet

# Protocols (for implementing custom adapters)
from ragnarok_ai.core.protocols import RAGProtocol, LLMProtocol, VectorStoreProtocol

# Evaluators
from ragnarok_ai.evaluators import FaithfulnessEvaluator, RelevanceEvaluator

# Adapters
from ragnarok_ai.adapters.llm import OllamaLLM, OpenAILLM
from ragnarok_ai.adapters.vectorstore import ChromaVectorStore, QdrantVectorStore

Implementing a Custom RAG

from ragnarok_ai.core.protocols import RAGProtocol
from ragnarok_ai.core.types import RAGResponse, Document

class MyCustomRAG:
    """Custom RAG implementing the RAGProtocol."""

    async def query(self, question: str, k: int = 5) -> RAGResponse:
        # Your retrieval logic here
        docs = await self.retrieve(question, k)
        answer = await self.generate(question, docs)

        return RAGResponse(
            answer=answer,
            retrieved_docs=[
                Document(id=d.id, content=d.text, metadata=d.meta)
                for d in docs
            ],
        )

# Use with ragnarok-ai
from ragnarok_ai import evaluate

results = await evaluate(
    rag_pipeline=MyCustomRAG(),
    testset=testset,
    metrics=["retrieval", "faithfulness"],
)

Feedback

Your feedback helps improve RAGnarok-AI. Pick the right channel:

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for guidelines.

Priority areas for contributions:

• Framework adapters (Haystack, Semantic Kernel)
• Agent evaluation features
• Streaming evaluation support
• Multi-modal evaluation
• Documentation & examples

License

RAGnarok-AI is dual-licensed:

Why dual licensing?

• AGPL ensures improvements stay open-source
• Commercial license enables enterprise adoption without copyleft obligations

For commercial licensing inquiries: abdel.touati@gmail.com

Acknowledgments

Built out of frustration with complex evaluation setups. We wanted something that just works — locally, fast, and without API keys.

_{Built with ❤️ in Lyon, France}