ragjudge

Typed, async evals for RAG pipelines — the Python package. TypeScript port at packages/js.

Install

pip install ragjudge              # core (pydantic + httpx + rich)
pip install "ragjudge[openai]"    # + OpenAI judge
pip install "ragjudge[anthropic]" # + Anthropic judge

Requires Python 3.10+. Strict-typed end to end.

60-second quickstart

import asyncio
from ragjudge import (
    Sample, Suite, ContextRelevance, Faithfulness, AnswerRelevance,
)
from ragjudge.judges.openai import OpenAIJudge

samples = [
    Sample(
        question="What is the capital of France?",
        contexts=["Paris is the capital of France."],
        answer="Paris.",
    ),
]

suite = Suite(
    name="my-rag",
    metrics=[ContextRelevance(), Faithfulness(), AnswerRelevance()],
    judge=OpenAIJudge(model="gpt-4o-mini"),
)
report = asyncio.run(suite.run(samples))
print(f"{report.passed_count}/{report.total} passed")

CLI

ragjudge run samples.jsonl \
  --judge openai \
  --model gpt-4o-mini \
  --metrics context_relevance,faithfulness,answer_relevance \
  --report-out report.json \
  --fail-under 0.8

Each line of samples.jsonl is a Sample — see examples/samples.jsonl.

Metrics

Custom judge

Anything matching this shape is a judge — Judge is a runtime_checkable Protocol, no inheritance needed:

class MyJudge:
    async def judge(self, prompt: str, schema: dict) -> "JudgeResponse":
        ...

Custom metric

from ragjudge import Judge, Metric, Sample, Score

class MyMetric:
    name = "my_metric"
    threshold = 0.5

    async def score(self, sample: Sample, judge: Judge) -> Score:
        ...

Pass instances of it to Suite(metrics=[...]). That's it.

License

MIT.