rubra 0.1.0

Agentic evaluation framework. Every aspect, nothing missed.

Rubra is a trace-first agent evaluation framework. Decorate your agent — Rubra automatically captures every tool call, LLM call, token, and cost. Then evaluate with 36 metrics including 11 tool-orchestration metrics no other framework provides.

import rubra

@rubra.agent(task="Answer questions using web search")
def my_agent(question: str) -> str:
    context = search_web(question)
    return call_llm(context, question)

my_agent("What is the capital of France?")

report = rubra.evaluate(rubra.get_last_trace())
print(f"Rubra Score: {report.rubra_score:.3f}")   # 0.923
print(f"Passed:      {report.passed}/{report.total_metrics}")

Why Rubra

Installation

pip install rubra                    # core (4 deps, no LLM required)
pip install "rubra[judge]"           # + LLM-judge metrics via litellm
pip install "rubra[openai]"          # + OpenAI SDK interceptor
pip install "rubra[anthropic]"       # + Anthropic Claude interceptor
pip install "rubra[langgraph]"       # + LangGraph node tracing
pip install "rubra[langchain]"       # + LangChain callback handler
pip install "rubra[otel]"            # + OpenTelemetry export
pip install "rubra[all]"             # everything

Quickstart

1. Basic agent (any framework)

import rubra

@rubra.tool
def search_web(query: str) -> str:
    return my_search_api(query)

@rubra.agent(
    task="Answer capital city questions",
    expected_tool_calls=["search_web"],   # optional: enables F1 metrics
)
def capital_agent(question: str) -> str:
    context = search_web(question)
    return my_llm(context, question)

capital_agent("What is the capital of Japan?")

trace = rubra.get_last_trace()
report = rubra.evaluate(trace, metrics="all")
print(report.summary())

2. With OpenAI — zero-change LLM tracing

import openai
import rubra

client = rubra.patch(openai.OpenAI())   # one line — that's it

@rubra.agent(task="Capital cities")
def agent(q: str) -> str:
    response = client.chat.completions.create(   # automatically traced
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": q}],
    )
    return response.choices[0].message.content

3. With Anthropic Claude

import anthropic
import rubra

client = rubra.patch_anthropic(anthropic.Anthropic())

@rubra.agent(task="Summarise documents")
def agent(text: str) -> str:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[{"role": "user", "content": text}],
    )
    return response.content[0].text

4. In pytest — evaluate your agent in CI

# test_agent.py  (no conftest.py needed — plugin registers automatically)

def test_capital_agent_quality(rubra_trace):
    result = capital_agent("What is the capital of Japan?")
    assert result == "Tokyo"

    report = rubra_trace.evaluate(metrics="execution")
    assert report.get("task_completion_rate").passed
    assert report.rubra_score >= 0.70

# One-liner shorthand:
def test_passes_score_threshold(rubra_trace):
    capital_agent("What is the capital of Germany?")
    rubra_trace.assert_score(min_rubra_score=0.70, min_pass_rate=0.80)

5. LangGraph

from langgraph.graph import StateGraph
from rubra.integrations.langgraph import patch
import rubra

graph = StateGraph(MyState)
graph.add_node("search", search_node)
graph.add_node("answer", answer_node)
app = patch(graph).compile()   # wraps every node as a tool span

@rubra.agent(task="Multi-hop question answering")
def run(question: str) -> str:
    return app.invoke({"question": question})["answer"]

6. LangChain

from rubra.integrations.langchain import RubraCallbackHandler
import rubra

handler = RubraCallbackHandler()

@rubra.agent(task="Chain execution")
def run(question: str) -> str:
    return my_chain.invoke({"question": question}, config={"callbacks": [handler]})

Available Metrics

Execution (13) — deterministic, no LLM needed

Tool Orchestration (11) — USP, unique to Rubra

Safety (3)

prompt_injection_resistance · scope_creep_score · pii_propagation_count

Quality (4)

answer_relevance_proxy · output_coherence_score · format_compliance_score · response_groundedness

Goal / LLM-judge (5) — requires rubra[judge]

goal_completion · answer_correctness · reasoning_quality · task_understanding · hallucination_score

The judge model is configurable and works with any litellm-supported model — including free local models via Ollama, so you can exercise these metrics with zero API cost:

report = rubra.evaluate(trace, metrics="all", judge_model="ollama/llama3.2")

Composite scores (automatic)

• rubra_score — weighted average across all scored metrics
• tool_intelligence_score — average of tool-category metrics
• agentic_efficiency_score — completion × average efficiency

REST API + Dashboard

See rubra-server for the self-hosted FastAPI backend and live dashboard.

git clone https://github.com/pm1715/rubra-server
cd rubra-server
docker compose up
# Dashboard → http://localhost:8000
# API docs  → http://localhost:8000/docs

CLI

rubra traces              # list recent traces
rubra eval                # evaluate latest trace
rubra eval <TRACE_ID>     # evaluate specific trace
rubra report -o out.html  # generate HTML report

Architecture

Rubra uses Python contextvars.ContextVar for async-safe, thread-safe trace propagation — no globals, no thread-locals, no locks. Each @rubra.agent call creates an isolated Trace with its own ContextVar token, making concurrent agents safe by design.

@rubra.agent ──► Trace (ContextVar)
    @rubra.tool ──► TOOL_CALL + TOOL_RESPONSE spans
    rubra.patch ──► LLM_CALL spans (auto)
evaluate(trace) ──► EvalReport (36 metrics + 3 composite scores)

License

Apache 2.0 — see LICENSE.

Contributing

See CONTRIBUTING.md.