ninjaeval (Python)

Show NinjaEval what happens inside your AI bot (every tool call and LLM call), so it can grade them and show them side by side with production in its debugger. No dependencies.

import ninjaeval
ninjaeval.instrument()                     # records OpenAI, Anthropic and LiteLLM chat calls automatically

@ninjaeval.tool                            # records arguments, result or error, and duration
def get_order(order_id: str): ...

@app.post("/chat")
def chat(req):
    with ninjaeval.capture(req.headers) as run:     # active only for NinjaEval test requests
        reply = agent.run(req.message)
    return run.attach({"reply": reply})             # adds the "ninjaeval" block for tests, nothing otherwise

• Tests only. NinjaEval marks its requests with X-NinjaEval-Test. Without that header, every hook does nothing and your response is unchanged.
• Never breaks your app. Errors inside the SDK are swallowed, and your tools' own errors are recorded and re-raised as usual.
• Redaction. capture(headers, redact=lambda text: mask_emails(text)) is applied to everything recorded.
• Other clients: call ninjaeval.record_turn(model=..., messages_in=..., output=..., tool_calls=[...], prompt_tokens=..., completion_tokens=..., latency_ms=...) yourself.
• OpenTelemetry (optional): with pip install "ninjaeval[otel]", each tool and LLM call is also a span, a child of the test's traceparent.
• Works with sync and async code (FastAPI, Flask, Django, Starlette). Contract: the ninjaeval block v1.