dj-evals

Internal Django helper for running one eval function with multiple argument sets in parallel, so you can visually compare output, tool calls, usage, cost, and completion behavior as the runs happen.

Django setup

Add dj_evals to INSTALLED_APPS:

INSTALLED_APPS = [
    "dj_evals",
]

Create one view that connects the package to your project. This is the only view required by dj-evals:

from django.contrib.admin.views.decorators import staff_member_required
from dj_evals import handle_eval_request


@staff_member_required
async def eval_run(request):
    return await handle_eval_request(
        request,
        allowed_paths={"myapp.evals.echo_eval"},
        # evaluation_model="meta/muse-spark-1.2-contributor",  # Optional.
    )

Mount that view in your URL configuration:

from django.urls import path

urlpatterns = [
    path("evals/run/", eval_run, name="eval-run"),
]

Replace staff_member_required with your project's authorization check if staff-only access is not appropriate.

Link to eval suites

A project can have a small page that lists its eval suites. Use generate_eval_url() to build the URL behind each link:

from django.contrib.admin.views.decorators import staff_member_required
from django.shortcuts import render
from django.urls import reverse
from dj_evals import generate_eval_url


@staff_member_required
def eval_index(request):
    question = (
        "Give me 7 tips to improve my game in chess. "
        "My ELO is 500. I make lots of blunders."
    )
    url = reverse("eval-run") + generate_eval_url(
        "myapp.evals.echo_eval",
        {
            "model": "gpt-5.6-terra",
            "question": question,
        },
        {
            "model": "gpt-5.6-luna",
            "question": question,
        },
        # Optional. Requires the configured evaluation model's API key.
        expectations=[
            "Don't leave things hanging",
            "Castle early",
            "The answer mentions using a checklist",
            "The checklist is something simple yet effective",
        ],
    )
    return render(
        request,
        "eval_index.html",
        {"eval_suites": [("Chess tips for beginners", url)]},
    )

Render those URLs as normal links in eval_index.html:

{% for name, url in eval_suites %}
  <a href="{{ url }}">{{ name }}</a>
{% endfor %}

Clicking the link opens the comparison page. It starts one POST request per argument set and renders the streamed results side by side.

Expectations and Auto-rate use meta/muse-spark-1.2 through OpenRouter by default. OpenRouter model IDs (provider/model) use OPENROUTER_API_KEY; direct OpenAI model names use OPENAI_API_KEY.

Auto-rate scores completed runs and shows a detailed comparison. Re-judge checks the current expectations against those outputs without rerunning the evals.

Eval function contract

The eval function must be importable by dotted path and whitelisted in allowed_paths. It can be a sync or async function that returns one event, or an async generator that yields events.

def echo_eval(question="hello", model=""):
    return question


async def async_eval(question="hello", model=""):
    return question


async def streaming_eval(question="hello"):
    yield {"type": "response.output_text.delta", "delta": question}
    yield {"type": "response.output_text.done", "text": question}
    yield {"type": "response.completed", "response": {"cost": 0}}

Eval argument dictionaries are passed to the eval function as keyword arguments. OpenAI Responses API events have first-class rendering for output text, reasoning, and tool calls.

Helper utilities

You can yield EvalEvent for custom progress messages in the output panel:

from dj_evals import EvalEvent


async def streaming_eval():
    event: EvalEvent = {
        "type": "dj_evals.event",
        "message": "Fetched documents",
    }
    yield event

Developers

Run a local demo server to try the library in a browser:

uv run --with daphne daphne -p 8003 examples.local_server:application
# or, if you have just installed:
just run

Then open http://127.0.0.1:8003/ and choose an example. The OpenAI example uses OPENAI_API_KEY; expectations use OPENROUTER_API_KEY.

Run checks from this repo:

uv run --with ruff ruff check .
uv run --with pytest --with pytest-asyncio pytest -q
# or, if you have just installed:
just test