dj-evals 0.5.0

Internal Django SSE eval runner

dj-evals

Internal Django helper for running one eval function with multiple argument sets in parallel, so you can visually compare output, tool calls, usage, cost, and completion behavior as the runs happen.

Django setup

Add dj_evals to INSTALLED_APPS and mount your eval view:

INSTALLED_APPS = [
    "dj_evals",
]

from django.urls import path

urlpatterns = [
    path("evals/run/", eval_run),
]

Protect the eval view with your app's normal authorization checks before using it in production.

Usage

from dj_evals import handle_eval_request

async def eval_run(request):
    return await handle_eval_request(
        request,
        allowed_paths={"myapp.evals.echo_eval"},
    )

Generate a URL for the view with one or more eval argument dictionaries:

from dj_evals import generate_eval_url

url = "/evals/run/" + generate_eval_url(
    "myapp.evals.echo_eval",
    {"model": "gpt-4.1", "text": "hello"},
    {"model": "gemini-3-flash-preview", "text": "hello"},
)

The GET request renders the comparison page. The page starts one POST request per argument set. Each POST response is a direct SSE stream from the eval run, and the browser renders those events into the comparison panels.

Eval function contract

The eval function must be importable by dotted path and whitelisted in allowed_paths. It can be a sync or async function that returns one event, or an async generator that yields events.

def echo_eval(text="hello", model=""):
    return {"type": "response.output_text.delta", "delta": text}


async def async_eval(text="hello", model=""):
    return {"type": "response.output_text.delta", "delta": text}


async def streaming_eval(text="hello"):
    yield {"type": "response.output_text.delta", "delta": text}
    yield {"type": "response.completed", "response": {"cost": 0}}

Eval argument dictionaries are passed to the eval function as keyword arguments.

Helper utilities

You can yield EvalEvent for custom progress messages in the output panel:

from dj_evals import EvalEvent


async def streaming_eval():
    event: EvalEvent = {
        "type": "dj_evals.event",
        "message": "Fetched documents",
    }
    yield event

Developers

Run a local demo server to try the library in a browser:

uv run --with daphne daphne -p 8003 examples.local_server:application
# or, if you have just installed:
just run

Then open http://127.0.0.1:8003/. The demo redirects to an eval URL that starts three argument sets side by side and streams their events over SSE.

Run checks from this repo:

uv run --with ruff ruff check .
uv run --with pytest --with pytest-asyncio pytest -q
# or, if you have just installed:
just test