Mistral Evaluations

Namespace package for Mistral AI evaluation (Dora) utilities.

Overview

This package sets up the mistralai.evaluations namespace for Dora-specific utilities. It extends the Mistral AI SDK v2 namespace structure to allow evaluation features to be developed and released from this repository.

SDK v2 Namespace Package

This package follows PEP 420 implicit namespace packaging to integrate with the Mistral AI SDK v2 structure. The code lives in this repository (Dora) but will be importable from mistralai.evaluations once SDK v2 is released.

Package Structure

mistralai/                  # Shared namespace (no __init__.py)
├── client/                 # Core SDK (mistralai package on PyPI)
├── workflows/              # Workflows SDK
├── evaluations/            # This package (Dora utilities)
└── extra/                  # Extra utilities

The mistralai/ directory has no __init__.py, allowing multiple packages to coexist under the shared mistralai namespace.

Installation

This package is currently under development. Once SDK v2 is released, it will be available as part of the mistralai namespace.

For local development:

uv sync

Usage

Basic evaluation

from mistralai.evaluations import Evaluation, Evaluator, Project

run = await client.evaluation.run(
    project=Project(name="My Project"),
    evaluation=Evaluation(name="My Evaluation"),
    dataset=[{"input": "hello", "expected": "greeting"}],
    task=lambda input_record: classify(input_record["input"]),
    evaluators=[
        Evaluator(name="accuracy", scorer=lambda input_record, output: 1 if output == input_record["expected"] else 0),
    ],
)
run.show(level="scores")

Run-level evaluators

Run evaluators execute after all records are processed and receive the full run context (records, statistics, metrics, metadata). Use them for global assertions or aggregate metrics.

from mistralai.evaluations import RunEvaluator

run = await client.evaluation.run(
    ...,
    evaluators=[
        Evaluator(name="accuracy", scorer=lambda input_record, output: 1 if output == input_record["expected"] else 0),
    ],
    run_evaluators=[
        RunEvaluator(
            name="accuracy_above_50pct",
            scorer=lambda ctx: ctx.statistics["accuracy"].avg > 0.5,
        ),
        RunEvaluator(
            name="all_records_scored",
            scorer=lambda ctx: all(
                len(r.output.generations) > 0 for r in ctx.records
            ),
        ),
    ],
)
# Results are in run.run_scores
print(run.run_scores)  # {"accuracy_above_50pct": True, "all_records_scored": True}

The RunEvaluatorContext provides:

Retrying failed records

If an evaluation only partially fails, use retry_failed_records instead of re-running the entire evaluation from scratch. It re-runs only the failed records and patches the original run in place, avoiding work on successful records when failures came from transient API errors, rate limits, or a bug in the task or scorer.

from mistralai.evaluations import Evaluation, Evaluator

run = await client.evaluation.run(
    evaluation=Evaluation(name="My Evaluation"),
    dataset=dataset,
    task=flaky_task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
)

# Fix the task or scorer if needed, then retry only the failed records.
result = await client.evaluation.retry_failed_records(
    run_id=run.run_id,
    dataset=dataset,
    task=fixed_task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
)

print(f"Retried: {result.retried_count}, Patched: {result.patched_count}")

Pass the same dataset used in the original run so the SDK can map failed records back to their inputs. You can pass a fixed task or updated evaluators before retrying. When run-level evaluators are provided, their scores are recomputed after patching.

get_score helper

When writing run evaluators, accessing individual scores requires navigating through generations and score lists. The get_score helper simplifies this:

from mistralai.evaluations import RunEvaluatorContext, Score, get_score

def f1_scorer(ctx: RunEvaluatorContext) -> Score:
    tp = fp = fn = 0
    for record in ctx.records:
        expected = str(record.input["expected"]).lower()
        # Instead of: record.output.generations[0].scores["accuracy"][0].value
        is_correct = get_score(record, "accuracy").value == 1
        ...

Returns a Score object (with value, rationale, metadata).

Raises if no scores are found. When num_generations > 1, raises with a helpful message unless an aggregate function is provided. For pre-computed aggregations, use record.output.statistics[evaluator_name] instead.

Development

# Install dependencies
uv sync

# Run tests
uv run pytest

# Lint and format
uv run ruff check --fix .
uv run ruff format .

# Type check
uv run mypy mistralai

Publishing

Release from GitHub

The version is not stored in code. This package uses uv-dynamic-versioning: the published version comes from the release tag (evaluations-sdk/vX.Y.Z), and the workflow stamps the version you pass onto the build via UV_DYNAMIC_VERSIONING_BYPASS. There is no version-bump PR — just run the workflow with the version you want.

Run the "Release Evaluations SDK" GitHub workflow manually. Choose which registries to publish to via the boolean inputs — they're independent, so you can dogfood internally without shipping to clients:

1. PyPI (public), via a trusted publisher (OIDC) — the primary channel. Default on.
2. Gemfury, for internal distribution. Default on.
3. Cloudsmith (mistral-ai/sdk-distribution), for private-preview client distribution. Default off — opt in when previewing to specific clients.

Run the workflow from main for normal SDK releases, or from the relevant mais-* release branch for on-prem patch lines. Enter the version as the workflow version input (PEP 440, e.g. 0.7.0 or 0.7.0rc1); the workflow fails if the tag evaluations-sdk/vX.Y.Z already exists.

The git tag evaluations-sdk/vX.Y.Z records the released commit for that version, so it is created as soon as any selected registry publishes successfully (and none of the selected ones failed). To dogfood privately, publish an rc to Gemfury only; the rc tag won't collide with the later public release.

If one registry succeeds and another fails, rerun the failed GitHub Actions jobs only. Do not rerun the full workflow, because the already-successful registry may reject the duplicate version.

The TypeScript mirror (ts/packages/local-observability-sdk/) still carries its own package.json / cli.ts version and is not published by this workflow; keep it in sync when practical.

Manual Gemfury publish

Retrieve the "Gemfury Upload Token - Mistral" credentials from Bitwarden. The make publish command expects GEMFURY_USERNAME and GEMFURY_PASSWORD environment variables.

# Publish to Gemfury
# this will clean up the dist/ directory then build and publish
make publish

Installing from PyPI (public)

pip install mistralai-evaluations

Installing from Gemfury (internal)

pip install --index-url https://pypi.fury.io/mistralai/ mistralai-evaluations --extra-index-url https://pypi.org/simple/