themis-eval 4.0.0

Evaluation scaffold for LLM research, benchmarking, and reproducible experiment runs.

Themis

Themis is a Python package for running reproducible LLM evaluations. It gives you a typed scaffold for defining datasets, generators, parsers, metrics, judge workflows, and persistent run artifacts without forcing you into one provider or benchmark.

The published package name is themis-eval. The Python import namespace and CLI command are both themis.

Install

uv add themis-eval

Optional extras:

• uv add "themis-eval[openai]"
• uv add "themis-eval[vllm]" on Linux
• uv add "themis-eval[langgraph]"
• uv add "themis-eval[datasets]"
• uv add "themis-eval[mongodb]"
• uv add "themis-eval[postgres]"
• uv sync --extra docs for local documentation builds from a repo checkout

Quick Start

from themis import evaluate
from themis.core.models import Case, Dataset

result = evaluate(
    model="builtin/demo_generator",
    data=[
        Dataset(
            dataset_id="sample",
            cases=[
                Case(
                    case_id="case-1",
                    input={"question": "2+2"},
                    expected_output={"answer": "4"},
                )
            ],
        )
    ],
    metric="builtin/exact_match",
    parser="builtin/json_identity",
)

print(result.run_id, result.status.value)

Custom Extensions

Themis is designed to be extended. You can plug in custom generators, parsers, reducers, metrics, judge models, and store backends through the Python API or config-driven workflows.

• Start with Experiment(...) when you want a reusable compiled evaluation definition.
• Start with evaluate(...) when you want the shortest path from inline data to a completed run.
• Use docs/how-to/author-custom-components.md for custom component authoring.

CLI

After installation, the package exposes the themis CLI:

themis quick-eval inline \
  --model builtin/demo_generator \
  --metric builtin/exact_match \
  --parser builtin/json_identity \
  --input '{"question":"2+2"}' \
  --expected-output '{"answer":"4"}'

Documentation

• Start here: docs/index.md
• Installation guide: docs/start-here/installation.md
• API layer chooser: docs/start-here/choose-your-api-layer.md
• Python API reference: docs/reference/python-api.md
• Extension boundaries: docs/explanation/extension-boundaries.md

Build the docs locally with:

uv sync --extra docs
uv run mkdocs build --strict

Contributing

Contributor setup and release guidance live in CONTRIBUTING.md.