annotations-orchestration

Generic Python library for orchestrating LLM evaluation runs: a resumable parallel scheduler over experiment manifests, plus span metrics for precision/recall/F1 against gold annotations — computed by annotations-evaluate, the family's spec-driven metrics library.

Part of the annotations-* family: the annotation pipeline lives in annotations4all, this library provides the evaluation/experiment infrastructure, and concrete run projects combine both.

Features

• Library-agnostic: the scheduler speaks a subprocess CLI contract (RunSpec → flags), not annotations4all types. Reusable for any LLM evaluation workflow.
• Conservative concurrency: documents within a run are processed linearly; remote runs share a pool (execution.remote_parallelism, default 3); self_hosted runs get one sequential lane per host.
• Resumable: run state is persisted; completed runs are skipped, interrupted/failed runs re-execute on restart.
• Manifest-driven and validated: experiment YAMLs + optional controller manifests (defaults/execution/order). Run fields (model, host, port, context, shots, data_dir, net, api_url, api_key_env, reasoning, tagger, nested) are checked for type and allowed values, and unknown keys are rejected instead of being ignored silently — with file, key and the list of allowed keys (plus a typo suggestion); errors are one line (error: …, exit code 2) instead of a traceback.
• Metrics: span evaluation with precision/recall/F1, provided by annotations-evaluate — the SemEval-2013 compatible flat schemes (strict, entity type, partial, exact) used for the run reports, and that library's nested standard report (levels, structure, decode rate) for nested annotations.

Installation

python -m pip install annotations-orchestration

Requires Python 3.11+.

The metrics library of the family, annotations-evaluate, is installed as a regular dependency from PyPI.

Quickstart

# Validate an experiments directory (dry run, no execution)
annotations-orchestrate --experiments-dir experiments/ --dry-run

# Run the experiments
annotations-orchestrate --experiments-dir experiments/

Python API

from pathlib import Path

from annotations_orchestration import calculate_metrics, evaluate_classifier

# One record in the harness format: gold spans per annotation tool, predicted spans.
records = [
    {
        "ground_truth": {"prodigy": [{"start": 0, "end": 3, "label": "PER"}]},
        "predictions": [{"start": 0, "end": 3, "label": "PER"}],
    }
]

# Pass records in memory, or omit "predictions" to read a run directory.
overall, per_tag = evaluate_classifier(
    data_dir=Path("data"),
    eval_dir=Path("results"),
    predictions=records,
)
f1 = calculate_metrics(overall, "ent_type")["f1"]  # 1.0

CLI

annotations-orchestrate options: --manifest, --experiments-dir, --state, --results-dir, --dry-run, --active-self-hosted, --graceful-shutdown-seconds.

Development

uv sync
uv run ruff check .
uv run ruff format --check .
uv run pytest

Citation

If you use this software in academic work, please cite it as follows:

Dresselhaus, Nicole. (2026). annotations-orchestration (Version 0.2.0) [Software]. Humboldt-Universität zu Berlin. https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration

DOI: 10.5281/zenodo.22797759

Machine-readable metadata is available in CITATION.cff.

License

MIT.

The metrics are computed by annotations-evaluate (MIT, same family) — the earlier vendored snapshot of MantisAI/nervaluate was removed in GitLab issue #3.