annotations-orchestration

Generic Python library for orchestrating LLM evaluation runs: a resumable parallel scheduler over experiment manifests, plus nervaluate-based metrics (precision/recall/F1 against gold annotations).

Part of the annotations-* family: the annotation pipeline lives in annotations4all, this library provides the evaluation/experiment infrastructure, and concrete run projects (e.g. evaluation_2026) combine both.

Features

• Library-agnostic: the scheduler speaks a subprocess CLI contract (RunSpec → flags), not annotations4all types. Reusable for any LLM evaluation workflow.
• Conservative concurrency: documents within a run are processed linearly; remote runs share a pool (execution.remote_parallelism, default 3); self_hosted runs get one sequential lane per host.
• Resumable: run state is persisted; completed runs are skipped, interrupted/failed runs re-execute on restart.
• Manifest-driven: experiment YAMLs + optional controller manifests (defaults/execution/order).
• Metrics: NER evaluation with precision/recall/F1 per entity type, using the classic nervaluate evaluation schemes (strict, entity type, partial, exact).

Installation

python -m pip install annotations-orchestration

Requires Python 3.11+.

Quickstart

# Validate an experiments directory (dry run, no execution)
annotations-orchestrate --experiments-dir experiments/ --dry-run

# Run the experiments
annotations-orchestrate --experiments-dir experiments/

Python API

from pathlib import Path

from annotations_orchestration import evaluate_classifier, calculate_metrics

# predictions: list of {"ground_truth": {"prodigy": [...]}, "predictions": [...]}
overall, per_tag = evaluate_classifier(
    data_dir=Path("data"),
    eval_dir=Path("results"),
    predictions=predictions,
)
f1 = calculate_metrics(overall, "ent_type")["f1"]

CLI

annotations-orchestrate options: --manifest, --experiments-dir, --state, --results-dir, --dry-run, --active-self-hosted, --graceful-shutdown-seconds.

Development

uv sync
uv run ruff check .
uv run ruff format --check .
uv run pytest

Citation

If you use this software in academic work, please cite it as follows:

Dresselhaus, Nicole. (2026). annotations-orchestration (Version 0.1.0) [Software]. Humboldt-Universität zu Berlin. https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration

DOI: 10.5281/zenodo.22015764

Machine-readable metadata is available in CITATION.cff.

License

MIT. The bundled nervaluate_compat/ package is a vendored, unmodified snapshot of MantisAI/nervaluate commit 0bf272a (MIT, © 2020 David S. Batista and Matthew A. Upson).