annotations-orchestration

Generic Python library for orchestrating LLM evaluation runs: a resumable parallel scheduler over experiment manifests, plus span metrics for precision/recall/F1 against gold annotations — computed by annotations-evaluate, the family's spec-driven metrics library.

Part of the annotations-* family: the annotation pipeline lives in annotations4all, this library provides the evaluation/experiment infrastructure, and concrete run projects combine both.

Features

• Decoupled scheduler: the existing legacy runner speaks a subprocess CLI contract (RunSpec → flags), not annotations4all types. The optional standard runner, provider discovery and hints use annotations4all; the base installation does not require that library.
• Conservative concurrency: documents within a run are processed linearly; remote runs share a pool (execution.remote_parallelism, default 3); self_hosted runs get one sequential lane per host. The exact HU endpoint gets a separate serial lane regardless of mode, with at most 10 requests per minute; request-level pacing is implemented by the client.
• Resumable: run state is persisted; completed runs are skipped, interrupted/failed runs re-execute on restart.
• Manifest-driven and validated: experiment YAMLs + optional controller manifests (defaults/execution/order). Run fields (model, host, port, context, shots, data_dir, net, api_url, api_key_env, reasoning, reasoning_effort, requests_per_minute, runner, annotation, tagger, nested) are checked for type and allowed values, and unknown keys are rejected instead of being ignored silently — with file, key and the list of allowed keys (plus a typo suggestion); errors are one line (error: …, exit code 2) instead of a traceback.
• Metrics: span evaluation with precision/recall/F1, provided by annotations-evaluate — the SemEval-2013 compatible flat schemes (strict, entity type, partial, exact) used for the run reports, and that library's nested standard report (levels, structure, decode rate) for nested annotations.

Installation

python -m pip install annotations-orchestration

Requires Python 3.11+.

The metrics library of the family, annotations-evaluate, is installed as a regular dependency from PyPI.

Quickstart

# Plan an experiments directory (no execution; writes runtime state)
annotations-orchestrate --experiments-dir experiments/ --dry-run

# Run the experiments
annotations-orchestrate --experiments-dir experiments/

Python API

from pathlib import Path

from annotations_orchestration import evaluate_classifier

# One record in the harness format: gold spans per annotation tool, predicted spans.
records = [
    {
        "ground_truth": {"prodigy": [{"start": 0, "end": 3, "label": "PER"}]},
        "predictions": [{"start": 0, "end": 3, "label": "PER"}],
    }
]

# Pass records in memory, or omit "predictions" to read a run directory.
overall, per_tag = evaluate_classifier(
    eval_dir=Path("results"),
    predictions=records,
)
f1 = overall["ent_type"]["f1"]  # 1.0

CLI

annotations-orchestrate options: --manifest, --experiments-dir, --state, --results-dir, --dry-run, --active-self-hosted, --graceful-shutdown-seconds.

Config editor

Version 0.3.0 includes an optional Textual TUI for viewing, creating and editing single-/multi-run experiments and controller manifests. A structured form, full YAML editor and effective-run preview share the CLI's manifest semantics. Saves include a change diff, atomic replacement and protection against unsaved or externally modified documents.

Install it with the tui extra:

python -m pip install 'annotations-orchestration[tui]'
annotations-tui --experiments-dir experiments/
annotations-tui --manifest experiments/controller.yaml --read-only

From a source checkout, uv sync --extra tui installs the same entry points, so uv run --extra tui annotations-tui … (or python -m annotations_orchestration.tui … with --file <path>) behaves identically.

For provider discovery, capabilities, hints and standard annotation, also install the annotate extra; it requires annotations4all>=0.3.1 (the 0.2 series has neither create_chat_client nor the capability modules). Local installation and examples: annotation workflow.

Interactive use requires a terminal; --help works without a TTY or the extra. The existing CLI does not require the extra.

• Ctrl+N new, Ctrl+O open, Ctrl+S save, Ctrl+Shift+S save as.
• F2 Form / F3 YAML / F4 Preview / F5 Duplicate; Ctrl+R reload, Ctrl+B file list, Ctrl+Q quit, F1 help.
• Read-only viewing and raw editing of invalid YAML for repair.
• Responsive cards group model/prompting, annotation/input and connection, with contextual fields. Groups start expanded; manual collapse choices persist.
• Experiment → Defaults → Runs: consistent inheritance, newly added runs stay {}; local overrides are red, custom inherited values normal, built-ins dim. Legacy headers remain readable and migrate only after explicit confirmation.
• Runs can be added, duplicated, removed and reordered; false, 0, empty strings and Unset remain distinct.
• Annotation dialog for labels/descriptions, context and language, plus a read-only data-directory inspector showing document/gold/span counts.
• Keyless manual configuration stays available. Current live effort metadata limits selection to the exact model levels plus Default/Unset; incompatible existing values are diagnosed, never silently rewritten.
• A separate green ✓ means positive live evidence for that exact model, reasoning or effort value, not guaranteed quality or successful inference.
• Contextual Hints… / F1 and More → Hints explain reasoning, costs, verification and evaluation from the central static library resource.
• F6 opens provider/model selection with a custom ENV name or manual model ID; provider-reported model lists have a per-credential session cache and Refresh.
• F8 manages local .env keys with masked display; F9 opens UI options. Packaged provider definitions are extended/overridden by cwd/models.yaml.
• NerdFont icons are on by default; --no-nerd-fonts overrides this for the session. The theme is chosen in F9 or with --theme (session only), also via Ctrl+P; explicitly saved preferences live in cwd/settings.yaml.
• No experiment execution, run control or state/lock files in the editor. Explicit network requests only read model metadata, not test completions; credentials never enter experiment YAML/settings. api_key_env remains an environment variable name, not a credential value.

For full controls, inheritance and storage guarantees, see the editor guide (English).

Declarative annotation

runner: standard with annotation configures Configurable/Per-Label jobs without a custom Python frontend; runner: legacy remains the default for existing consumers. The standard runner preflights all input before inference and requires fresh output paths, even with --force-predict; no item resume.

Top-level Prodigy spans and ground_truth.prodigy/tags are normalized in memory, never in the source files. Missing gold is annotation-only; explicitly empty gold remains evaluable. Per-label makes one model call per label/document; nesting requires the configurable tagger and user-written hierarchy instructions.

Local-library setup, full manifest, preflight, outputs and the extended consumer CLI contract: annotation workflow. On 2026-10-09 the maintainer confirmed a successful live consumer run using a TUI-generated configuration, not blanket support for all providers.

Development

The full suite requires both extras and the development dependencies. No sibling checkout is needed; the compatible annotation library is installed from PyPI:

uv sync --locked --all-extras
uv run --locked --all-extras ruff check .
uv run --locked --all-extras ruff format --check .
uv run --locked --all-extras pytest

uv sync --extra tui alone does not supply library integration-test dependencies. For development against local library sources, see the annotation workflow; uv run may replace manually installed editable packages with the locked version. Build with uv build; see the linked annotation workflow for compatibility and packaging boundaries.

Citation

If you use this software in academic work, please cite it as follows:

Dresselhaus, Nicole. (2026). annotations-orchestration (Version 0.3.0) [Software]. Humboldt-Universität zu Berlin. https://doi.org/10.5281/zenodo.23495464

Concept DOI: 10.5281/zenodo.22015763 (always resolves to the newest version)

Machine-readable metadata is available in CITATION.cff.

License

MIT.

The metrics are computed by annotations-evaluate (MIT, same family), a regular PyPI dependency. This package contains no metric code of its own: it only adapts a run's records and writes the report.