annotations-evaluate

Python library for evaluating span annotations — built for nested and overlapping annotations as they occur in historical corpora (BeNASch/HLR-GT) and in LLM-assisted tagging. It provides a nested report by default and an opt-in, SemEval-conformant flat mode for comparability with published NER numbers.

Part of the annotations-* family: the annotation pipeline lives in annotations4all, evaluation orchestration in annotations-orchestration; run projects combine both. This library is the family's home for metrics.

Why a separate library

• The previous evaluation setup relied on a vendored nervaluate snapshot (identical to PyPI nervaluate 0.2.0) which miscounts as soon as predictions are not exactly equal to a gold span — also on flat data: one prediction is scored once per overlapping gold span, and a differently labelled prediction "consumes" a gold span so that the actually matching prediction is counted as spurious.
• Upstream nervaluate 1.2.x supports nesting but thinks flat: no level, no parent–child structure, no crossing, and undecodable model output shows up as an ordinary error.
• What a state means is research work — it belongs in a versioned spec in this repository, not in a vendored snapshot.

Background, measurements and the decisions E1–E12 are documented in the project's internal notes.

Features

• Nested report (default): precision/recall/F1 per nesting level and per element type; structural precision/recall/F1 over parent–child edges (catches "spans correct, nesting wrong"); recall bucketed by span length and depth; decode rate including a breakdown by parser warning code (unparsed is its own state).
• Flat mode (opt-in): strict, exact, partial, ent_type following SemEval-2013, validated against the reference example in the nervaluate documentation and against nervaluate 1.2.1.
• Deterministic matching: per label, order-independent, each span matched at most once.
• Half-open spans ([start, end)), consistent with the family convention.
• Re-scoring (annotations-rescore): re-scores stored predictions without calling a model — nested and/or flat, with an old-versus-new table against the metrics recorded by earlier runs. The completeness of the input (read, ignored, without predictions) is part of the report, and an incomplete report is marked as such.
• Record adapters (document_from_record, spans_from_ground_truth, spans_from_predictions, decode_from_predictions): records held in memory (harness prodigy/tags shapes) become documents of spec §2.6 directly — the same translation annotations-rescore applies per line. predictions may be the plain span list or the mapping with answers/unparsed (§2.6), so a re-scored run reports a decode rate (§6.2.4) when it was recorded instead of n/a.

Usage

from annotations_evaluate import evaluate

documents = [
    {
        "document_id": "doc-1",
        "text": "Der Karl Schmidt liest in Wien.",
        "gold": [
            {"label": "PERSON", "start": 4, "end": 16},
            {"label": "CITY", "start": 26, "end": 30},
        ],
        "prediction": {
            "spans": [
                {"label": "PERSON", "start": 4, "end": 16},
                {"label": "CITY", "start": 26, "end": 29},
            ],
            "answers": 1,
            "unparsed": [],
        },
    }
]

nested = evaluate(documents)  # nested report — the default mode
print(nested.overall.counts)
# StateCounts(correct=1, part_boundary=1, correct_label_other_bounds=0, spurious=0, missed=0, duplicates=0)
print(nested.overall.f1)  # 0.75
print(nested.element["CITY"].counts)
# StateCounts(correct=0, part_boundary=1, correct_label_other_bounds=0, spurious=0, missed=0, duplicates=0)
print(nested.decode.answers, nested.decode.decode_rate)  # 1 1.0 — the answer was decoded

flat = evaluate(documents, mode="flat")  # SemEval-2013 scenarios (opt-in)
print(flat.scenarios["strict"].f1, flat.scenarios["partial"].f1, flat.overlap_threshold)
# 0.5 0.75 1.0

Documents are the records of spec §2.6: gold and prediction.spans are half-open spans ([start, end)), answers/unparsed carry the decode metadata of the run, so the report can name a decode rate instead of n/a.

Recorded runs are re-scored without a model call:

annotations-rescore results/runs/<run_id> --mode both \
    --legacy-metrics results/runs/<run_id>/metrics.json

Specification

The evaluation semantics are normative and versioned; terms, span relations, deterministic matching, the state catalogue and both metric modes are defined in docs/evaluation-semantics.md (Spec 1.1, English). It is the implementation contract and the reference for every published number.

Scope

• No scheduler and no model calls — that is annotations-orchestration's and the run project's job.
• No prompt or parser logic — that lives in annotations4all.
• No runtime dependencies: standard library only.

Installation

pip install annotations-evaluate

The package is published on PyPI as annotations-evaluate; the version follows Semantic Versioning, the metrics follow the spec version of docs/evaluation-semantics.md.

Development

uv sync
uv run ruff check .
uv run ruff format --check .
uv run pytest

.agent-memories/ is gitignored (family convention). README.md carries the repository metadata front matter and is excluded from release archives; the PyPI readme is this file.

Citation

If you use this software in academic work, please cite it as:

Dresselhaus, Nicole. (2026). annotations-evaluate (Version 0.1.0) [Software]. Humboldt-Universität zu Berlin. https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate

DOI: 10.5281/zenodo.22795383

The concept DOI 10.5281/zenodo.22795382 always resolves to the latest version.

Machine-readable metadata is in CITATION.cff.

License

MIT.