annotations4all

Languages: Deutsch · English

annotations4all is a schema-first Python library for LLM-assisted span annotation (commonly used for named-entity recognition). It generates prompts from a user-defined tag schema, parses annotated LLM responses in <<TAG>>…</TAG>> format, and returns matches as offset-based spans.

Status: 0.2.0 is intended as an alpha release. The stable v0.2 surface consists of prompt taggers, merge helpers, parsers, and client helpers. Experimental legacy clients are marked as such. DOI of this version: 10.5281/zenodo.22790107.

Overview of the annotations-* library family — raw data and gold annotations, annotating, running experiments, evaluating, reporting:

Documentation

The tutorial documents the v0.2 API surface and shows examples against local, OpenAI-compatible installations (e.g. a llama.cpp server or Ollama's OpenAI-compatible endpoint).

Installation

After the PyPI release:

python -m pip install annotations4all

For development, from the repository:

python -m venv .venv
. .venv/bin/activate
python -m pip install -e .
python -m pytest

Schema-first quick start

The following example shows the preferred v0.2 entry point: define the tag schema first, then generate prompt messages and map the model response back to spans.

from annotations4all import ConfigurableTagger

text = "Max Mustermann lives in Berlin."
tagger = ConfigurableTagger(
    tags=[("PER", "Person names"), ("LOC", "Places")],
    context="Short modern example sentence.",
    language="en",
)

messages = tagger.get_prompt(text)
for message in messages:
    print(message["role"])
    print(message["content"])

# Response of an LLM, e.g. from an OpenAI-compatible endpoint:
response = "<<PER>>Max Mustermann</PER>> lives in <<LOC>>Berlin</LOC>>."
spans = tagger.parse_response(response, text)
print(spans)

tags describes the tag schema of the concrete workflow. context holds material- or task-specific annotation hints, not arbitrary runtime data. Common NER tags such as PER, LOC, and ORG are well suited for quick starts, but the library does not enforce a fixed ontology.

Minimal example without an LLM call

The following example shows the smallest stable core: an already annotated response is mapped back to character positions in the original text.

from annotations4all import parse_region_response, parse_region_response_detailed

text = "Max Mustermann lives in Berlin."
response = "<<PER>>Max Mustermann</PER>> lives in <<LOC>>Berlin</LOC>>."

spans = parse_region_response(response, text)
detailed = parse_region_response_detailed(response, text)
print(spans)
print(detailed.warnings)
# [{'label': 'PER', 'meta': None, 'start': 0, 'end': 14}, {'label': 'LOC', 'meta': None, 'start': 24, 'end': 30}]

Writing custom taggers

For advanced use cases, custom taggers can be implemented. The base class ChatTagger remains importable for this purpose but is not part of the highlighted package-root API of v0.2.

from annotations4all.taggers.base import ChatTagger, Message
from annotations4all.utils.response_parser import parse_region_response


class MyTagger(ChatTagger):
    def name(self) -> str:
        return "my-tagger"

    def get_prompt(self, text: str) -> list[Message]:
        return [
            {
                "role": "system",
                "content": "Annotate persons as <<PER>>…</PER>> and places as <<LOC>>…</LOC>>.",
            },
            {"role": "user", "content": text},
        ]

    def parse_response(self, response: str, text: str, logfile=None):
        return parse_region_response(response, text, logfile=logfile)

The model response must reproduce the original text as exactly as possible and mark spans with opening and closing tags:

<<PER>>Max Mustermann</PER>> lives in <<LOC>>Berlin</LOC>>.

Tag schemas

The library does not enforce a fixed ontology. The tag schema belongs to the respective research or annotation workflow. Common quick-start tags include:

• PER: persons
• LOC: places
• ORG: organizations
• project-specific tags such as DOM, DAT, KG, etc.

Tags can optionally carry metadata, e.g. <<LOC:city>>Berlin</LOC>>. The parser returns this metadata as meta when present. The returned objects are spans with at least label, start, and end.

Hierarchical tagging in one pass

Nested answers are parsed directly — the parser is a stack decoder, not a flat one:

from annotations4all import ConfigurableTagger

text = "Ernst August von Testingen"
tagger = ConfigurableTagger(
    tags=[("PER", "person names (outer span)"), ("FN", "given name"), ("LN", "family name")],
    context="PER encloses the whole name mention; FN and LN are nested inside it.",
    language="en",
)
response = "<<PER>><<FN>>Ernst August</FN>> <<LN>>von Testingen</LN>></PER>>"
print(tagger.parse_response(response, text))
# [{'label': 'PER', 'meta': None, 'start': 0, 'end': 26},
#  {'label': 'FN', 'meta': None, 'start': 0, 'end': 12},
#  {'label': 'LN', 'meta': None, 'start': 13, 'end': 26}]

• The prompt decides whether the model nests: the packaged templates allow nesting explicitly and forbid crossing tags, and the intended structure belongs in context as well.
• Children are always searched inside their parent span, and repeated text is anchored via the tag position in the response (no silent first-match drift). A child that cannot be placed inside its parent is reported as nested-out-of-parent: nested_policy="strict" (default) drops it, "repair" keeps it at the anchor that was found.
• A closing tag written one > short (</TAG> instead of </TAG>>) is accepted and reported as close-tag-single-bracket; tolerate_single_bracket_close=False rejects it.
• Overlap is allowed as long as no span encloses the other — two flat spans such as <<GIV>>…</GIV>> <<RCV>>…</RCV>> are both parsed.
• Crossing tags (<<GIV>>…<<RCV>>…</GIV>>…</RCV>>) cannot be represented in the <<TAG>> format: the parser keeps the span that closes first, drops the other one and reports unclosed-nested-tag / unmatched-close-tag. Use the per-label mode for those (see below) — it keeps both spans and reports the crossing conflict.
• For long outer spans raise max_match_length (default 100); otherwise they are truncated (span-too-long).

Nested fallback: per-label mode + merge

For nested or hierarchical schemas a single, nested model answer can be too unreliable (see the section above). The fallback annotates one label (or one label group) per model call and merges the flat runs deterministically afterwards:

from annotations4all import OpenAICompatClient, PerLabelTagger

text = "Anna Müller lives in Berlin."
tagger = PerLabelTagger(
    tags=[("PER", "Person names"), ("LOC", "Places")],
    context="Short modern example sentence.",
    language="en",
)
client = OpenAICompatClient(api_url="https://example.invalid/v1", api_key="<token>")

result = tagger.annotate(text, client, model="my-model", stream=False)
print(result.flat_spans())
# [{'label': 'PER', 'meta': None, 'start': 0, 'end': 11}, {'label': 'LOC', 'meta': None, 'start': 21, 'end': 27}]
print(result.spans)  # nested: parent spans with `children`
print(result.conflicts)  # conflict list

The merge is also usable on its own — a pure function on flat spans (no client, no prompt):

from annotations4all import merge_span_sets

result = merge_span_sets(
    {
        "PER": [{"label": "PER", "meta": None, "start": 0, "end": 11}],
        "LOC": [{"label": "LOC", "meta": None, "start": 21, "end": 27}],
    }
)

Merge rules:

• Anchor rule: the hierarchy is derived from containment only — a span's parent is the smallest span that strictly encloses it, never a similar span found elsewhere in the document. Bounds are half-open ([start, end)), so touching spans are adjacent, not overlapping.
• Conflicts are reported, never silently repaired: duplicate (same label, same bounds, twice), intra-label-overlap (same label, partial overlap) and crossing (different labels, partial overlap).
• Order-independent: permuting runs or spans yields identical output.

Cost: one model call per run — with one label per run that is ≈ the number of labels. For large hierarchical schemas the mode is therefore only sensible for the outer level; recursive prompting per detected element is the alternative. In addition, parse_region_response and parse_region_response_detailed accept bounds=(start, end) to anchor candidates only inside an already known parent span (no global fuzzy search).

Backends and clients

For v0.2, only a narrow, explicit OpenAI-compatible chat-completions interface is officially supported. The target server must offer an endpoint such as /v1/chat/completions and be compatible with the request/response shape of the README examples.

from annotations4all import ConfigurableTagger, OpenAICompatClient

text = "Max Mustermann lives in Berlin."
tagger = ConfigurableTagger(
    tags=[("PER", "Person names"), ("LOC", "Places")],
    context="Short modern example sentence.",
    language="en",
)
client = OpenAICompatClient(
    api_url="https://example.invalid/v1",
    api_key="<token>",
)

chunks = client.chat.create(
    model="my-model",
    messages=tagger.get_prompt(text),
    stream=True,
    temperature=0,
)
response = "".join(chunk.answer for chunk in chunks)
spans = tagger.parse_response(response, text)
print(spans)

OpenAICompatClient reads API keys either from api_key= or from an environment variable. The default is OPENAI_API_KEY; api_key_env= selects a different name. Local servers that do not require authentication automatically receive a dummy key, because the underlying OpenAI SDK still expects a value.

Provider-specific request fields (e.g. reasoning options) are passed through generically via extra_body= — the library does not interpret the payload, the endpoint defines the schema:

client.chat.create(
    model="my-model",
    messages=tagger.get_prompt(text),
    extra_body={"reasoning": {"enabled": False}},
)

The v0.2 compatibility promise is deliberately narrow: standard content responses and the streaming form used in the tests/examples are the target. The semantics of provider-specific fields are explicitly not a stability promise — the generic extra_body passthrough itself is.

Tests

python -m pytest

The tests check prompt invariants, parser golden cases, a small fuzz baseline, client helper structures, the merge rules (order independence, conflict classes) and the per-label mode. A live smoke test of the per-label mode against an OpenAI-compatible endpoint lives in tests/integration/test_endpoint_smoke.py (env-driven; it is skipped without configuration).

Citation

If you use this software in academic work, please cite it as follows:

Dresselhaus, Nicole. (2026). annotations4all (Version 0.2.0) [Software]. Humboldt-Universität zu Berlin. https://scm.cms.hu-berlin.de/annotations4all/annotations4all

DOI: 10.5281/zenodo.22790107

The concept DOI 10.5281/zenodo.22011370 always resolves to the latest version (0.1.1: 10.5281/zenodo.22011371).

Machine-readable metadata is available in CITATION.cff.

License

This software is licensed under the MIT License. See LICENSE for details.