annotators

The Python client for annotators.ai: human review for any model output. Send what your model returned to have it checked, or leave the output out and have the item labeled. A machine check goes first, and people see only what it doubts.

pip install annotators

Python 3.10 or later. The full API reference is at annotators.ai/docs.

Get an API key

Sign up at annotators.ai/console, create a workspace, and make a key under API keys. The client reads it from the environment:

export ANNOTATORS_API_KEY=ak_...

Your first annotation

A task says what an item looks like and which fields to fill. Create one once:

from annotators import Annotators

client = Annotators()  # reads ANNOTATORS_API_KEY

client.tasks.create(
    slug="ticket-triage",
    name="Support ticket triage",
    instructions="Pick the team that should handle the ticket.",
    fields=[
        {"name": "team", "kind": "choice", "options": ["billing", "bug", "account", "other"]},
        {"name": "urgent", "kind": "bool"},
    ],
)

Field kinds are choice, multi_label, bool, score (with levels), extract (a list of strings) and text.

Then send an item, with what your model returned and how sure it was:

a = client.annotate(
    "ticket-triage",
    input={"subject": "Charged twice", "body": "I was billed twice for October."},
    output={"team": "billing", "urgent": False},   # what your model returned
    source={"model": "my-model", "confidence": 0.62},
    metadata={"ticket_id": "T-1042"},              # comes back with the verdict
    idempotency_key="ticket-T-1042",               # retries don't create a second item
)
print(a["id"], a["status"])

At or above the task's accept_threshold (0.9 by default) the output is approved without anyone looking. At 0.62 it goes to people, and you can review it yourself under Review in the console. Wait for the verdict, or have it sent to a webhook (below):

a = client.annotations.wait(a["id"])     # polls; fine for scripts
print(a["verdict"], a["final_output"])   # approved, corrected or rejected

final_output is the answer to use: your output if it was approved, the reviewer's if they corrected it, and None if it was rejected.

Who reviews

Left out, workforce is your own team (private): the members of your console workspace and the reviewers you invite.

r = client.reviewers.invite("sam@example.com", display_name="Sam")   # the token is shown once
client.reviewers.remove(r["id"])                                     # frees their seat

Invited reviewers and members who review in the console share your plan's reviewer seats. Past the limit, invite fails with AnnotatorsError 402.

The annotators.org crowd (workforce="open" or "vetted") is on a waitlist. Join it under Review in the console. Until your workspace has crowd access, asking for the crowd fails with AnnotatorsError 403.

Webhooks

Set a webhook URL under Settings in the console, or pass callback_url= with a call. Each completed annotation or survey is POSTed there, signed with your workspace's signing secret (also under Settings), and retried with backoff until your endpoint answers 2xx. Verify the signature against the raw request body:

from flask import Flask, request
from annotators import WebhookVerificationError, verify

app = Flask(__name__)

@app.post("/annotators")
def annotators_webhook():
    try:
        event = verify(request.get_data(), request.headers.get("Annotators-Signature", ""),
                       WEBHOOK_SECRET)
    except WebhookVerificationError:
        return "", 400
    if event["type"] == "annotation.completed":
        a = event["data"]
        save_triage(a["metadata"]["ticket_id"], a["final_output"], a["verdict"])
    return "", 204

verify rejects a bad signature, and a timestamp more than five minutes off, then returns the parsed event. Retried deliveries keep their event id, so you can skip ones you've seen.

Machine checks

With a confidence in source, overall or per field ("field_confidence": {"urgent": 0.4}), your number decides: at or above accept_threshold the output is approved, below it people review it. Fields you give no confidence for go to people. Without a confidence, our decision model checks the output. Pass check="annotators-judge-1" (with the call or in the task's defaults) to have the decision model check it even when you send a confidence, or check="none" to skip the machine.

mode picks who looks: "auto" (the default; machine first, people only if doubtful), "human" (always people), "machine" (never people) or "sample" (people review a sample_rate share).

No model of your own

Leave output out and the item is labeled for you, from the task's examples and any you pass with the call:

a = client.annotate(
    "ticket-triage",
    input={"subject": "Can't log in", "body": "Password reset email never arrives."},
    examples=[{"input": past.input, "output": past.output}],
    mode="machine",            # never people: doubtful results come back as low_confidence
)
team, confidence = a["final_output"], a["agreement"]

Surveys

Ask a number of people for their reaction to a message, headline or document (a task with kind="survey"):

s = client.surveys.create("message-reaction", input={"message": draft}, n_responses=50)

Errors and settings

API errors raise AnnotatorsError, with the HTTP status and the API's detail.

Annotators(api_key="ak_...", base_url="https://api.annotators.ai", timeout=30)

Both default to the environment: ANNOTATORS_API_KEY and ANNOTATORS_BASE_URL. The client is a context manager (with Annotators() as client:) and closes its HTTP connection on exit.

Links

• annotators.ai
• API reference
• Console
• MCP server for agents: request_human_review as a tool