What is this?

The annotator is a headless labeling worker that runs on contributor hardware. It claims batches of unlabeled (query, document) pairs from the kombinat server, scores relevance using a local LLM, and submits annotations back — all without sending your data to any third-party API.

┌─────────────┐     claim batch     ┌─────────────┐
│  kombinat   │ ◄────────────────── │  annotator  │
│   server    │ ──────────────────► │  (your hw)  │
│             │   (query, doc) pairs│             │
│             │                     │  ┌────────┐ │
│             │   submit labels     │  │ Local  │ │
│             │ ◄────────────────── │  │  LLM   │ │
└─────────────┘                     │  └────────┘ │
                                    └─────────────┘

How it works

1. Authenticate via GitHub Device Flow — the CLI prints a short code, you enter it at https://github.com/login/device on any browser (laptop, phone, anywhere). Works on remote SSH'd hosts (Runpod, Lambda, EC2) and inside Docker without any port forwarding.
2. Detect hardware — NVIDIA GPU, Apple Silicon, or CPU-only
3. Download & load the best-fit LLM for your hardware from HuggingFace
4. Claim → Label → Submit in streaming micro-batches (lose at most one chunk on interrupt)

Each pair gets a relevance score from 0 (not relevant) to 3 (highly relevant) with a short reasoning.

Getting Started

Install

# NVIDIA GPU
pip install embedkombinat-annotator[vllm]

# Apple Silicon (M1/M2/M3/M4)
pip install embedkombinat-annotator[mlx]

# CPU-only
pip install embedkombinat-annotator[cpu]

Or run without installing:

uvx --from "embedkombinat-annotator[mlx]" annotator run

Authenticate

annotator login

Run

# Starts labeling (will prompt login if not authenticated)
annotator run

Docker (NVIDIA)

docker compose up

Supported Models

The annotator auto-selects the best model for your hardware. You can override with --model and --backend.

NVIDIA GPU (vLLM)

Apple Silicon (MLX)

CPU (llama.cpp)

CLI Reference

Usage: annotator [COMMAND] [OPTIONS]

Commands:
  run      Start the labeling loop (default)
  login    Authenticate via GitHub
  status   Show contributor profile and stats
  logout   Remove stored credentials

Options (run):
  --batch-size INT             Pairs per batch (default: 100, max: 500)
  --model TEXT                 Override model ID
  --quantization TEXT          Override quantization
  --backend [vllm|mlx|cpu]    Override backend
  --gpu-memory-utilization FLOAT  GPU fraction (default: 0.9)
  --dry-run                    Resolve hardware & model, then exit

Leaderboard

See the live leaderboard at embedkombinat.github.io.

Contributing

See CONTRIBUTING.md for development setup and workflow. The short version:

git clone https://github.com/embedkombinat/annotator.git
cd annotator
uv sync --all-extras
uv run ruff check . && uv run ruff format --check .
uv run mypy annotator/
uv run pytest -v

License

Apache 2.0 — see LICENSE for details.