test-ann 0.2.0

Distributed annotation worker for EmbedKombinat

Run local LLM inference on your hardware to label query-document pairs for open embedding model training.

---

Website | Getting Started | Models | Leaderboard | Contributing

What is this?

The annotator is a headless labeling worker that runs on contributor hardware. It claims batches of unlabeled (query, document) pairs from the kombinat server, scores relevance using a local LLM, and submits annotations back — all without sending your data to any third-party API.

┌─────────────┐     claim batch     ┌─────────────┐
│  kombinat   │ ◄────────────────── │  annotator  │
│   server    │ ──────────────────► │  (your hw)  │
│             │   (query, doc) pairs│             │
│             │                     │  ┌────────┐ │
│             │   submit labels     │  │ Local  │ │
│             │ ◄────────────────── │  │  LLM   │ │
└─────────────┘                     │  └────────┘ │
                                    └─────────────┘

How it works

1. Authenticate via GitHub Device Flow — the CLI prints a short code, you enter it at https://github.com/login/device on any browser (laptop, phone, anywhere). Works on remote SSH'd hosts (Runpod, Lambda, EC2) and inside Docker without any port forwarding.
2. Detect hardware — NVIDIA GPU, Apple Silicon, or CPU-only
3. Download & load the best-fit LLM for your hardware from HuggingFace
4. Claim → Label → Submit in streaming micro-batches (lose at most one chunk on interrupt)

Each pair gets a relevance score from 0 (not relevant) to 3 (highly relevant) with a short reasoning.

Getting Started

Install

# NVIDIA GPU
pip install test-ann[vllm]

# Apple Silicon (M1/M2/M3/M4)
pip install test-ann[mlx]

# CPU-only
pip install test-ann[cpu]

Or run without installing:

uvx --from "test-ann[mlx]" annotator run

Authenticate

annotator login

Run

# Starts labeling (will prompt login if not authenticated)
annotator run

Docker (NVIDIA)

docker compose up

Supported Models

The annotator auto-selects the best model for your hardware. You can override with --model and --backend.

NVIDIA GPU (vLLM)

Apple Silicon (MLX)

CPU (llama.cpp)

CLI Reference

Usage: annotator [COMMAND] [OPTIONS]

Commands:
  run      Start the labeling loop (default)
  login    Authenticate via GitHub
  status   Show contributor profile and stats
  logout   Remove stored credentials

Options (run):
  --batch-size INT             Pairs per batch (default: 100, max: 500)
  --model TEXT                 Override model ID
  --quantization TEXT          Override quantization
  --backend [vllm|mlx|cpu]    Override backend
  --gpu-memory-utilization FLOAT  GPU fraction (default: 0.9)
  --dry-run                    Resolve hardware & model, then exit

Annotator Leaderboard

Top contributors by total annotations submitted. Updated in real-time by the kombinat server.

Want to see your name here? pip install test-ann[mlx] && annotator run

Contributing

# Clone and install dev dependencies
git clone https://github.com/embedkombinat/annotator.git
cd annotator
pip install -e ".[dev,mlx]"  # or .[dev,vllm] for NVIDIA

# Run checks
ruff check .
mypy annotator/
pytest

License

Apache 2.0 — see LICENSE for details.