Skip to main content

annomate

MCP server for VIA v3 — lets Claude Code read and write image annotations in real time alongside the user.

Part of caliperhq — open-source tools that give Claude senses for the things it builds.

Sister project to jscad-mcp.

What it does

  • Serves the VIA image annotator at a localhost URL
  • Implements VIA's push/pull protocol so annotations sync to a local store
  • Auto-refreshes the browser when Claude makes changes (no manual pull needed)
  • Exposes a suite of MCP tools so Claude can read, add, edit, and delete regions
  • Optional local-model assistance — open-vocabulary detection (GroundingDINO / YOLO-World), promptable segmentation (SAM 2), VLM verification (Florence-2), scene classification, annotation grading, and free-form Q&A (Qwen2.5-VL)
  • Optional format conversion — HEIC / HEIF / AVIF (iPhone photos), PDF pages, EXIF / GPS / camera metadata
  • Optional OCR — Tesseract over image regions, returns word-level boxes as detection candidates

All optional features advertise themselves on the MCP tool surface even when their dependencies aren't installed — they return a structured install hint instead of erroring.

Install

Base (annotation server only)

python -m venv ~/.local/annomate
~/.local/annomate/bin/pip install annomate-mcp

Pick the extras you want

# Local AI models (~3 GB on disk, lazy-downloaded on first use)
pip install 'annomate-mcp[ai]'

# Faster detection (YOLO-World, ~95 MB)
pip install 'annomate-mcp[ai,yolo]'

# Free-form Q&A via chat VLM (Qwen2.5-VL-3B, ~6 GB)
pip install 'annomate-mcp[ai,chat]'

# Format conversion (HEIC, PDF) + EXIF metadata
pip install 'annomate-mcp[io]'

# OCR via Tesseract
pip install 'annomate-mcp[ocr]'

# Everything
pip install 'annomate-mcp[ai,yolo,chat,io,ocr]'

System packages

Several optional features need command-line tools the Python extras can't install themselves. Install only what you'll use.

Feature Tool Debian/Ubuntu macOS Gentoo Fedora/RHEL Arch
PDF loading ([io]) pdftoppm apt install poppler-utils brew install poppler emerge app-text/poppler dnf install poppler-utils pacman -S poppler
OCR ([ocr]) tesseract apt install tesseract-ocr brew install tesseract emerge app-text/tesseract dnf install tesseract pacman -S tesseract
OCR language packs tesseract-<lang> apt install tesseract-ocr-eng tesseract-ocr-spa … brew install tesseract-lang set LINGUAS="en es …" then re-emerge tesseract dnf install tesseract-langpack-eng … pacman -S tesseract-data-eng …
Rich EXIF ([io]) exiftool apt install libimage-exiftool-perl brew install exiftool emerge media-libs/exiftool dnf install perl-Image-ExifTool pacman -S perl-image-exiftool
AI accelerator (optional) NVIDIA drivers + CUDA distribution-specific n/a (use MPS) emerge nvidia-drivers dnf install akmod-nvidia pacman -S nvidia

HEIC support (pillow-heif) bundles its own libheif; no system package is required.

Setup

Add to your Claude Code MCP config (.mcp.json):

{
  "mcpServers": {
    "annomate": {
      "command": "/path/to/annomate"
    }
  }
}

Find your entry point after install:

which annomate                    # global/user install
~/.local/annomate/bin/annomate    # venv

Usage

annomate                         # start server (port OS-assigned)
annomate --port 9669             # pin the port
annomate --browser               # open the UI on startup
annomate --no-ai                 # skip the model registry entirely
annomate --models-config FILE    # override ~/.config/annomate/models.toml

The local URL prints to stderr on startup. Open it to use the annotator; load images, draw boxes, then ask Claude about them — or ask Claude to add annotations directly. Changes appear in your browser within a few seconds.

Skills

Install the companion Claude Code skill so Claude knows the annotation workflow, the AI tools, and the trigger-phrase patterns automatically:

cp -r skills/annomate ~/.claude/skills/

The skill is split into a small SKILL.md plus sibling reference files (region-encoding, attributes, perception-gotchas, ai-tools, common-patterns) — Claude loads only what's relevant to a given request.

Tool reference (high level)

Core (always available)

via_get_annotator_url, via_add_file, via_get_image, via_get_image_crop, via_get_project, via_list_files, via_get_annotations, via_add_region, via_update_region, via_delete_region, via_update_project, via_save_project

Local-model assistance (needs [ai])

via_model_status, via_suggest_regions, via_tighten_region, via_verify_region, via_grade_annotations, via_classify_scene, via_ask_model, via_find_similar

IO layer (needs [io] / [ocr])

via_load_document (PDFs), via_read_metadata, via_run_ocr

See skills/annomate/ for the full per-tool guidance, and docs/design/ for the longer-form design docs.

License

MIT — see LICENSE. VIA is included under its BSD 2-Clause License — see NOTICE.

YOLOE (an optional adapter) is upstream-licensed AGPL-3.0; if you enable that pipeline make sure that license fits your deployment context. See src/annomate/models/yoloe.py for the in-tree notice.

Development

python -m venv venv
venv/bin/pip install -e ".[dev]"
venv/bin/pytest
venv/bin/annomate --browser

The VIA HTML is not committed directly; it's generated from the VIA submodule and committed to releases:

git submodule update --init --recursive
cd via/via-3.x.y/scripts && python3 pack.py via image_annotator && cd ../../..
python scripts/build_html.py
python -m build

Metadata

Release files for annomate-mcp 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for annomate-mcp 0.1.0
File Size Uploaded Provenance Actions
annomate_mcp-0.1.0.tar.gz 178.6 kB With provenance Details for annomate_mcp-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for annomate-mcp 0.1.0
File Interpreter ABI Platform Actions
annomate_mcp-0.1.0-py3-none-any.whl Python 3 none any Details for annomate_mcp-0.1.0-py3-none-any.whl

Total release size: 344.9 kB

Release files / annomate_mcp-0.1.0.tar.gz

Download URL annomate_mcp-0.1.0.tar.gz
Size 178.6 kB
Tags Source
SHA-256 checksum
How to use checksums
0795683bb12c62675399bddc895092c88814d51612d735b9f64f1e0dbdb5d78c
BLAKE2b-256 checksum
How to use checksums
d168518e960033f4302853323156717ba471ba72c5a51e31f59ccc275a49b10e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 29, 2026.

Transparency log

Release files / annomate_mcp-0.1.0-py3-none-any.whl

Download URL annomate_mcp-0.1.0-py3-none-any.whl
Size 166.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0c6e8556c07663ba927e18c03f14a9663e51114ebd048f2471c1370927803486
BLAKE2b-256 checksum
How to use checksums
644da10e4b06e182fbfd5ae8355b6b4ff8e5593f44c3ea41668b54e043eb51bc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 29, 2026.

Transparency log

Security for annomate-mcp 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for annomate_mcp-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 29, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page