annomate
MCP server for VIA v3 — lets Claude Code read and write image annotations in real time alongside the user.
Part of caliperhq — open-source tools that give Claude senses for the things it builds.
Sister project to jscad-mcp.
What it does
- Serves the VIA image annotator at a localhost URL
- Implements VIA's push/pull protocol so annotations sync to a local store
- Auto-refreshes the browser when Claude makes changes (no manual pull needed)
- Exposes a suite of MCP tools so Claude can read, add, edit, and delete regions
- Optional local-model assistance — open-vocabulary detection (GroundingDINO / YOLO-World), promptable segmentation (SAM 2), VLM verification (Florence-2), scene classification, annotation grading, and free-form Q&A (Qwen2.5-VL)
- Optional format conversion — HEIC / HEIF / AVIF (iPhone photos), PDF pages, EXIF / GPS / camera metadata
- Optional OCR — Tesseract over image regions, returns word-level boxes as detection candidates
All optional features advertise themselves on the MCP tool surface even when their dependencies aren't installed — they return a structured install hint instead of erroring.
Install
Base (annotation server only)
python -m venv ~/.local/annomate
~/.local/annomate/bin/pip install annomate-mcp
Pick the extras you want
# Local AI models (~3 GB on disk, lazy-downloaded on first use)
pip install 'annomate-mcp[ai]'
# Faster detection (YOLO-World, ~95 MB)
pip install 'annomate-mcp[ai,yolo]'
# Free-form Q&A via chat VLM (Qwen2.5-VL-3B, ~6 GB)
pip install 'annomate-mcp[ai,chat]'
# Format conversion (HEIC, PDF) + EXIF metadata
pip install 'annomate-mcp[io]'
# OCR via Tesseract
pip install 'annomate-mcp[ocr]'
# Everything
pip install 'annomate-mcp[ai,yolo,chat,io,ocr]'
System packages
Several optional features need command-line tools the Python extras can't install themselves. Install only what you'll use.
| Feature | Tool | Debian/Ubuntu | macOS | Gentoo | Fedora/RHEL | Arch |
|---|---|---|---|---|---|---|
PDF loading ([io]) |
pdftoppm |
apt install poppler-utils |
brew install poppler |
emerge app-text/poppler |
dnf install poppler-utils |
pacman -S poppler |
OCR ([ocr]) |
tesseract |
apt install tesseract-ocr |
brew install tesseract |
emerge app-text/tesseract |
dnf install tesseract |
pacman -S tesseract |
| OCR language packs | tesseract-<lang> |
apt install tesseract-ocr-eng tesseract-ocr-spa … |
brew install tesseract-lang |
set LINGUAS="en es …" then re-emerge tesseract |
dnf install tesseract-langpack-eng … |
pacman -S tesseract-data-eng … |
Rich EXIF ([io]) |
exiftool |
apt install libimage-exiftool-perl |
brew install exiftool |
emerge media-libs/exiftool |
dnf install perl-Image-ExifTool |
pacman -S perl-image-exiftool |
| AI accelerator (optional) | NVIDIA drivers + CUDA | distribution-specific | n/a (use MPS) | emerge nvidia-drivers |
dnf install akmod-nvidia |
pacman -S nvidia |
HEIC support (pillow-heif) bundles its own libheif; no system package is
required.
Setup
Add to your Claude Code MCP config (.mcp.json):
{
"mcpServers": {
"annomate": {
"command": "/path/to/annomate"
}
}
}
Find your entry point after install:
which annomate # global/user install
~/.local/annomate/bin/annomate # venv
Usage
annomate # start server (port OS-assigned)
annomate --port 9669 # pin the port
annomate --browser # open the UI on startup
annomate --no-ai # skip the model registry entirely
annomate --models-config FILE # override ~/.config/annomate/models.toml
The local URL prints to stderr on startup. Open it to use the annotator; load images, draw boxes, then ask Claude about them — or ask Claude to add annotations directly. Changes appear in your browser within a few seconds.
Skills
Install the companion Claude Code skill so Claude knows the annotation workflow, the AI tools, and the trigger-phrase patterns automatically:
cp -r skills/annomate ~/.claude/skills/
The skill is split into a small SKILL.md plus sibling reference files
(region-encoding, attributes, perception-gotchas, ai-tools, common-patterns)
— Claude loads only what's relevant to a given request.
Tool reference (high level)
Core (always available)
via_get_annotator_url, via_add_file, via_get_image,
via_get_image_crop, via_get_project, via_list_files,
via_get_annotations, via_add_region, via_update_region,
via_delete_region, via_update_project, via_save_project
Local-model assistance (needs [ai])
via_model_status, via_suggest_regions, via_tighten_region,
via_verify_region, via_grade_annotations, via_classify_scene,
via_ask_model, via_find_similar
IO layer (needs [io] / [ocr])
via_load_document (PDFs), via_read_metadata, via_run_ocr
See skills/annomate/ for the full per-tool guidance,
and docs/design/ for the longer-form design docs.
License
MIT — see LICENSE. VIA is included under its BSD 2-Clause License — see NOTICE.
YOLOE (an optional adapter) is upstream-licensed AGPL-3.0; if you enable
that pipeline make sure that license fits your deployment context. See
src/annomate/models/yoloe.py for the in-tree notice.
Development
python -m venv venv
venv/bin/pip install -e ".[dev]"
venv/bin/pytest
venv/bin/annomate --browser
The VIA HTML is not committed directly; it's generated from the VIA submodule and committed to releases:
git submodule update --init --recursive
cd via/via-3.x.y/scripts && python3 pack.py via image_annotator && cd ../../..
python scripts/build_html.py
python -m build
Metadata
Release files for annomate-mcp 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| annomate_mcp-0.1.0.tar.gz | 178.6 kB | With provenance | Details for annomate_mcp-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| annomate_mcp-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for annomate_mcp-0.1.0-py3-none-any.whl |
Total release size: 344.9 kB
Release files / annomate_mcp-0.1.0.tar.gz
| Download URL | annomate_mcp-0.1.0.tar.gz |
|---|---|
| Size | 178.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0795683bb12c62675399bddc895092c88814d51612d735b9f64f1e0dbdb5d78c
|
|
BLAKE2b-256 checksum How to use checksums |
d168518e960033f4302853323156717ba471ba72c5a51e31f59ccc275a49b10e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 29, 2026.
Transparency logRelease files / annomate_mcp-0.1.0-py3-none-any.whl
| Download URL | annomate_mcp-0.1.0-py3-none-any.whl |
|---|---|
| Size | 166.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0c6e8556c07663ba927e18c03f14a9663e51114ebd048f2471c1370927803486
|
|
BLAKE2b-256 checksum How to use checksums |
644da10e4b06e182fbfd5ae8355b6b4ff8e5593f44c3ea41668b54e043eb51bc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 29, 2026.
Transparency logSecurity for annomate-mcp 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for annomate_mcp-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 29, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.