Histopathology Annotation MCP

Image Annotation, including Histopathology slides; From H&E slide to training data in one conversation: nuclei, outlines and slide QC, exported to QuPath, COCO or YOLO. No GPU needed.

Accuracy: see Benchmark. Your data: see Your data. Changes: see CHANGELOG.

An MCP server for annotating histopathology and other images in Image Label, a free annotation tool. It lets Claude and other AI assistants upload images, outline every nucleus in an H&E slide automatically (StarDist), trace regions from rough boxes (Segment Anything, or MedSAM for medical images), check slide quality, normalize stain colours, clean up annotations, and export training data as COCO JSON, YOLO labels, YOLO classes or QuPath GeoJSON. A LangGraph agent is included.

Try it in the browser: https://agent.sachin-kale.com/#annotate: upload a slide and annotate it, detect nuclei, check its quality or get an AI interpretation, with no setup.

Published as label-image-mcp (PyPI) and io.github.sachinbkale27/label-image-mcp (MCP Registry).

Use cases and setup guide: https://labels.sachin-kale.com/mcp

LangGraph agent (label-image-agent)      Claude Desktop / Claude Code / any MCP client
              │  langchain-mcp-adapters             │
              └──────────────┬──────────────────────┘
                             ▼
                    label-image-mcp (stdio or HTTP)
                             │
                             ▼
                 Image Label API  (/api/images ...)

Account

The tools need an Image Label account (request one). They see, change and upload only images saved to that account; without one they see no images. Work kept only in your browser is not visible to them.

Install

Claude Desktop: download label-image-mcp.mcpb from the latest release, open it, and enter your username and password when asked.

Claude Code:

claude mcp add label-image -e LABEL_USERNAME=... -e LABEL_PASSWORD=... -- uvx label-image-mcp

Any MCP client (needs uv):

{
  "mcpServers": {
    "label-image": {
      "command": "uvx",
      "args": ["label-image-mcp"],
      "env": { "LABEL_USERNAME": "...", "LABEL_PASSWORD": "..." }
    }
  }
}

Benchmark

detect_nuclei checked against expert outlines on NuInsSeg (CC BY 4.0): H&E images of 31 human and mouse tissues, 30,698 hand-segmented nuclei. The detection threshold was chosen on the even-numbered images of each tissue; these results are on the other 332 images (15,701 nuclei), which were not used for any choice. A nucleus counts as found when its outline overlaps the expert's by IoU ≥ 0.5.

• Better on human tissue (F1 0.67) than mouse (0.50 to 0.53). Best on pancreas, cerebellum, placenta and tongue (0.73 to 0.81); weakest on umbilical cord, brain, mouse heart and bone (0.13 to 0.34), where nuclei are sparse, thin or faint. Per-tissue results are in benchmarks/results.
• The model is StarDist's pretrained H&E model, not trained on NuInsSeg: models trained on a dataset score higher on it. Read these as what to expect on new images, and review outlines before training on them.
• Reproduce: benchmarks/nuinsseg.py runs the same method locally.
• Cell types (classify_cells) are not benchmarked here yet: NuInsSeg has no cell-type labels. HoVer-Net's published results on PanNuke are the reference until then.

Your data

What happens to images you upload through this server (hosted on AWS in us-east-1):

• Private to your account. Only you can list, read or change your images; the storage bucket blocks all public access, and viewing links are signed and expire after 24 hours.
• Deleted after 90 days, automatically: files, slides, annotations and analysis results. Deleting an image removes it straight away.
• Encrypted at rest (AES-256) and in transit (HTTPS).
• Not used to train models, and not sent to other AI services by this server: StarDist, Segment Anything and MedSAM run on the service itself. Your AI assistant sees what its tools return (view_image returns the picture), under that assistant's own terms.
• Usage counts. Each call to the service names the AI client (for example claude-ai/0.1.0, from the MCP handshake) and the server version, so usage can be counted by client; the account is logged only as a hash. Nothing from your conversation is sent.
• No patient identifiers, please. Remove names, record numbers and slide labels (crop the label area or use a de-identified copy) before uploading. Results are for research and education, not diagnosis.

Agent skill

skills/label-image/SKILL.md teaches the agent when and how to use these tools: which annotation types end up in exports, cleaning up labels safely, and merging per-image exports into one COCO or YOLO dataset with consistent class ids (bundled scripts/merge_exports.py). It follows the open SKILL.md format, so it works in Claude Code and other agents that support skills.

# Claude Code: install for all your projects (or copy into a project's .claude/skills/)
git clone --depth 1 https://github.com/sachinbkale27/label-image-mcp /tmp/label-image-mcp
mkdir -p ~/.claude/skills && cp -r /tmp/label-image-mcp/skills/label-image ~/.claude/skills/

Configuration

Tools

Every tool is marked read-only or destructive (replace_annotations, remove_annotations), so clients can ask before the destructive ones. The server also offers three prompts: Analyse a histology slide, Outline objects in an image and Build a training dataset.

Whole slides

After the analysis the slide's image in the library is its overview, with the analysed tiles marked (the three densest in red; with full coverage, a density heatmap and the ten densest tiles), so view_image and the web app show it. The three densest tiles are saved as images of their own, with their nuclei outlined, for a close look or further annotation. Every nucleus is in a QuPath GeoJSON in slide coordinates: open the slide in QuPath, then File > Import objects. Slides are read with tifffile (Aperio SVS including JPEG 2000, Hamamatsu NDPI, pyramidal TIFF and OME-TIFF), decoding only the tiles needed. Other images are up to 20 MB.

There is deliberately no tool for deleting images.

segment_objects calls the API's POST /images/{id}/segment, which runs SAM (facebook/sam-vit-base) or MedSAM (wanglab/medsam-vit-base) in a Lambda container (backend/segment). After a quiet spell the model takes a while to start; the client retries then. Results are for research and dataset building, not diagnosis.

The histopathology tools call POST /images/{id}/pathology, which runs StarDist (BSD-3; its pretrained 2D_versatile_he model was trained on public H&E datasets, some under non-commercial terms, so check them before commercial use), torchstain (MIT) and OpenCV quality measures in a second Lambda container (backend/pathology). Microscope photos come at many magnifications, so detect_nuclei also tries the image enlarged 2x and 3x and keeps the scale that finds the most nuclei.

HTTP mode

label-image-mcp --http [--host 127.0.0.1] [--port 8765] serves Streamable HTTP at http://HOST:PORT/mcp, so several clients can share one running server:

claude mcp add --transport http label-image http://127.0.0.1:8765/mcp

Hosted mode

label-image-mcp --http --hosted --host 0.0.0.0 is for running the server remotely, for example on AWS Lambda behind the Lambda Web Adapter. It keeps no session state, answers with JSON, and requires Authorization: Bearer <token> on every request; with no token configured it refuses all requests.

LangGraph agent

A terminal chat agent that uses these tools. The graph is START → agent ⇄ tools, with conversation memory kept for the session.

uv tool install "label-image-mcp[agent]"
export ANTHROPIC_API_KEY=... LABEL_USERNAME=... LABEL_PASSWORD=...
label-image-agent

Development

uv sync --extra agent
uv run pytest
npx @modelcontextprotocol/inspector uv run label-image-mcp

In the Image Label app repo, ./start.sh runs this server in HTTP mode next to the web app. It reads credentials from mcp-server/.env. Releases are described in PUBLISHING.md.

License

MIT