Histopathology Annotation MCP

An MCP server for annotating histopathology and other images in Image Label, a free annotation tool. It lets Claude and other AI assistants upload images, outline every nucleus in an H&E slide automatically (StarDist), trace regions from rough boxes (Segment Anything, or MedSAM for medical images), check slide quality, normalize stain colours, clean up annotations, and export COCO, YOLO or QuPath GeoJSON. A LangGraph agent is included.

Try it in the browser: https://agent.sachin-kale.com/#annotate: upload a slide and annotate it, detect nuclei, check its quality or get an AI interpretation, with no setup.

Published as label-image-mcp (PyPI) and io.github.sachinbkale27/label-image-mcp (MCP Registry).

Use cases and setup guide: https://labels.sachin-kale.com/mcp

LangGraph agent (label-image-agent)      Claude Desktop / Claude Code / any MCP client
              │  langchain-mcp-adapters             │
              └──────────────┬──────────────────────┘
                             ▼
                    label-image-mcp (stdio or HTTP)
                             │
                             ▼
                 Image Label API  (/api/images ...)

Account

The tools need an Image Label account (request one). They see, change and upload only images saved to that account; without one they see no images. Work kept only in your browser is not visible to them.

Install

Claude Desktop: download label-image-mcp.mcpb from the latest release, open it, and enter your username and password when asked.

Claude Code:

claude mcp add label-image -e LABEL_USERNAME=... -e LABEL_PASSWORD=... -- uvx label-image-mcp

Any MCP client (needs uv):

{
  "mcpServers": {
    "label-image": {
      "command": "uvx",
      "args": ["label-image-mcp"],
      "env": { "LABEL_USERNAME": "...", "LABEL_PASSWORD": "..." }
    }
  }
}

Agent skill

skills/label-image/SKILL.md teaches the agent when and how to use these tools: which annotation types end up in exports, cleaning up labels safely, and merging per-image exports into one COCO or YOLO dataset with consistent class ids (bundled scripts/merge_exports.py). It follows the open SKILL.md format, so it works in Claude Code and other agents that support skills.

# Claude Code: install for all your projects (or copy into a project's .claude/skills/)
git clone --depth 1 https://github.com/sachinbkale27/label-image-mcp /tmp/label-image-mcp
mkdir -p ~/.claude/skills && cp -r /tmp/label-image-mcp/skills/label-image ~/.claude/skills/

Configuration

Tools

There is deliberately no tool for deleting images.

segment_objects calls the API's POST /images/{id}/segment, which runs SAM (facebook/sam-vit-base) or MedSAM (wanglab/medsam-vit-base) in a Lambda container (backend/segment). After a quiet spell the model takes a while to start; the client retries then. Results are for research and dataset building, not diagnosis.

The histopathology tools call POST /images/{id}/pathology, which runs StarDist (BSD-3; its pretrained 2D_versatile_he model was trained on public H&E datasets, some under non-commercial terms, so check them before commercial use), torchstain (MIT) and OpenCV quality measures in a second Lambda container (backend/pathology). Microscope photos come at many magnifications, so detect_nuclei also tries the image enlarged 2x and 3x and keeps the scale that finds the most nuclei.

HTTP mode

label-image-mcp --http [--host 127.0.0.1] [--port 8765] serves Streamable HTTP at http://HOST:PORT/mcp, so several clients can share one running server:

claude mcp add --transport http label-image http://127.0.0.1:8765/mcp

Hosted mode

label-image-mcp --http --hosted --host 0.0.0.0 is for running the server remotely, for example on AWS Lambda behind the Lambda Web Adapter. It keeps no session state, answers with JSON, and requires Authorization: Bearer <token> on every request; with no token configured it refuses all requests.

LangGraph agent

A terminal chat agent that uses these tools. The graph is START → agent ⇄ tools, with conversation memory kept for the session.

uv tool install "label-image-mcp[agent]"
export ANTHROPIC_API_KEY=... LABEL_USERNAME=... LABEL_PASSWORD=...
label-image-agent

Development

uv sync --extra agent
uv run pytest
npx @modelcontextprotocol/inspector uv run label-image-mcp

In the Image Label app repo, ./start.sh runs this server in HTTP mode next to the web app. It reads credentials from mcp-server/.env. Releases are described in PUBLISHING.md.

License

MIT