videre-mcp

MCP server that bridges vision models to text-only coding models using Florence-2.

Non-vision LLMs can't see images — videre-mcp fixes that. It loads a Florence-2 vision model locally and exposes six MCP tools that convert images (including SVGs) and screenshots into structured text descriptions that any text-based model can consume.

Screenshot tool → videre-mcp (Florence-2) → Text description → Coding model

Installation

pip install videre-mcp

Or with uv:

uv pip install videre-mcp

Requires Python 3.11+ and ~300MB disk space for the Florence-2-base model weights (downloaded automatically on first use).

Usage

Add to your OpenCode configuration:

{
  "mcpServers": {
    "videre-mcp": {
      "command": "videre-mcp"
    }
  }
}

Or run directly:

videre-mcp
# or
python -m videre_mcp

Model Modes

All Florence-2 tools support a model_mode parameter to balance speed and quality:

• "fast" (default) — Uses Florence-2-base. Fast, lightweight, runs on CPU/GPU.
• "deep" — Uses MiniCPM-V 4.6. Significantly higher quality for complex visual reasoning.
  • Requires: pip install videre-mcp[deep]
  • Hardware: ~8GB VRAM recommended.

Tools

describe_image

Generate a natural language description of an image.

Parameters:

• image_path (str) — Path to the image file (supports PNG, JPEG, SVG)
• detail_level (str, optional) — "normal" (default) for brief caption, "high" for detailed description
• model_mode (str, optional) — "fast" (default) or "deep"

Example:

result = describe_image("/path/to/photo.png", detail_level="high")
# Returns:
# {
#   "description": "A sunlit meadow with wildflowers in bloom...",
#   "model": "Florence-2-base",
#   "prompt_used": "<MORE_DETAILED_CAPTION>"
# }

ocr_image

Extract text from an image using optical character recognition.

Parameters:

• image_path (str) — Path to the image file (supports PNG, JPEG, SVG)
• detail_level (str, optional) — "normal" (default) for plain text, "high" for text with bounding regions
• model_mode (str, optional) — "fast" (default) or "deep"

Example:

result = ocr_image("/path/to/document.png", detail_level="high")
# Returns:
# {
#   "text": "Invoice Number 12345",
#   "regions": [
#     {"label": "Invoice Number 12345", "bbox": [10, 20, 30, 40, 50, 60, 70, 80]}
#   ]
# }

describe_screenshot

Describe UI regions in a screenshot — designed for coding agents that need to understand screen layouts.

Parameters:

• image_path (str) — Path to the screenshot file (supports PNG, JPEG, SVG)
• detail_level (str, optional) — "normal" (default) for dense region captions, "high" for per-region descriptions
• model_mode (str, optional) — "fast" (default) or "deep"

Example:

result = describe_screenshot("/path/to/screenshot.png")
# Returns:
# {
#   "regions": [
#     {"bbox": [10, 20, 30, 40], "label": "search bar"},
#     {"bbox": [100, 200, 300, 250], "label": "submit button"}
#   ],
#   "model": "Florence-2-base"
# }

take_screenshot

Capture a screenshot and optionally describe it using Florence-2. Supports multi-monitor setups via the monitor parameter.

Parameters:

• output_path (str, optional) — Path to save the screenshot PNG. If None, saves to a temp file.
• monitor (int, optional) — Monitor index: 0 = all monitors combined, 1 = primary, etc. (default: 0)
• describe (bool, optional) — If True, also run describe_screenshot on the captured image (default: True)
• model_mode (str, optional) — "fast" (default) or "deep"

Example:

result = take_screenshot(monitor=1, describe=True)
# Returns:
# {
#   "path": "/tmp/tmpxxxxxx.png",
#   "width": 1920,
#   "height": 1080,
#   "monitor": 1,
#   "regions": [
#     {"label": "search bar", "bbox": [10, 20, 30, 40]},
#     ...
#   ]
# }

ocr_paddle

Dedicated OCR using PaddleOCR (100+ languages, PP-OCRv6). Superior accuracy for multi-language documents.

Parameters:

• image_path (str) — Path to the image file
• language (str, optional) — Language code: "en", "ch", "japan", "korean", "french", "german", "spanish", "arabic", "multilingual", etc. (default: "en")
• detail_level (str, optional) — "normal" for plain text, "high" for text with bounding boxes and confidence scores
• use_angle_cls (bool, optional) — Use angle classification to correct rotated text (default: True)

Requires: pip install videre-mcp[paddle]

Example:

result = ocr_paddle("/path/to/document.png", language="multilingual", detail_level="high")
# Returns:
# {
#   "text": "Invoice Number 12345\nDate: 2024-01-15",
#   "regions": [
#     {"text": "Invoice Number 12345", "bbox": [...], "confidence": 0.98}
#   ]
# }

parse_document

Parse documents (PDF, DOCX, PPTX, HTML, MD) into structured output using IBM Docling. Extracts text, tables, charts, formulas, and code blocks.

Parameters:

• file_path (str) — Path to the document file
• output_format (str, optional) — "markdown" (default), "json", "text", or "html"
• extract_tables (bool, optional) — Extract and structure tables (default: True)
• extract_images (bool, optional) — Extract embedded images (default: False)

Requires: pip install videre-mcp[docling]

Example:

result = parse_document("/path/to/report.pdf", output_format="markdown", extract_tables=True)
# Returns:
# {
#   "content": "# Report Title\n\n...",
#   "metadata": {"title": "...", "author": "...", "pages": 10},
#   "tables": [...]
# }

Optional Dependencies

Extra	Package	Enables
[deep]	accelerate, bitsandbytes	MiniCPM-V 4.6 deep mode (~8GB VRAM)
[docling]	docling>=2.0.0	Document parsing (PDF, DOCX, PPTX, HTML, MD)
[paddle]	paddleocr>=2.8.0	PaddleOCR (100+ languages)
[optimize]	dspy-ai>=2.5.0	DSPy prompt optimization CLI

Install with: pip install videre-mcp[deep,docling]

Requirements

• Python 3.11+
• ~300MB disk for model weights (auto-downloaded on first inference)
• Works on CPU; GPU (CUDA) is auto-detected and used if available

Continuous Integration

The Florence-2 slow tests (real model load + inference) run on a nightly schedule via GitHub Actions. See .github/workflows/slow-tests.yml.

License

MIT — see LICENSE.

Third-party licenses

This package vendors a patched copy of Microsoft's Florence-2 processor (src/videre_mcp/_vendor/processing_florence2.py) under Microsoft's MIT license. See src/videre_mcp/_vendor/LICENSE-Microsoft-Florence-2.