perception-mcp

Perception & verification MCP server for AI agents — the measuring instruments a text-and-image model lacks. Where a model can only look at media, this server lets it measure, compare and verify: local recognized algorithms (no cloud APIs), compact JSON answers, and PNG renderings the model can look at. The successor and superset of audio-dsp-mcp.

The idea

An agent that generates or edits media works blind: it can't hear the podcast it produced, can't tell whether two screenshots differ by a real regression or antialiasing noise, can't read the exact color of a button. Every tool here follows one formula:

opaque file → compact JSON with numbers → optional PNG for the model's vision → an honest "not sure" when the signal isn't there.

Tools

media_assert — declarative QC (the headline)

One call turns the whole server into a test harness for media:

media_assert("episode.mp3", expect={
  "duration_s": [1790, 1810],
  "lufs": [-17, -13],
  "no_clipping": true,
  "no_silence_longer_s": 2.0
})

→ {"passed": false, "summary": "3 of 4 checks passed (1 failed)", "checks": [{"check": "lufs", "status": "fail", "actual": -9.2, ...}]}

Checks run only for what you ask, files are decoded once, an unknown check name returns the full supported list (instant self-correction), and a check that doesn't apply to the file type is reported skipped, not failed. Generate → assert → regenerate.

Video files get their own keys (needs ffmpeg): resolution, fps, has_audio, no_black_longer_s, no_freeze_longer_s, no_silence_longer_s, min_sharpness, duration_s.

view_media — SEE any image or PDF page in one call

Chat clients cap a tool result at ~1 MB and give the model no direct file access, so "look at this local photo" used to mean stitching zoomed fragments. view_media renders the whole image (or one PDF page) downscaled to what a vision model can actually use and compressed to always fit the cap — and every image any tool returns now honors the same byte budget (PNG, falling back to JPEG for busy frames). The response reports the applied scale, so follow-up zoom_grid calls can address original pixel coordinates.

find_media — from "the photo I just sent you" to a real path

A chat attachment never reaches an MCP server: the protocol has no client→server file channel, so the model gets rendered pixels — for images not even a filename. But the original file almost always exists on the user's machine (they attached it from there). find_media closes the gap: it searches the user's home (Spotlight on macOS) and the standard folders — Downloads, Desktop, Documents, Pictures, Movies, Music — by type, recency and name fragment, newest first. So instead of demanding paths, the model asks "IMG_4021.jpg, 4 min ago, 3.1 MB — this one?". Only names, sizes and dates are returned; contents are never read. Extra folders via the PERCEPTION_SCAN_DIRS env var.

image ( [image] extra )

• compare_images — perceptual pixel diff (pixelmatch YIQ metric)
  • SSIM structural score, with a global-shift compensation via phase correlation (a 1-px scroll no longer lights up the whole diff), antialiasing separation, and changed-pixel clusters with exact bounding boxes. Returns an A | B | highlight PNG panel. Tolerance modes: strict / normal / layout.
• measure_image — exact pixel colors (hex/rgb/CSS name) at given points, pixel distances between points, dominant palette (k-means in CIELab).
• zoom_grid — a microscope: magnify a region with a labeled coordinate grid in absolute source coordinates, so tiny text, 1-px borders and exact positions become visible and referenceable.

ocr ( [ocr] extra )

• extract_text_boxes — printed text with exact pixel coordinates per line (RapidOCR: PaddleOCR's DBNet + CRNN via ONNX, models ship in the wheel — no downloads, no cloud).
• prepare_text_tiles — the small-print / handwriting reader. A vision LLM fails on tiny text because tokenization gives each glyph too few pixels; this tool fixes that with classic document preprocessing — CLAHE contrast, projection-profile deskew, Sauvola ink mask, line segmentation — and returns each line as a big magnified labeled strip for the calling model to read itself. No recognition model in the loop, so it works for any script and any handwriting the calling model can read.
• check_contrast — WCAG 2.x audit of a screenshot: per text line, exact W3C contrast ratio + AA/AAA pass/fail.
• media_assert keys: contains_text, min_contrast_ratio.

audio ( [audio] extra — the full audio-dsp-mcp toolset )

detect_pitch, estimate_tempo, detect_key (+ Camelot), analyze_loudness (LUFS, ITU-R BS.1770), detect_silence, analyze_audio, compare_audio, render_spectrogram, render_waveform, describe_audio, get_metadata; with the [youtube] extra get_youtube_transcript, with [transcribe] local Whisper transcribe_audio.

video ( no extra — just ffmpeg on PATH )

• describe_video — the whole video in one call: stream facts, shot structure (cuts, avg shot length), audio profile, speech transcript (with [transcribe]), and a contact sheet with a timestamped frame per scene.
• get_frames — SEE the footage: exact timestamps, an even sweep (every), or a storyboard of one moment (around=[t, span, fps]); many frames pack into one timecoded contact sheet.
• check_video — QC with exact timestamps: black stretches, frozen picture, silent audio (ffmpeg blackdetect / freezedetect / silencedetect), soft focus (Laplacian variance).
• video_dynamics — pacing on one timeline: cut rhythm, motion energy, EBU R128 momentary loudness, hook metrics for the opening seconds (first cut, cold-start flags) + a chart image.
• video_heatmap — WHERE things happen: motion accumulation and spectral-residual saliency (Hou & Zhang) drawn over a real frame, with a 3x3 grid breakdown for crop/caption decisions.
• compare_videos — scene-level diff of two cuts (Needleman-Wunsch alignment + SSIM): identical / changed / added / removed scenes.
• detect_scenes — shot boundaries as timestamps.

Every analysis tool takes offset_s / duration_s, so any part of a long file is reachable; reported timestamps are absolute.

Sources: every tool takes a local path, an http(s) URL (25 MB cap), or a base64 data URI. Share links resolve automatically — Google Drive ("Anyone with the link"), Dropbox and tmpfiles.org links are rewritten to direct downloads, and with [youtube], video-platform page URLs resolve via yt-dlp. For chat clients where attachments never reach the server (a protocol-level limit), the served MCP instructions teach the model an escalation ladder: client-provided path → find_media on the user's disk → ask for a path (with the copy-path gesture) → ask for a share link. All failures come back as an "error" field — the server never crashes the conversation.

docs ( [docs] extra )

• compare_pdf — page-level visual diff of two PDFs (pypdfium2 render → the same perceptual diff as screenshots): layout shifts, swapped images, font substitution; diff panels for changed pages.
• check_pdf — will it open everywhere: structural validation (pikepdf/qpdf), encryption, non-embedded fonts, pages that crash rendering, blank pages.

fonts ( [fonts] extra )

• render_font_specimen — SEE a typeface and verify it: rendered specimen sheet (pangrams for every script the font covers) + exact cmap coverage ("cyrillic": "complete" / "partial (74%)" / "none"), metrics, variable axes.

mesh ( [mesh] extra )

• render_model — seven orthographic views of a 3D model on one sheet (GLB/GLTF/STL/OBJ/PLY), pure-CPU rendering, works headless.
• check_mesh — printability QC: watertightness, winding, degenerate faces, volume, extents, printable verdict with reasons.
• compare_mesh — geometric diff: ICP alignment + two-sided point-to-surface distances (chamfer / Hausdorff) + overlay render.

Install

Core is lean; add what you need: [image], [ocr], [audio], [youtube], [transcribe], [docs], [fonts], [mesh], or [all].

Claude Code

claude mcp add perception -- uvx --from "perception-mcp[all]" perception-mcp

Claude Desktop / any MCP client

{
  "mcpServers": {
    "perception": {
      "command": "uvx",
      "args": ["--from", "perception-mcp[all]", "perception-mcp"]
    }
  }
}

Roadmap

MCP sampling mode, A/V sync detection, mesh wall-thickness analysis.

License

MIT