Skip to main content

Percept Vision 🎥 — store videos in Redis, then ask questions about them (MCP)

The MP4 bytes live in object storage. The understanding lives in Redis — and that's what you query.

pip install percept-vision-plugin gives any agent the ability to ingest a video, run computer vision over it (OpenCV frame sampling + YOLO object detection + CLIP embeddings), and store the result in Redis — so you can find the exact moment something happens, list what objects appear, and ask natural-language questions about your video library. Exposed over the Model Context Protocol (MCP).

It's the multimodal layer of Percept Context — video becomes first-class, searchable nodes alongside your context graph, all in one Redis.


Why this exists

Redis's vector stack is text-only — every shipping RedisVL vectorizer is a …Text… class; there's no image/video vectorizer, no frame-level search, no media model. And per Redis's own guidance, you shouldn't put MP4 bytes in Redis. Percept Vision does it the right way:

Concern How
Video bytes Supabase Storage → a public URL (Redis never holds the file)
Frame understanding OpenCV samples frames; YOLO detects objects; CLIP embeds them
Searchable index RedisVL vector index of CLIP frame vectors (percept_frames)
"Find the moment" CLIP text→image search returns timestamped, deep-linkable moments
"What's in it" YOLO detections aggregated per video
"Ask about it" retrieved frames + detections → optional LLM answer with timestamps

Pipeline

  ingest_video(path|url)
        │
        ├─►  Supabase Storage  ──►  public MP4 URL ─────────────┐
        │                                                       │
        └─►  OpenCV sample frames                               │
                 ├─►  YOLO  ──►  objects per frame              ▼
                 └─►  CLIP  ──►  512-d vector ──►  R E D I S  (percept_frames index)
                                                   pv:video:{id}  +  pv:frame:{id}
        ┌───────────────────────────────────────────────────────────────────────┐
   search_moments("a person holding a phone")  → CLIP text vector → top frames
   ask_video("what happens at the end?")       → frames + detections → answer
   list_video_objects(id)                      → {person: 12, laptop: 4, ...}

Install

Requires Python ≥ 3.10, a Redis with the Search module (Redis Stack / Redis 8 / Redis Cloud), and a Supabase project. First run downloads YOLOv8n (~6 MB) and CLIP ViT-B-32 (~600 MB).

pip install percept-vision-plugin

Configure (.env or env vars):

SUPABASE_URL=https://YOUR_REF.supabase.co
SUPABASE_SERVICE_KEY=your_service_role_key   # bypasses RLS for uploads
SUPABASE_BUCKET=percept-videos               # public bucket
REDIS_URL=redis://localhost:6379
REDIS_PROTOCOL=2

Try it:

python examples/quickstart.py /path/to/video.mp4

Register with Claude Code

claude mcp add percept-vision --scope local \
  --env SUPABASE_URL=... --env SUPABASE_SERVICE_KEY=... --env SUPABASE_BUCKET=percept-videos \
  --env REDIS_URL=redis://localhost:6379 --env REDIS_PROTOCOL=2 \
  -- percept-vision

Tools

Tool What it does
ingest_video(source, video_id?) Upload + CV-analyze a video (path or URL) into Redis.
search_moments(query, k?, video_id?) Find moments matching a phrase → timestamps + deep links + objects.
ask_video(question, video_id?, k?) Q&A over retrieved frames (LLM answer if ANTHROPIC_API_KEY set).
list_video_objects(video_id) YOLO object counts for a video.
list_videos() All ingested videos.
vision_stats() Redis health + video count.

Configuration reference

Env var Default Purpose
SUPABASE_URL / SUPABASE_SERVICE_KEY – (required) Object storage for MP4s.
SUPABASE_BUCKET percept-videos Public bucket name.
REDIS_URL / REDIS_PROTOCOL redis://localhost:6379 / 2 Redis with Search.
PERCEPT_YOLO_MODEL yolov8n.pt Ultralytics model.
PERCEPT_CLIP_MODEL clip-ViT-B-32 CLIP model (image+text).
PERCEPT_FRAME_INTERVAL 1.0 Seconds between sampled frames.
PERCEPT_MAX_FRAMES 60 Cap on frames per video.
ANTHROPIC_API_KEY – Enables written answers in ask_video.

License

MIT

Metadata

Release files for percept-vision-plugin 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for percept-vision-plugin 0.1.0
File Size Uploaded Provenance Actions
percept_vision_plugin-0.1.0.tar.gz 11.7 kB Without provenance Details for percept_vision_plugin-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for percept-vision-plugin 0.1.0
File Interpreter ABI Platform Actions
percept_vision_plugin-0.1.0-py3-none-any.whl Python 3 none any Details for percept_vision_plugin-0.1.0-py3-none-any.whl

Total release size: 26.3 kB

Release files / percept_vision_plugin-0.1.0.tar.gz

Download URL percept_vision_plugin-0.1.0.tar.gz
Size 11.7 kB
Tags Source
SHA-256 checksum
How to use checksums
8c2a11f7943dd8d529f98b41421096bb415fbdff5d4845f163c8c1f5af6da477
BLAKE2b-256 checksum
How to use checksums
93e7622ecd17ed5f729a0f38bf8836d9a9c08568d2b96806255adbdeaa7920f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.5

Release files / percept_vision_plugin-0.1.0-py3-none-any.whl

Download URL percept_vision_plugin-0.1.0-py3-none-any.whl
Size 14.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
230661018add57f1f086c53b0fc98b273ecf2e368749c60482515c02eab89a73
BLAKE2b-256 checksum
How to use checksums
3c6594c768644bc82776bf9722d9de44f1c239eaf5fb91a2283f8d47e09ac3c5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.5

Security for percept-vision-plugin 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page