Cinematlas

Search inside video: the frames and the spoken words, not just titles and tags.

Cinematlas turns a video (a YouTube link, a URL to a file, or an upload) into searchable scenes in MongoDB Atlas. It detects scene cuts, picks a keyframe for each scene, transcribes the dialogue, and embeds both with Voyage AI. You can then ask a question and get a deep link to the scene that answers it.

pip install "cinematlas[whisper]"
cinematlas setup
cinematlas ingest "www.b.com/keynote.mp4"
cinematlas search "when do they talk about pricing?"

Built on PySceneDetect, yt-dlp, faster-whisper, Voyage AI and MongoDB Atlas Vector Search, with or without Atlas autoEmbed. The engineering story is in blog.md.

How it works

 URL / youtube / upload / stdin
            │
            ▼
   ┌──────────────────┐   yt-dlp (H.264 + audio)  or  chunked upload → temp file
   │  fetch + guard   │   SSRF guard · http(s) only · size cap · credential redaction
   └────────┬─────────┘
            ▼
   ┌──────────────────┐        ┌───────────────────────────┐
   │  PySceneDetect   │        │ ffmpeg → 16 kHz mono MP3  │
   │  cuts + 30s cap  │        │ faster-whisper + VAD      │
   └────────┬─────────┘        └─────────────┬─────────────┘
            ▼                                │  overlap-aware
   middle-frame keyframes                    │  scene assignment
            ▼                                ▼
   voyage-multimodal-3.5            transcript per scene
   (image vectors)                  ├─ Atlas autoEmbed (voyage-4), or
            │                       └─ client-side voyage-4 vectors
            ▼                                ▼
   ┌──────────────────────────────────────────────────────┐
   │ MongoDB Atlas: one document per scene                 │
   │ $vectorSearch: visual_embedding · transcript          │
   └──────────────────────────────────────────────────────┘

Install

ffmpeg must be on your PATH.

brew install ffmpeg                  # macOS  (Ubuntu: sudo apt-get install -y ffmpeg)

pip install "cinematlas[whisper]"    # recommended: + local speech-to-text (faster-whisper, no PyTorch)
pip install cinematlas               # core only (use OPENAI_API_KEY for Whisper API transcription)
pip install "cinematlas[all]"        # + S3 keyframe uploads + OpenAI Whisper API

export MONGODB_URI="mongodb+srv://<user>:<password>@cluster.mongodb.net/"   # MDB_URI also accepted
export VOYAGE_API_KEY="pa-..."
export OPENAI_API_KEY="sk-..."                    # optional: Whisper API instead of local
export S3_BUCKET_NAME="my-cinematlas-keyframes"   # optional: keyframe thumbnails

Quickstart

from cinematlas import Cinematlas

engine = Cinematlas()           # reads MONGODB_URI / MDB_URI and VOYAGE_API_KEY
engine.ensure_indexes()         # one time; idempotent; returns "autoembed" or "client"

# From a URL: YouTube, a direct file link, or anything yt-dlp supports
engine.ingest_video("https://www.youtube.com/watch?v=5NhYvbMdbBU")
engine.ingest_video("www.b.com/v.mp4")          # scheme-less works; fetched over https

# From a file or an upload
engine.ingest_file("talk.mp4")

# Search what was said
for hit in engine.search_transcript("how loud is a sonic boom?", top_k=3):
    print(f"{hit['score']:.3f}  {hit['timestamp_start']}s  {hit['deep_link']}\n   {hit['transcript']}")

# Search what was shown (text -> keyframes)
engine.search_visual_vector("an aircraft on a runway", top_k=3)

Every search takes video_id= to scope results to one video.

Ingestion sources

Remote URLs

ingest_video() accepts YouTube links, direct file URLs (https://cdn.example.com/v.mp4, presigned S3/GCS/Azure URLs) and anything else yt-dlp supports. It's designed so you can safely pass URLs supplied by your users:

Redirects followed by yt-dlp aren't re-checked. If you accept arbitrary URLs from the public internet, also route downloads through an egress proxy.

File uploads

ingest_file() accepts a path, bytes, a binary file object, a FastAPI UploadFile, or a Flask/Werkzeug FileStorage.

# FastAPI
@app.post("/videos")
def upload(file: UploadFile):
    return {"scenes": engine.ingest_file(file)}   # sync def: FastAPI runs it in a threadpool

# Flask
@app.post("/videos")
def upload():
    return {"scenes": engine.ingest_file(request.files["video"])}

• Uploads are streamed to disk in 1 MiB chunks, so memory stays flat for large files.
• Without a video_id, the ID is the content hash (file_<sha256[:16]>). Uploading the same file again replaces it instead of duplicating it.
• Empty or undecodable files are rejected with IngestionError.

CLI

cinematlas setup                                     # {"transcript_mode": "autoembed"}
cinematlas ingest "https://www.youtube.com/watch?v=5NhYvbMdbBU"
cinematlas ingest www.b.com/v.mp4                    # remote file
cinematlas ingest ./talk.mp4 --video-id talk-01      # local file
curl -sL https://b.com/v.mp4 | cinematlas ingest - --filename v.mp4    # stdin upload
cinematlas search "how loud is a sonic boom" -k 3    # JSON lines
cinematlas search "an aircraft in the sky" --by visual

Options: --uri, --db, --collection, --transcript-mode, -v. Exit code 1 on errors, with a message instead of a traceback.

With or without Atlas autoEmbed

Transcripts can be searched in two ways, through one call (search_transcript):

• autoembed: Atlas Automated Embedding embeds transcript and the query text itself with voyage-4 (Atlas feature, currently in Preview).
• client: Cinematlas stores a voyage-4 transcript_embedding and embeds queries itself. This works on any deployment with Vector Search, including mongodb/mongodb-atlas-local.

ensure_indexes() tries autoEmbed first and falls back to client if the cluster rejects it. If a transcript index already exists, it keeps using that mode and never switches modes on existing data. To force a mode, pass transcript_mode="autoembed" or "client".

Reliability

• Replacing a video never loses it. A re-ingest inserts the new scenes under a fresh ingest_id and only then deletes older versions. After a failure, the previous version stays searchable next to a FAILED tombstone.
• Vectors stay aligned with scenes. Scenes without a usable keyframe get None, and every other vector stays on its own scene. Short API responses are treated as failures, not shifted.
• Speech goes to the right scene. Timestamps come from voice-activity-trimmed Whisper output. A segment belongs to every scene it overlaps by ≥ 0.5s, and always to the scene it overlaps most, so normal timestamp drift doesn't copy sentences across cuts.
• Scenes have a maximum length. Footage without hard cuts is split into scenes of at most 30s (max_scene_seconds).
• Things degrade instead of crashing. Voyage and download calls retry with backoff. A missing audio track means visual-only indexing, and a failed transcription means an empty transcript.
• Documents record provenance. embedding_models stores which models produced the vectors, for future re-embedding migrations.

Configuration

Every external client can be passed in (mongo_client=, voyage_client=, s3_client=, openai_client=), which is how the test suite runs offline.

Development & testing

uv sync
uv run pytest -m "not integration and not media"   # unit: offline, ~8s
uv run pytest -m media                              # real ffmpeg / yt-dlp / Whisper on the fixture, offline
uv run pytest -m integration                        # live Atlas (autoEmbed) + Docker Atlas Local (client)
uv run pytest                                       # everything

No test contacts YouTube. End-to-end runs use tests/fixtures/x59_quiet_crew.mp4 (847 KiB, NASA, public domain): three sentence-aligned topics joined by hard cuts, which gives the tests ground truth for scene alignment. URL ingestion is served from a loopback HTTP server. The fixture is checked by SHA-256 and rebuilt with uv run python tests/fixtures/build_fixture.py.

Integration env (.env): MDB_URI or MONGODB_URI, VOYAGE_API_KEY, and optionally VOYAGE_MODEL. The Atlas Local container starts and stops by itself on port 27028; set ATLAS_LOCAL_URI to reuse one. Tests write to cinematlas_ci.scenes under a unique video_id and clean up after themselves.

Releasing

# bump version in pyproject.toml, then:
rm -rf dist && uv build && uv publish   # uses UV_PUBLISH_TOKEN

Roadmap

• Async / queued ingestion (Celery, Temporal)
• OCR of on-screen text into scene documents
• Word-level timestamps for sentences that straddle a cut
• AdaptiveDetector option for dissolve-heavy footage

License

MIT. Test fixture: NASA media, public domain (see build_fixture.py).