Cinematlas

Search inside video, down to the second: what was shown and what was said.

Cinematlas turns videos (YouTube links, file URLs, uploads) into searchable scenes in MongoDB Atlas. Ask a question and get back the scene that answers it, the exact sentence, and a deep link to the second it's said.

pip install "cinematlas[whisper]"
cinematlas doctor                                   # what works, what doesn't, how to fix it
cinematlas setup
cinematlas ingest "www.b.com/keynote.mp4"
cinematlas search "when do they talk about pricing?"

 1. vid_3f2a…#12 @    7:11  Pricing starts at ten dollars a seat.
    https://www.b.com/keynote.mp4#t=431
 2. vid_3f2a…#13 @    7:40  Enterprise plans include SSO and audit logs.
    https://www.b.com/keynote.mp4#t=460

Built on PySceneDetect, yt-dlp, faster-whisper, Voyage AI and MongoDB Atlas (Vector Search, autoEmbed, Atlas Search, $rankFusion, $rerank). How it was built and measured: blog.md · benchmark · production considerations.

How it works

 URL / YouTube / upload / stdin
            │
            ▼
   fetch + guard ─── SSRF guard · http(s) only · size cap · credential redaction
            │
   ┌────────┴──────────────────────────┐
   ▼                                   ▼
 PySceneDetect cuts (≤30s scenes)   ffmpeg → 16 kHz mono → faster-whisper
   │                                   (VAD + word timestamps)
   ▼                                   │ sentences assigned to scenes,
 middle-frame keyframes                │ split at cuts word by word
   │                                   ▼
   ├─ voyage-multimodal-3.5 ──────► visual_embedding   (keyframe)
   ├─ voyage-multimodal-3.5 ──────► scene_embedding    (keyframe + speech, interleaved)
   └─ voyage-4 (autoEmbed/client) ─► transcript vectors · BM25 full-text index
                                       │
                                       ▼
          MongoDB Atlas: one document per scene, timestamped sentences inside

 search() ─► $rankFusion(visual, scene, transcript, text) ─► sentence-level rerank
          ─► results with ranks, relevance, moment and moment_link

Install

ffmpeg must be on your PATH (brew install ffmpeg / apt-get install -y ffmpeg).

pip install "cinematlas[whisper]"    # recommended: + local speech-to-text (faster-whisper, no PyTorch)
pip install cinematlas               # core only (transcribe via OPENAI_API_KEY + [openai])
pip install "cinematlas[all]"        # + S3 keyframe uploads + OpenAI Whisper API

export MONGODB_URI="mongodb+srv://<user>:<password>@cluster.mongodb.net/"   # MDB_URI also accepted
export VOYAGE_API_KEY="pa-..."

Quickstart

from cinematlas import Cinematlas

engine = Cinematlas()        # reads MONGODB_URI / MDB_URI and VOYAGE_API_KEY
engine.ensure_indexes()      # one time, idempotent

engine.ingest("https://www.youtube.com/watch?v=5NhYvbMdbBU")   # YouTube
engine.ingest("www.b.com/v.mp4")                               # any file URL (scheme optional)
engine.ingest("talk.mp4")                                      # local path, bytes, file object, or upload

results = engine.search("how loud is a sonic boom?")
print(results)               # readable table; renders as markdown in Jupyter
results.top.link             # 'https://…#t=34', the second it's said
results.top.text             # 'Sonic booms are about 110 decibels.'
results.top.explain()        # 'rerank #1, transcript #1, scene #2, text #3, visual #9, relevance 0.91, …'

One ingest() takes anything, and one search() answers most questions. Results are plain dicts underneath (json.dumps works), with attribute shortcuts on top.

result = engine.ingest("www.b.com/v.mp4", progress=lambda stage, info: print(stage, info))
# fetched {...} → scenes {'count': 14} → transcribed {...} → embedded {...} → stored {'scenes': 14}
print(result)   # Indexed 14 scenes (12 with speech) as 'vid_3f2a…' in 21.4s [autoembed]

Doctor

$ cinematlas doctor
 ✓ MongoDB                        connected · server 9.0.2 · cinematlas.scenes
 ✓ Transcript search              Atlas autoEmbed (voyage-4, embedded server-side)
 ! Index cinematlas_vector_index  outdated: visual_embedding.quantization: None -> 'scalar'
                                  → engine.ensure_indexes(update=True)  ·  cinematlas setup --update  (in place, no downtime)
 ✓ Index cinematlas_text_index    ready
 ✓ $rankFusion                    hybrid search runs as one native query
 ! $rerank                        $rerank is disabled for this Atlas project; using the Voyage rerank API (same model)
                                  → Atlas UI → Project Settings → enable Native Reranking. Nothing else to change.
 ✓ Voyage AI                      API key works (voyage-multimodal-3.5, voyage-4)
 ✓ Data                           65 scenes across 6 videos
 ✓ ffmpeg                         /opt/homebrew/bin/ffmpeg
 ✓ Speech-to-text                 faster-whisper (small), runs locally

0 problem(s), 2 warning(s). Cinematlas degrades gracefully on warnings.

engine.doctor() returns the same report as an object (.ok, .problems, .to_dict()), and cinematlas doctor --json exits non-zero on failures, so it works as a deploy gate. It's cheap and non-destructive, and it primes the engine: the first search skips capability discovery.

Nothing is fatal that doesn't have to be. If $rankFusion or $rerank isn't available, you get one warning per process saying why (disabled in the project, server too old, not offered on this deployment type) and how to fix it, and search keeps working on an equivalent path with the same results.

Indexes heal in place. ensure_indexes() compares each index with the definition this version recommends, ignoring defaults the server adds. With update=True (or cinematlas setup --update), outdated ones are updated through updateSearchIndex; the old version keeps serving until the new one is built. That's how a 0.1 collection picks up scalar quantization without downtime.

Search

search() fuses up to five ranked lists with Reciprocal Rank Fusion. Every search takes video_id= to scope results to one video, and each source is also available on its own (search_transcript, search_text, search_visual_vector, search_scene_vector), all returning the same SearchResults.

Default weights (visual 0.25 · scene 1 · transcript 1 · text 1 · rerank 2) were tuned on a labelled benchmark. With equal weights, hybrid search was worse than transcript search alone (Hit@1 0.67 vs 0.83), because keyframe vectors are noisy for questions about speech. Override with weights={...} or narrow with sources=(...).

Each result includes:

Build your own ask()

Cinematlas doesn't pick an LLM for you. Results turn into citable context in one call, and you can use any model:

hits = engine.search("How loud is a sonic boom, and why is it banned over land?", top_k=5)
prompt = f"""Answer using only these video excerpts. Cite them like [1].

{hits.to_context()}

Question: How loud is a sonic boom, and why is it banned over land?"""

answer = my_llm(prompt)                                   # any provider, any SDK
links = {f"[{i}]": h.link for i, h in enumerate(hits, 1)}           # citations -> timestamps

to_context() (also cinematlas.to_context(results)) renders [1] <video> @ 7:11 <link> followed by the sentence. From the shell: cinematlas search "…" --format context | your-llm-cli.

Ingestion sources

Remote URLs

ingest_video() accepts YouTube links, direct file URLs (including presigned S3/GCS/Azure URLs) and anything else yt-dlp supports. It's built to handle URLs your users paste in:

Redirects followed by yt-dlp aren't re-checked. For public-facing apps, also route downloads through an egress proxy.

Uploads

ingest_file() accepts a path, bytes, a binary file object, a FastAPI UploadFile, or a Flask FileStorage. Uploads stream to disk in 1 MiB chunks. Without a video_id, the ID is the content hash, so a re-upload replaces instead of duplicating. Empty or undecodable files raise IngestionError.

@app.post("/videos")                          # FastAPI (sync def runs in a threadpool)
def upload(file: UploadFile):
    return {"scenes": engine.ingest_file(file)}

CLI

cinematlas doctor                                       # diagnose; --json for CI, exit 1 on failures
cinematlas setup                                        # create indexes; --update fixes outdated ones in place
cinematlas ingest www.b.com/v.mp4                       # URL (YouTube or file)
cinematlas ingest ./talk.mp4 --video-id talk-01         # local file
curl -sL https://b.com/v.mp4 | cinematlas ingest - --filename v.mp4   # stdin
cinematlas search "how loud is a sonic boom" -k 3       # table in a terminal, JSON lines when piped
cinematlas search "a person in a hangar" --by visual    # hybrid | transcript | visual | text
cinematlas search "…" --format context                  # citable text for an LLM

MongoDB Atlas features used

Native stages are detected automatically: the engine tries each once, remembers the answer, and falls back with one explained warning. Force them with native_fusion= / native_rerank=. $rerank must be enabled in your Atlas project settings. cinematlas doctor tells you whether it is.

Reliability

• Replacing a video never loses it. Re-ingest inserts the new version, then deletes older ones (ingest_id). A failed re-ingest leaves the previous version searchable next to a FAILED tombstone.
• Speech lands in the right scene. VAD-trimmed word timestamps. A sentence clearly straddling a cut is split word by word; timestamp jitter never copies sentences across cuts.
• Vectors stay aligned. Scenes without a keyframe get None; every other vector stays on its own scene.
• Degrades instead of failing. Voyage and download calls retry with backoff. A failed search source is skipped. A reranker outage falls back to word-overlap moments. Missing audio means visual-only indexing.
• Provenance. embedding_models on every document records which models produced its vectors.

Configuration

Every external client can be injected (mongo_client=, voyage_client=, s3_client=, openai_client=). That's how the test suite runs offline.

Development

uv sync
uv run pytest -m "not integration and not media"   # unit: offline, ~8s
uv run pytest -m media                              # real ffmpeg / yt-dlp path / Whisper on the fixture, offline
uv run pytest -m integration                        # live Atlas (autoEmbed) + Docker Atlas Local (client)
uv run python bench/ingest.py && uv run python bench/run.py   # retrieval benchmark

No test contacts YouTube. End-to-end runs use tests/fixtures/x59_quiet_crew.mp4 (847 KiB, NASA, public domain). Integration env (.env): MDB_URI or MONGODB_URI, VOYAGE_API_KEY. The Atlas Local container starts and stops by itself. Release: bump version, then rm -rf dist && uv build && uv publish.

License

MIT. Test and benchmark media: NASA, public domain.