Cinematlas

Search inside video, down to the second: what was shown and what was said.

Cinematlas turns videos (YouTube links, file URLs, uploads) into searchable scenes in MongoDB Atlas. Ask a question and get back the scene that answers it, the exact sentence, and a deep link to the second it's said.

pip install "cinematlas[whisper]"
cinematlas setup
cinematlas ingest "www.b.com/keynote.mp4"
cinematlas search "when do they talk about pricing?"

{"video_id": "vid_3f2a…", "scene_id": 12, "score": 0.047,
 "moment": {"start": 431.2, "end": 436.8, "text": "Pricing starts at ten dollars a seat."},
 "moment_link": "https://www.b.com/keynote.mp4#t=431",
 "ranks": {"scene": 1, "transcript": 1, "text": 2, "visual": 9, "rerank": 1}, "relevance": 0.91}

Built on PySceneDetect, yt-dlp, faster-whisper, Voyage AI and MongoDB Atlas (Vector Search, autoEmbed, Atlas Search, $rankFusion, $rerank). How it was built and measured: blog.md · benchmark · production considerations.

How it works

 URL / YouTube / upload / stdin
            │
            ▼
   fetch + guard ─── SSRF guard · http(s) only · size cap · credential redaction
            │
   ┌────────┴──────────────────────────┐
   ▼                                   ▼
 PySceneDetect cuts (≤30s scenes)   ffmpeg → 16 kHz mono → faster-whisper
   │                                   (VAD + word timestamps)
   ▼                                   │ sentences assigned to scenes,
 middle-frame keyframes                │ split at cuts word by word
   │                                   ▼
   ├─ voyage-multimodal-3.5 ──────► visual_embedding   (keyframe)
   ├─ voyage-multimodal-3.5 ──────► scene_embedding    (keyframe + speech, interleaved)
   └─ voyage-4 (autoEmbed/client) ─► transcript vectors · BM25 full-text index
                                       │
                                       ▼
          MongoDB Atlas: one document per scene, timestamped sentences inside

 search() ─► $rankFusion(visual, scene, transcript, text) ─► sentence-level rerank
          ─► results with ranks, relevance, moment and moment_link

Install

ffmpeg must be on your PATH (brew install ffmpeg / apt-get install -y ffmpeg).

pip install "cinematlas[whisper]"    # recommended: + local speech-to-text (faster-whisper, no PyTorch)
pip install cinematlas               # core only (transcribe via OPENAI_API_KEY + [openai])
pip install "cinematlas[all]"        # + S3 keyframe uploads + OpenAI Whisper API

export MONGODB_URI="mongodb+srv://<user>:<password>@cluster.mongodb.net/"   # MDB_URI also accepted
export VOYAGE_API_KEY="pa-..."

Quickstart

from cinematlas import Cinematlas

engine = Cinematlas()        # reads MONGODB_URI / MDB_URI and VOYAGE_API_KEY
engine.ensure_indexes()      # one time, idempotent: vector, full-text and lookup indexes

engine.ingest_video("https://www.youtube.com/watch?v=5NhYvbMdbBU")   # YouTube
engine.ingest_video("www.b.com/v.mp4")                                # any file URL (scheme optional)
engine.ingest_file("talk.mp4")                                        # path, bytes, file object, upload

for hit in engine.search("how loud is a sonic boom?", top_k=3):
    print(hit["moment_link"], "-", hit["moment"]["text"] if hit["moment"] else hit["transcript"])

search() is the one call most apps need. The single-source searches are still there if you want them: search_transcript, search_text, search_visual_vector, search_scene_vector. Every search accepts video_id= to scope results to one video.

Search

search() fuses up to five ranked lists with Reciprocal Rank Fusion:

Default weights (visual 0.25 · scene 1 · transcript 1 · text 1 · rerank 2) were tuned on a labelled benchmark. With equal weights, hybrid search was worse than transcript search alone (Hit@1 0.67 vs 0.83), because keyframe vectors are noisy for questions about speech. Override with weights={...} or narrow with sources=(...).

Each result includes:

Build your own ask()

Cinematlas doesn't pick an LLM for you. Results turn into citable context in one call, and you can use any model:

from cinematlas import to_context

hits = engine.search("How loud is a sonic boom, and why is it banned over land?", top_k=5)
prompt = f"""Answer using only these video excerpts. Cite them like [1].

{to_context(hits)}

Question: How loud is a sonic boom, and why is it banned over land?"""

answer = my_llm(prompt)                                   # any provider, any SDK
links = {f"[{i}]": h["moment_link"] for i, h in enumerate(hits, 1)}   # citations -> timestamps

to_context renders [1] <video> @ 7:11 <link> followed by the sentence. From the shell: cinematlas search "…" --format context | your-llm-cli.

Ingestion sources

Remote URLs

ingest_video() accepts YouTube links, direct file URLs (including presigned S3/GCS/Azure URLs) and anything else yt-dlp supports. It's built to handle URLs your users paste in:

Redirects followed by yt-dlp aren't re-checked. For public-facing apps, also route downloads through an egress proxy.

Uploads

ingest_file() accepts a path, bytes, a binary file object, a FastAPI UploadFile, or a Flask FileStorage. Uploads stream to disk in 1 MiB chunks. Without a video_id, the ID is the content hash, so a re-upload replaces instead of duplicating. Empty or undecodable files raise IngestionError.

@app.post("/videos")                          # FastAPI (sync def runs in a threadpool)
def upload(file: UploadFile):
    return {"scenes": engine.ingest_file(file)}

CLI

cinematlas setup                                        # indexes -> {"transcript_mode": "autoembed"}
cinematlas ingest www.b.com/v.mp4                       # URL (YouTube or file)
cinematlas ingest ./talk.mp4 --video-id talk-01         # local file
curl -sL https://b.com/v.mp4 | cinematlas ingest - --filename v.mp4   # stdin
cinematlas search "how loud is a sonic boom" -k 3       # hybrid, JSON lines
cinematlas search "a person in a hangar" --by visual    # hybrid | transcript | visual | text
cinematlas search "…" --format context                  # citable text for an LLM

MongoDB Atlas features used

Native stages are detected automatically: the engine tries each once and remembers if the cluster rejects it. Force them with native_fusion= / native_rerank=. $rerank must be enabled in your Atlas project settings.

Reliability

• Replacing a video never loses it. Re-ingest inserts the new version, then deletes older ones (ingest_id). A failed re-ingest leaves the previous version searchable next to a FAILED tombstone.
• Speech lands in the right scene. VAD-trimmed word timestamps. A sentence clearly straddling a cut is split word by word; timestamp jitter never copies sentences across cuts.
• Vectors stay aligned. Scenes without a keyframe get None; every other vector stays on its own scene.
• Degrades instead of failing. Voyage and download calls retry with backoff. A failed search source is skipped. A reranker outage falls back to word-overlap moments. Missing audio means visual-only indexing.
• Provenance. embedding_models on every document records which models produced its vectors.

Configuration

Every external client can be injected (mongo_client=, voyage_client=, s3_client=, openai_client=). That's how the test suite runs offline.

Development

uv sync
uv run pytest -m "not integration and not media"   # unit: offline, ~8s
uv run pytest -m media                              # real ffmpeg / yt-dlp path / Whisper on the fixture, offline
uv run pytest -m integration                        # live Atlas (autoEmbed) + Docker Atlas Local (client)
uv run python bench/ingest.py && uv run python bench/run.py   # retrieval benchmark

No test contacts YouTube. End-to-end runs use tests/fixtures/x59_quiet_crew.mp4 (847 KiB, NASA, public domain). Integration env (.env): MDB_URI or MONGODB_URI, VOYAGE_API_KEY. The Atlas Local container starts and stops by itself. Release: bump version, then rm -rf dist && uv build && uv publish.

License

MIT. Test and benchmark media: NASA, public domain.