Cinematlas

Ask a question. Get the second in the video that answers it.

Cinematlas turns videos into searchable moments in MongoDB Atlas. It listens to what's said and looks at what's shown, and for each question it decides which to trust.

pip install "cinematlas[whisper]"
cinematlas doctor
cinematlas ingest "www.b.com/keynote.mp4"
cinematlas search "when do they announce pricing?"

 1. vid_3f2a…#12 @    7:11  Pricing starts at ten dollars a seat.
    https://www.b.com/keynote.mp4#t=431

The story behind it: blog.md · the numbers: bench/RESULTS.md · the limits: honest.md · running it in production: considerations.md

How it works

 video (URL, YouTube, upload)
   │
   ├─ scenes ────── PySceneDetect cuts, ≤30s each
   ├─ said ──────── faster-whisper → timestamped sentences, aligned to scenes word by word
   └─ shown ─────── a keyframe per scene
                      │
   Voyage AI ─────── keyframe vector · transcript vector · joint keyframe+speech vector
                      │
   MongoDB Atlas ─── one document per scene: vectors, sentences, deep links

 search(question)
   ├─ $rankFusion over keyframe, scene, transcript and full-text retrieval   (one query)
   ├─ $rerank over candidate sentences                                        (finds the moment)
   └─ route: is this about what was said or what was shown?                   (reranker confidence)

Questions about video come in two kinds. "How many medals has his beer won?" is about what was said; "the one with the girl on hay bales" is about what was shown. Each needs a different specialist:

Hit@1 on a 60-question benchmark (details and caveats).

The reranker scores how well any transcript sentence answers the question. High means "about what was said", low means "about what was shown". search() turns that into a speech confidence and weights the two specialists accordingly. The thresholds are calibrated per reranker; on one without a calibration, search uses fixed fusion rather than guess.

Quickstart

export MONGODB_URI="mongodb+srv://…"     # MDB_URI also works
export VOYAGE_API_KEY="pa-…"

from cinematlas import Cinematlas

engine = Cinematlas()
engine.ensure_indexes()                                  # once; idempotent

engine.ingest("https://www.youtube.com/watch?v=5NhYvbMdbBU")
engine.ingest("www.b.com/talk.mp4")                      # any file URL, scheme optional
engine.ingest("lecture.mov")                             # or a path, bytes, file object, web upload

results = engine.search("how loud is a sonic boom?")
print(results)                  # a readable table (markdown in Jupyter)
results.top.link                # 'https://…#t=34': the second it's said
results.top.text                # 'Sonic booms are about 110 decibels.'
results.top.explain()           # why it ranked: rank per source, relevance, score
results.speech_confidence       # 1.0: routed as a question about what was said

Results are plain dicts underneath (json.dumps works) with attribute shortcuts on top.

Build your own answer layer

Cinematlas doesn't choose an LLM for you. Results become numbered, citable context in one call:

hits = engine.search("How loud is a sonic boom, and why is it banned over land?")
answer = my_llm(f"Answer from these excerpts, citing [n]:\n\n{hits.to_context()}")
sources = {f"[{i}]": h.link for i, h in enumerate(hits, 1)}

From the shell: cinematlas search "…" --format context | your-llm-cli.

Search

Every hit carries moment (the best sentence, {start, end, text}), moment_link (YouTube ?t=431s, direct files #t=431), ranks (position in each source), relevance, score, plus the scene's fields (video_id, scene_id, timestamp_start/end, transcript, segments, video_url, …). results.speech_confidence and results.weights show how the question was routed.

All of them accept video_id= and return SearchResults. Query embeddings are cached per engine (exact text, per model), so repeated searches skip the Voyage round trip; engine.query_cache_info() shows hits and misses.

Ingest

engine.ingest(source) accepts a URL (YouTube or any file link, scheme optional), a local path, bytes, a binary file object, or a FastAPI UploadFile / Flask FileStorage. It returns an IngestResult (video id, scene counts, per-stage timings); pass progress=lambda stage, info: … to follow along.

@app.post("/videos")                     # FastAPI
def upload(file: UploadFile):
    return {"scenes": engine.ingest(file).scenes}

• Uploads stream to disk in 1 MiB chunks. Without a video_id, the ID is a content hash, so a re-upload replaces instead of duplicating.
• Remote URLs are treated as untrusted: private and metadata addresses are refused (allow_private_urls=True for trusted hosts), only http(s) is allowed, downloads are capped at max_download_mb, credentials in signed URLs are stripped before storage, and deep links use #t= fragments so signatures stay valid.
• Re-ingesting a video replaces it without a gap: the new version is written before the old one is removed.

CLI

cinematlas doctor                               # health check with fixes; --json, exit 1 on failure
cinematlas setup [--update]                     # create indexes; --update upgrades outdated ones in place
cinematlas ingest <url|path|->                  # progress on stderr, IngestResult JSON on stdout
cinematlas search "<question>" [-k 5]           # table in a terminal, JSON lines when piped
    [--by hybrid|transcript|visual|text] [--format table|json|context] [--video-id ID]

Global options: --uri, --db, --collection, --transcript-mode, -v.

Doctor

$ cinematlas doctor
 ✓ MongoDB                        connected · server 9.0.2 · cinematlas.scenes
 ✓ Transcript search              Atlas autoEmbed (voyage-4, embedded server-side)
 ! Index cinematlas_vector_index  outdated: visual_embedding.quantization: None -> 'scalar'
                                  → engine.ensure_indexes(update=True)  ·  cinematlas setup --update  (in place, no downtime)
 ✓ $rankFusion                    hybrid search runs as one native query
 ✓ $rerank                        sentence reranking runs inside Atlas (rerank-2.5)
 ✓ Routing                        adaptive, calibrated for rerank-2.5 (0.45–0.55)
 ✓ Voyage AI                      API key works (voyage-multimodal-3.5, voyage-4)
 ✓ Data                           65 scenes across 6 videos
 ✓ ffmpeg                         /opt/homebrew/bin/ffmpeg
 ✓ Speech-to-text                 faster-whisper (small), runs locally

Also available as engine.doctor() (.ok, .problems, .to_dict()). When a native stage isn't available, Cinematlas says why once per process (disabled, too old, or not offered on this deployment) and keeps working on an equivalent path.

MongoDB Atlas features

Configuration

Every external client can be injected (mongo_client=, voyage_client=, s3_client=, openai_client=).

Development

uv sync
uv run pytest -m "not integration and not media"   # unit, offline (~9s)
uv run pytest -m media                              # real ffmpeg / Whisper / downloads, on a committed fixture
uv run pytest -m integration                        # live Atlas + Docker Atlas Local (reads .env)
uv run python bench/ingest.py && uv run python bench/run.py   # the benchmark

CI runs the unit and media tiers. No test contacts YouTube: end-to-end runs use an 847 KiB public-domain NASA clip with known cuts and known speech (build script), served over loopback HTTP. The integration tier runs locally from .env.

MIT license. Test and benchmark media: NASA, public domain.