Cinematlas

Ask a question. Get the second in the video that answers it.

pip install "cinematlas[whisper]"
cinematlas doctor
cinematlas ingest "https://www.youtube.com/watch?v=5NhYvbMdbBU"
cinematlas search "how loud is a sonic boom?"

 1. 5NhYvbMdbBU#11 @    0:56  Sonic booms can be about as loud as a balloon popping.
    https://www.youtube.com/watch?v=5NhYvbMdbBU&t=56s
 2. 5NhYvbMdbBU#10 @    0:51  These sonic booms are really loud.
    https://www.youtube.com/watch?v=5NhYvbMdbBU&t=51s

The finding: fuse in the embedding, not in the ranking

Video search has two kinds of question. "How many medals has his beer won?" is about what was said. "The one with the girl on hay bales" is about what was shown.

The standard design indexes speech and pictures separately, retrieves from each, and merges the ranked lists. That fails, because the lists disagree on every question that's about only one of the two, and merging averages the disagreement away. Cinematlas embeds each scene's keyframe and its transcript into one vector, so there's nothing to reconcile.

It holds on data we never tuned on. The held-out video is a different domain (a silent station tour, astronaut Q&A, science demos; 386 scenes, no burned-in captions), and the photos (394 NASA photos with titles and descriptions) aren't video at all. Their 160 questions were written by an AI agent that saw only the media, never the code or results.

It isn't reading subtitles. On the first corpus, where every frame has burned-in captions, cropping them made keyframes alone worse on speech (0.53 → 0.40) but left the joint vector intact (0.73 → 0.77).

So the default ranks with that one vector. search() finds scenes with the joint vector and uses a sentence reranker only to pick the exact second. The router we built to rescue merged rankings ties it on both corpora (p = 1.0 and p = 0.69) at about twice the latency, so it's now opt-in. The two do differ: the default is better on questions about what was shown; routing leans ahead on what was said. Both find the exact second equally well once they have the right scene. If your users mostly ask about speech, pass routing="adaptive".

It holds even when it shouldn't. Pair each photo with another photo's text and early fusion does degrade (0.93 → 0.70), but merged rankings collapse (0.62 → 0.11): rank fusion needs a record's lists to agree. We predicted the opposite.

The paper: methods, predictions and limits · full results · the story.

Quickstart

export MONGODB_URI="mongodb+srv://…"     # MDB_URI also works
export VOYAGE_API_KEY="pa-…"

from cinematlas import Cinematlas

engine = Cinematlas()
engine.ensure_indexes()                                  # once; idempotent

engine.ingest("https://www.youtube.com/watch?v=5NhYvbMdbBU")   # NASA: 60 Second Science, Sonic Booms
engine.ingest("lecture.mov")                             # or a URL, bytes, file object, web upload

results = engine.search("how loud is a sonic boom?")     # the scene, down to the second
results.top.link                                         # 'https://www.youtube.com/watch?v=5NhYvbMdbBU&t=56s'
results.top.text                                         # 'Sonic booms can be about as loud as a balloon popping.'
results.top.explain()                                    # rank per source, relevance, score

engine.search_scene_vector("an airplane in the sky")    # the joint vector alone: scenes only, ~100 ms

Results are plain dicts underneath (json.dumps works). Cinematlas doesn't pick an LLM for you; results.to_context() gives you numbered, citable excerpts to pass to one.

Beyond video: cinematlas.core

The finding isn't about video. Any records whose signals describe the same thing (product photos and titles, slides and their text, diagrams and captions) search better with one joint vector than with separate indexes merged afterwards. cinematlas.core is that idea as a small library:

from cinematlas.core import Atlas, Text, Image

photos = Atlas().collection("nasa.photos",
    embed=Text("title") + Image("image"),      # parts compose into ONE joint vector
    moment="description",                      # the reranker picks the best sentence
    filters=["center"], key="nasa_id")         # filterable fields; re-adding a key replaces it
photos.setup()                                 # the vector index; idempotent, updates in place
photos.add(records)                            # any iterable of dicts, or a loader
photos.wait_until_searchable()

photos.search("astronaut fixing a telescope in space").top.title   # 'Making Room for Hubble's New Camera'
photos.search(Image("mars.jpg"), where={"center": "JPL"})           # query by picture, filtered

That output is real: examples/photos.py indexes about 200 NASA photos and runs it.

Built in. Parts: Text (labels, nested or computed fields, truncation) and Image (PIL, bytes, path or URL, downscaled to Voyage's limits). Loaders, each with a suggested setup you can borrow with atlas.collection("decks", like=Slides):

decks = atlas.collection("decks", like=Slides)
decks.add(Slides("q3-review.pptx", pdf="q3-review.pdf"))
decks.search("the slide where we showed Q3 churn").top.text    # the speaker-note sentence about churn

shots = atlas.collection("shots", like=Screenshots)
shots.add(Screenshots("qa-run-42/"))
shots.search("the screen with the red error banner").top.text  # 'Payment failed: card declined'

Check it on your own data. Every vector library says its approach wins. Create the collection with late=True (it also stores one vector per part), label 30–50 questions, and evaluate() runs the joint vector against merged per-part rankings with the same paired test as the benchmarks:

photos = atlas.collection("photos", embed=Text("title") + Image("image"), key="id", late=True)
...
print(photos.evaluate([{"q": "astronaut fixing a telescope", "relevant": ["sts082-717-029"]}, ...]))

                  Hit@1  Hit@10    MRR
joint vector       0.93    0.99   0.95
merged rankings    0.62    0.90   0.71

The joint vector wins on your data: joint 0.93 vs merged 0.62 Hit@1 on 80 questions (25 vs 1 disputed, p = < 0.001).

Extend it. A part is anything that turns a record into text or images for the vector; a loader is anything that yields records:

from cinematlas.core import Part, Loader, Collection

class Price(Part):                                   # numbers become words the model understands
    def inputs(self, record):
        return [f"costs ${record[self.field]:.0f}"] if record.get(self.field) else []

class Tickets(Loader):                               # records from anywhere
    key, moment = "id", "body"
    embed = Text("subject") + Text("body")
    def __iter__(self):
        yield from my_helpdesk_api.tickets()

@Collection.extend                                   # add methods to every collection, jQuery-style
def newest(self, k=5):
    return list(self.mongo.find({}, {"embedding": 0}).sort("_id", -1).limit(k))

Ship a plugin as a package with a cinematlas.plugins entry point, and cinematlas.core.plugins() lists it next to the built-ins.

Examples

Five runnable scripts in examples/. They need only MONGODB_URI and VOYAGE_API_KEY in .env. The first two search a demo corpus of six NASA interviews, the next two index your own video, and the last one searches photos.

uv run python examples/search.py "a little girl standing on hay bales"
uv run python examples/search.py --adaptive "what did he say about his first flight?"
uv run python examples/ask.py "What first got these people interested in aviation?"
uv run python examples/index_and_search.py lecture.mp4 "when is the exam?"
uv run python examples/photos.py "a rover's tracks on red sand" --center JPL

How it works

 ingest(video)
   ├─ scenes ── PySceneDetect cuts, ≤30 s each
   ├─ said ──── faster-whisper → timestamped sentences, aligned to scenes
   ├─ shown ─── the middle keyframe of each scene
   └─ Voyage ── one joint keyframe+transcript vector per scene (plus keyframe-only and transcript vectors)
                → one MongoDB Atlas document per scene

 search(question)
   ├─ joint-vector search over scenes                                        one query, ranks scenes
   └─ $rerank over those scenes' sentences                                   picks the second

 search(question, routing="adaptive")
   ├─ $rankFusion over scene, keyframe, transcript and BM25 retrieval        one query
   └─ weights set per question by the reranker's confidence                  said vs shown

All of them accept video_id=. Every hit carries moment ({start, end, text}), moment_link (YouTube ?t=431s, files #t=431), ranks, relevance and the scene's fields.

ingest() accepts a URL (YouTube or any file link, scheme optional), a path, bytes, a file object, or a FastAPI UploadFile / Flask FileStorage. Remote URLs are treated as untrusted: private addresses are refused, downloads are capped, and signed-URL credentials are stripped before storage. Re-ingesting a video replaces it without a gap.

CLI

cinematlas doctor                        # checks the deployment and prints the exact fix for each problem
cinematlas setup [--update]              # create indexes; --update upgrades them in place
cinematlas ingest <url|path|->           # progress on stderr, JSON on stdout
cinematlas search "<question>" [-k 5] [--by hybrid|adaptive|transcript|visual|text] [--format table|json|context]

Global options: --uri, --db, --collection, --transcript-mode, -v.

Atlas features used

Atlas Vector Search for the joint vectors, $rerank (8.3+) for in-database sentence reranking, $rankFusion (8.0+) for adaptive routing's one-query fusion, Automated Embedding for transcripts, Atlas Search for BM25, scalar quantization and BSON float32 vectors. Each has an equivalent fallback, and cinematlas doctor tells you which path is in use.

Development

uv sync
uv run pytest -m "not integration and not media"    # unit, offline (~9 s)
uv run pytest -m media                               # real ffmpeg / Whisper on a committed NASA fixture
uv run pytest -m integration                         # live Atlas + Docker Atlas Local (reads .env)
uv run python bench/ingest.py                        # the benchmark corpora, once:
uv run python bench/ingest.py --no-captions          #   caption ablation
uv run python bench/ingest.py --station              #   held-out corpus
uv run python bench/run.py                           # both corpora, caption ablation, paired tests

MIT license. Test and benchmark media: NASA, public domain.