Cinematlas

Ask a question. Get the second in the video that answers it.

pip install "cinematlas[video,whisper]"
cinematlas doctor
cinematlas ingest "https://www.youtube.com/watch?v=5NhYvbMdbBU"
cinematlas search "how loud is a sonic boom?"

 1. 5NhYvbMdbBU#11 @    0:56  Sonic booms can be about as loud as a balloon popping.
    https://www.youtube.com/watch?v=5NhYvbMdbBU&t=56s
 2. 5NhYvbMdbBU#10 @    0:51  These sonic booms are really loud.
    https://www.youtube.com/watch?v=5NhYvbMdbBU&t=51s

Or try it in the browser: pip install "cinematlas[demo,whisper]" and cinematlas demo. Add a video by URL or upload, ask a question, and the player jumps to the second that answers it.

The finding: fuse within a unit, chunk across units

A record carries several signals about the same thing: a scene's picture and speech, a photo and its caption. The standard design gives each signal its own index and merges the ranked lists. That loses: on any question about one signal, the lists disagree, and merging averages the right answer away.

Fuse. Embed a record's signals together, into one vector.

It wins on data nobody tuned on (240 questions written blind by an AI agent, on space footage, space photography and museum art), isn't reading burned-in subtitles, wins even when a record's parts describe different things (0.70 vs 0.11), and holds on the kind of queries people really type: terse and from memory (0.80 vs 0.70 against the strongest merge) or full of typos (0.84 vs 0.61).

Chunk. One vector per record breaks when a part is long. Bury each photo's description among 31 others and the joint vector falls below chunked late fusion; the long text even drowns out the photo. Embed the photo together with each chunk instead:

Both come with paired significance tests and fourteen predictions written down before each run, four of which failed. TL;DR · Paper · every table · the story · review · what's new

In the library: video search() ranks scenes with the joint vector and uses a reranker only to pick the exact second (.adaptive() leans ahead on speech questions, at twice the latency). cinematlas.core applies both halves to any records: Text("title") + Image("photo") to fuse, Text("body", chunk=…) to chunk.

Quickstart

export MONGODB_URI="mongodb+srv://…"     # MDB_URI also works
export VOYAGE_API_KEY="pa-…"

from cinematlas import Cinematlas

engine = Cinematlas()
engine.ensure_indexes()                                  # once; idempotent

engine.ingest("https://www.youtube.com/watch?v=5NhYvbMdbBU")   # NASA: 60 Second Science, Sonic Booms
engine.ingest("lecture.mov")                             # or a URL, bytes, file object, web upload

results = engine.search("how loud is a sonic boom?")     # lazy; runs on first read
results.top.link                                         # 'https://www.youtube.com/watch?v=5NhYvbMdbBU&t=56s'
results.top.text                                         # 'Sonic booms can be about as loud as a balloon popping.'
results.top.explain()                                    # rank per source, relevance, score

engine.search("an airplane in the sky").only("scene")   # the joint vector alone: scenes only, ~100 ms

Narrow any search by video or by your own metadata. Declare the filter fields once, and they're built into every search index, so filtering happens before ranking:

engine = Cinematlas(filters=("genre", "course"))
engine.ensure_indexes()
engine.ingest("lecture.mov", metadata={"genre": "lecture", "course": "cs101"})

engine.search("when is the exam?").where(course="cs101").limit(3)
engine.search("the rocket launch").video("5NhYvbMdbBU", "abc123").where(genre=["science", "news"])

Plain pip install cinematlas is search-only (pymongo + voyageai), so it can serve queries from a backend. Ingesting video needs cinematlas[video] (OpenCV, PySceneDetect, yt-dlp), and speech-to-text needs whisper (or an OPENAI_API_KEY).

Cinematlas doesn't pick an LLM for you; results.to_context() gives you numbered, citable excerpts to pass to one.

Every Voyage call is metered. result.usage is what one ingest sent (per model: calls, inputs, tokens, image pixels), engine.usage is the running total, and engine.usage.cost(prices) turns it into dollars at the prices you pass. Rate limits (429) back off longer than other errors, with jitter. A notebook version of this lives in docs/quickstart.ipynb.

In an async server, await any query. It runs the same ranking code in a worker thread, so the event loop never blocks and concurrent searches overlap:

@app.get("/search")
async def search(q: str):
    return [{"at": h.timestamp, "text": h.text, "link": h.link} for h in await engine.search(q).limit(5)]

Results are dicts underneath: every field you stored comes back, and json.dumps works. On top of that, every hit has rank, score, moment ({text, start, end, relevance}), text and explain(). Video hits (SearchHit) add video_id, scene_id, timestamp, link and per-source ranks.

Observability. With cinematlas[otel], each ingest is an OpenTelemetry span with one child per stage (fetch, scenes, speech, embed, store), and each search is a span. Both carry the Voyage calls and tokens they used. cinematlas[progress] draws a progress bar for cinematlas ingest.

Beyond video: cinematlas.core

The finding isn't about video. Any records whose signals describe the same thing (product photos and titles, slides and their text, diagrams and captions) search better with one joint vector than with separate indexes merged afterwards. cinematlas.core is that idea as a small library:

from cinematlas.core import Atlas, Text, Image

photos = Atlas().collection("nasa.photos",
    embed=Text("title") + Image("image"),      # parts compose into ONE joint vector
    moment="description",                      # the reranker picks the best sentence
    filters=["center"], key="nasa_id")         # filterable fields; re-adding a key replaces it
photos.setup()                                 # the vector index; idempotent, updates in place
photos.add(records)                            # any iterable of dicts, or a loader
photos.wait_until_searchable()

photos.search("astronaut fixing a telescope in space").top.title   # 'Making Room for Hubble's New Camera'
photos.search(Image("mars.jpg")).where(center="JPL")               # query by picture, filtered

That output is real: examples/photos.py indexes about 200 NASA photos and runs it.

Video and records share one connection. atlas.videos(...) is a video collection on the same cluster, with the same Voyage client, the same usage meter, the same retries, and the same query builder (.where, .limit, .rerank, .candidates):

from cinematlas.core import Atlas, Slides

atlas = Atlas()
talks = atlas.videos("media.talks", filters=("course",))       # everything Cinematlas does, shared connection
slides = atlas.collection("media.slides", like=Slides)

talks.search("when is the exam?").where(course="cs101").top.link
slides.search("the exam schedule").where(deck="week1").top.text
print(atlas.usage)                                              # both, on one meter

Long text? Chunk it, fused. Text("body", chunk=800) splits long text into its paragraphs (up to 800 characters each) and embeds each piece together with the record's other parts; search keeps each record's best piece and returns it as the moment. For text without paragraphs (transcripts, OCR, scraped pages) use chunk=Semantic(800), which cuts where the topic changes:

manuals = atlas.collection("manuals", embed=Text("title") + Text("body", chunk=800) + Image("cover"),
                           key="id", moment="body")
manuals.search("how do I reset the pressure valve?").top.text   # the passage that answers

Built in. Parts: Text (labels, nested or computed fields, truncation, chunk=) and Image (PIL, bytes, path or URL, downscaled to Voyage's limits). Loaders, each with a suggested setup you can borrow with atlas.collection("decks", like=Slides):

decks = atlas.collection("decks", like=Slides)
decks.add(Slides("q3-review.pptx", pdf="q3-review.pdf"))
decks.search("the slide where we showed Q3 churn").top.text    # the speaker-note sentence about churn

shots = atlas.collection("shots", like=Screenshots)
shots.add(Screenshots("qa-run-42/"))
shots.search("the screen with the red error banner").top.text  # 'Payment failed: card declined'

Check it on your own data. Every vector library says its approach wins; this one lets you check. Create the collection with late=True (it also stores one vector per part), label 30–50 questions, and evaluate() runs the joint vector against merged per-part rankings with the same paired test:

photos = atlas.collection("photos", embed=Text("title") + Image("image"), key="id", late=True)
...
print(photos.evaluate([{"q": "astronaut fixing a telescope", "relevant": ["sts082-717-029"]}, ...]))

                  Hit@1  Hit@10    MRR
joint vector       0.93    0.99   0.95
merged rankings    0.62    0.90   0.71

The joint vector wins on your data: joint 0.93 vs merged 0.62 Hit@1 on 80 questions (25 vs 1 disputed, p = < 0.001).

The default challenger is Reciprocal Rank Fusion (what Atlas $rankFusion does); fusion="sum" tests against the strongest merge we found. To see the merged ranking for a single question, use photos.search(q).merged("sum").

Extend it. A part is anything that turns a record into text or images for the vector; a loader is anything that yields records:

from cinematlas.core import Part, Loader, Collection

class Price(Part):                                   # numbers become words the model understands
    def inputs(self, record):
        return [f"costs ${record[self.field]:.0f}"] if record.get(self.field) else []

class Tickets(Loader):                               # records from anywhere
    key, moment = "id", "body"
    embed = Text("subject") + Text("body")
    def __iter__(self):
        yield from my_helpdesk_api.tickets()

@Collection.extend                                   # add methods to every collection, jQuery-style
def newest(self, k=5):
    return list(self.mongo.find({}, {"embedding": 0}).sort("_id", -1).limit(k))

Ship a plugin as a package with a cinematlas.plugins entry point, and cinematlas.core.plugins() lists it next to the built-ins.

Examples

Five runnable scripts in examples/. They need only MONGODB_URI and VOYAGE_API_KEY in .env. The first two search a demo corpus of six NASA interviews, the next two index your own video, and the last one searches photos.

uv run python examples/search.py "a little girl standing on hay bales"
uv run python examples/search.py --adaptive "what did he say about his first flight?"
uv run python examples/ask.py "What first got these people interested in aviation?"
uv run python examples/index_and_search.py lecture.mp4 "when is the exam?"
uv run python examples/photos.py "a rover's tracks on red sand" --center JPL

How it works

 ingest(video)
   ├─ scenes ── PySceneDetect cuts, ≤30 s each
   ├─ said ──── faster-whisper → timestamped sentences, aligned to scenes
   ├─ shown ─── the middle keyframe of each scene
   └─ Voyage ── one joint keyframe+transcript vector per scene (plus keyframe-only and transcript vectors)
                → one MongoDB Atlas document per scene

 search(question)
   ├─ joint-vector search over scenes                                        one query, ranks scenes
   └─ $rerank over those scenes' sentences                                   picks the second

 search(question).adaptive()
   ├─ $rankFusion over scene, keyframe, transcript and BM25 retrieval        one query
   └─ weights set per question by the reranker's confidence                  said vs shown

Each one chains with .video(...), .where(...), .limit(k) and .rerank(False). Each call returns a new query, so a base query can be shared and refined safely. Every hit carries moment ({start, end, text}), moment_link (YouTube ?t=431s, files #t=431), ranks, relevance and the scene's fields.

ingest() accepts:

Remote URLs are treated as untrusted: private addresses are refused, downloads are capped at max_download_mb (cloud objects are checked before downloading), and signed-URL credentials are stripped before storage. Re-ingesting a video replaces it without a gap.

CLI

cinematlas doctor                        # checks the deployment and prints the exact fix for each problem
cinematlas setup [--update]              # create indexes; --update upgrades them in place
cinematlas ingest <url|path|-> [--meta course=cs101]    # progress on stderr, JSON on stdout
cinematlas search "<question>" [-k 5] [--where course=cs101] [--video-id ID]
                  [--by hybrid|adaptive|scene|transcript|visual|text] [--format table|json|context]
cinematlas demo [--port 8765]            # local web app: add videos, search, jump to the second

Global options: --uri, --db, --collection, --transcript-mode, --filter FIELD, -v. A field used in --where is declared as a filter automatically. Before the first search on a new field, declare it and run cinematlas --filter course setup --update so the indexes can filter on it.

Atlas features used

Atlas Vector Search for the joint vectors, $rerank (8.3+) for in-database sentence reranking, $rankFusion (8.0+) for adaptive routing's one-query fusion, Automated Embedding for transcripts, Atlas Search for BM25, scalar quantization and BSON float32 vectors. Each has an equivalent fallback, and cinematlas doctor tells you which path is in use.

Development

cinematlas/
  engine.py        Cinematlas: the facade (configuration + public API)
  media.py         download, uploads, audio, scene cuts, keyframes, S3
  urlsafety.py     remote URLs are untrusted input (SSRF guard)
  transcribe.py    speech to timestamped sentences (OpenAI Whisper or faster-whisper)
  embed.py         keyframe, joint, transcript and query vectors
  ingest.py        the pipeline and its gapless replace
  search.py        single sources, fusion, reranking, routing
  query.py         the lazy query builder shared by video and records (Search, RecordSearch)
  results.py       Hit / Hits: plain dicts with typed accessors
  usage.py         Voyage usage metering, cost estimates, the one retry path
  tracing.py       optional OpenTelemetry spans
  capabilities.py  native-stage fallbacks and routing calibration
  doctor.py        what's wrong and how to fix it
  demo/            cinematlas demo: FastAPI server + a one-file web page
  core/            joint-vector search for any records: parts, chunkers, loaders, evaluate()

uv sync
uv run pytest -m "not integration and not media"    # unit, offline (~9 s)
uv run pytest -m media                               # real ffmpeg / Whisper on a committed NASA fixture
uv run pytest -m integration                         # live Atlas + Docker Atlas Local (reads .env)
uv run python bench/ingest.py                        # benchmark corpora, once: interviews,
uv run python bench/ingest.py --no-captions          #   caption ablation,
uv run python bench/ingest.py --station              #   held-out video,
uv run python bench/photos.py --ingest               #   photos (aligned and misaligned)
uv run python bench/met.py --ingest                  #   Met artworks
uv run python bench/boundary.py --ingest             #   boundary tests (see paper.md for the full list)
uv run python bench/run.py                           # every table and paired test → bench/RESULTS.md

MIT license. Test and benchmark media: NASA, public domain.