StreamEmbed

Search your own screen history by describing what you saw. StreamEmbed indexes screenshots, from a folder or from your screen as you work, and finds a past screen from a sentence. Everything runs on your machine.

It embeds the pixels with Qwen3-VL-Embedding-2B, and encodes screenshots as a stream so that this keeps up on a laptop: blocks that repeat a neighbor are skipped, and the keys and values of blocks that did not change are reused from the previous frame.

Use

import streamembed as se

se.configure(index_type="flat")                      # optional; see below
se.build_index_from_directory("screenshots")         # a folder of .png / .jpg / .webp, searched recursively
capture = se.build_index_from_now(interval=10)       # capture the screen every 10 s, in the background

results = se.query_index("the flight search page with fares to Denver", top_k=10)
se.visualize_results(results)                        # writes an HTML page and opens it

capture.stop()

examples/main.py is a complete script.

Screens that did not change are not stored again: a capture is encoded when more than 10 % of its kept blocks changed since the last stored one, or 30 s later if anything changed (the paper's change gate).

The index folder

Index types

An index folder can be reopened with the other type. It also records the encoder, and refuses vectors from a different one.

Models

A model is a base model from its own Hugging Face repository, pinned to a commit, plus two LoRAs from the StreamEmbed model repository: one for screenshots (a consistency adapter that keeps pruned, streamed frames close to full frames) and one for questions. Only the LoRAs live in the StreamEmbed repository; its models.json names each model's base repository, commit and files.

The model repository also holds LoRAs for SigLIP2 So400m, SigLIP2 Base and VultronRetrieverFlash 0.8B. available_models() lists them, but the stream encoder supports Qwen3-VL models only, so selecting one raises NotImplementedError when StreamEmbed first encodes.

configure(models_repo=...) or $STREAMEMBED_MODELS points to another Hugging Face repository or a local folder with the same layout. An index records its model and refuses vectors from another one.

The first run of qwen3-vl-2b downloads the base model (about 4 GB) and the LoRAs (70 MB each); after that nothing is downloaded. Encoding needs about 8 GB of memory and takes 0.7 to 1.3 s per frame on an M2 Pro GPU; the change gate skips screens that did not change.

Requirements

Python 3.12 or later on macOS (Apple silicon, tested) or Linux under X11 (capture not yet tested). On macOS, capturing the screen needs Screen Recording permission for the app that runs Python; build_index_from_now raises PermissionError until it is granted.

Dependencies are pinned to torch 2.8.0 and transformers 5.17.0: the stream encoder re-implements parts of the Qwen3-VL forward pass against transformers internals, and its outputs were checked for equality with these versions. Install StreamEmbed in its own environment (for example uv tool, pipx, or a venv) if your project needs other versions.

Privacy

• Screenshots, vectors and metadata stay in the index folder (owner-only). Screenshots are stored unencrypted.
• Capture is skipped when the screen is locked, when the frontmost app is a password manager, and when the frontmost window is a private browsing window. Other windows on screen are not checked; use capture.pause() for sensitive work.
• Nothing is sent over the network after the model download.

Development

uv venv && uv pip install -e ".[dev]"
.venv/bin/python -m pytest          # API tests with a stand-in encoder; no model needed

src/streamembed/engine/ is the stream encoder from the paper's code and keeps its names: Token Pruning is spatial pruning, KV Reuse is temporal reuse.

License

Apache-2.0. See LICENSE and NOTICE; the model weights are downloaded under their own license.