curu-vision

What is this

curu-vision is a Python library for programs that work with video, pictures and speech. It gets frames from a video and finds the scenes in it. It finds copies of a picture and measures blur, exposure and noise. It gives tags for pictures, makes vectors for search by meaning, makes transcripts with timed words and asks a vision-language model about a picture. The other curu packages use it, so each of these operations has one implementation. Only the model download, a video URL and a vision-language model on a model server or a cloud provider use the network.

What you can do

Install

You need these on the computer that installs the package:

• Python 3.12 or later.
• A Rust toolchain (rustup). The package builds a Rust extension when uv installs it.
• ffmpeg and ffprobe 6.0 or later on the path, and 8.1 or later on arm64. Install them with apt install ffmpeg or brew install ffmpeg. On arm64, Ubuntu 24.04's ffmpeg 6.1 is too old, and Homebrew's ffmpeg is correct. curu-vision does not include or download a decoder.

Add the package to your project by its release tag:

# pyproject.toml
[project]
dependencies = ["curu-vision"]

[tool.uv.sources]
curu-vision = { git = "https://github.com/darth-veitcher/curu-vision", tag = "v0.9.0" }

Two optional extras add transcription engines. Use curu-vision[mlx] for Parakeet on Apple silicon. Use curu-vision[gpu] for MOSS, which needs PyTorch and transformers. Your project chooses the CUDA build of PyTorch from its own package index.

The extra curu-vision[cloud] adds cloud vision-language models through PydanticAI. You need it only when you name a cloud model. A local model on Ollama or on a llama.cpp server needs no extra.

Download the models

The video module and the image signals module need no model. Tags, search by meaning and transcripts need model files, which are not in the package. Download each model one time before you use it:

python -m curu_vision fetch embeddinggemma-2      # search by meaning, about 3 GB
python -m curu_vision fetch embeddinggemma-2 --modality text --modality image  # 1.8 GB, no audio
python -m curu_vision fetch parakeet-tdt-0.6b-v3      # transcripts on the CPU, about 2.6 GB
python -m curu_vision fetch parakeet-tdt-0.6b-v3-mlx  # transcripts on Apple silicon, mlx extra
python -m curu_vision fetch moss-transcribe-diarize   # transcripts with speakers, gpu extra
python -m curu_vision fetch wd-v1-4-swinv2-tagger-v2  # tags, about 455 MB

The download is the only operation that goes to the network for curu-vision itself. Each file has a pinned revision, size and SHA-256, and the command verifies each file. If a download stops, run the same command again. The files go to CURU_VISION_CACHE_DIR, else $XDG_CACHE_HOME/curu-vision, else ~/.cache/curu-vision. When curu-vision loads a model, it never downloads it: a missing model is an error (ModelNotAvailable).

A vision-language model is not one of these files. It runs on a model server, and you get it with that server, for example ollama pull qwen3-vl:2b-instruct. curu-vision does not pull or start a model.

Quickstart

This example finds the shots in a video, and then measures the frame at the middle of each shot. It uses the video module and the image signals module. It needs ffmpeg, and it needs no model.

from curu_vision.media import VideoSource, load_media
from curu_vision.signals import load_signals

media = load_media()  # finds ffmpeg and ffprobe and checks their versions, one time
signals = load_signals()  # needs no model
source = VideoSource(path="episode.mkv")

print(media.probe(source).duration)  # the length of the video, in seconds
shots = media.shots(source)
middles = media.frames_at(source, [(shot.start + shot.end) / 2 for shot in shots.shots])
for shot, result in zip(shots.shots, signals.compute(middles.frames)):
    detail, phash = result.measurements.detail, result.hashes.phash
    print(shot.index, shot.start, shot.end, round(detail), f"{phash:016x}")

The README tests run this example on a generated video of 36 s with three shots of 12 s. It prints this:

36.0
0 0.0 12.0 3402 b2bfd084314c1f8f
1 12.0 24.0 3093 93de15d12d245573
2 24.0 36.0 2152 9b932b369705b368

Your video gives different values. The generated frames are small blocks of colour with sharp edges and a little noise. Thus their detail is usually higher than the 1,200 to 2,200 of the sharp test photos in Find blurred, dark, bright or noisy pictures.

What the results mean:

• The first line is the length of the video in seconds.
• Each line after it is one shot: its index, its start and its end in seconds, then two values of the frame at its middle.
• The fourth value, detail, measures fine detail and sharp edges. A lower value usually means a more blurred frame. See Find blurred, dark, bright or noisy pictures.
• The last value is the pHash, a 64-bit number from the large shapes of the frame. Copies of one picture have almost the same pHash. See Find copies of a picture.
• signals.compute takes the frames from the video module as they are. These frames are already decoded, so they never give an ItemFailure.

The task sections that follow tell more about each step.

Get frames at given times

from curu_vision.media import VideoSource, load_media

media = load_media()  # finds ffmpeg and ffprobe and checks their versions, one time
source = VideoSource(path="episode.mkv")

info = media.probe(source)
print(info.duration, info.video[0].width, info.video[0].height)

frames = media.frames_at(source, [12.0, 30.5], size=(448, 252))
for frame in frames.frames:
    print(frame.time, frame.image.shape)  # the frame's own time, and (252, 448, 3)

every = media.frames_every(source, rate=1.0, start=10.0, end=20.0)
print([frame.time for frame in every.frames])  # one frame each second from 10 s to 19 s

What the results mean:

• probe reads the properties of the file: its streams, duration, frame rates, rotation, HDR and container. It does not decode the frames.
• frames_at gives one frame for each time, in the order of the times. Each frame is the frame on the screen at that time: the last frame that starts at or before it.
• frame.time is the frame's own presentation time in the source, in seconds. It can be a little before the time that you gave. This is also correct for a variable frame rate.
• frame.image is an RGB uint8 array of shape (height, width, 3). It is upright, with the display aspect ratio. size=(width, height) scales it, and with no size it has the display size.
• frames_every gives the frames on the screen at start, start + 1/rate and so on, before end. With no start and end, it starts at the first frame and stops at the end of the video.
• A time outside the video raises TimeOutOfRange, and its message gives the valid range.

The video guide gives the full rules.

Find the scenes in a video

A shot is a continuous run of frames from one camera take. A video is a sequence of shots, and a scene is one shot or a few shots together. media.shots finds the shots. curu-vision does not join shots into scenes.

from curu_vision.media import VideoSource, load_media

media = load_media()
source = VideoSource(path="episode.mkv")

shots = media.shots(source)
for shot in shots.shots:
    print(shot.index, shot.start, shot.end, shot.boundary.kind)  # "start", "cut" or "gradual"

picked = media.shot_frames(source, shots.shots[0], count=3, size=(448, 252))
print([frame.time for frame in picked.frames])  # three frames from the first shot

What the results mean:

• The shots cover the full video with no gap. Each shot ends where the next shot starts, and the last shot ends at the end of the last frame. Times are in seconds in the source.
• shot.boundary.kind tells how the shot starts. It is "start" for the first shot, "cut" for a sudden change and "gradual" for a fade or a dissolve.
• For a gradual boundary, boundary.start to boundary.end is the transition. The new shot starts at the first frame of the transition.
• boundary.score is the change of colours at the boundary, from 0 (the same colours) to 2 (no colours in common). Most cuts have a score of 0.4 or more. A gradual boundary has a score of 0.6 or more. A high score does not always make a cut. Use the score to sort boundaries. It is not a probability.
• shot_frames gives frames from inside the shot only, with its first and last frame. Give count (2 or more, at equal time intervals) or rate (frames each second). With neither, you get every frame of the shot.

Accuracy, measured on public test sets, and speed:

• Cuts are accurate. On 11 BBC Planet Earth episodes, 97 % of the cuts found were real cuts, and it found 95 % of the real cuts.
• Gradual boundaries are less accurate. The RAI set is a public set of Italian television videos with labelled shot boundaries. On RAI videos 6 to 10, 39 % of the gradual boundaries found were real, and it found 85 % of the real ones. On all ten RAI videos, the values were 58 % and 72 %.
• It reads every frame. In a test on 2026-10-25 with a generated 720p video, one call ran about 29 times faster than real time, with 4 CPU threads.

Find copies of a picture

A perceptual hash (pHash) is a 64-bit number that comes from the large shapes of a picture. Two copies of one picture have almost the same bits. The Hamming distance is the number of bits that are different, from 0 to 64.

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.signals import hamming, load_signals

signals = load_signals()  # needs no model and reads no file
a, b = signals.compute(
    [ImageBytes(Path("photo.jpg").read_bytes()), ImageBytes(Path("copy.jpg").read_bytes())]
)
print(hamming(a.hashes.phash, b.hashes.phash))  # 0 to 64: small for a copy
print(f"{a.hashes.phash:016x}")  # the hash as 16 hexadecimal digits

What the distance means in practice:

Frames of a video and of a copy at half size and lower quality were at most 2 bits apart. A copy with 10 % cut from each side was 14 to 20 bits away, so a cropped copy can look unrelated. A mirrored copy was 30 to 32 bits away.

Other values:

• a.hashes.dhash is a second 64-bit hash, the difference hash (dHash). Compare it with hamming too.
• a.hashes.flat is True for a picture of one colour. The hashes of all such pictures are at most 1 bit apart, so they all match each other. Their pHash is 8000000000000000, or 0 for black.

pHash does not work well for all pictures. See Limits.

Find blurred, dark, bright or noisy pictures

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.signals import load_signals

signals = load_signals()
(result,) = signals.compute([ImageBytes(Path("photo.jpg").read_bytes())])
values = result.measurements
print(values.detail, values.directionality, values.blur_direction)  # blur
print(values.dark_clipped, values.bright_clipped, values.contrast_spread)  # exposure, contrast
print(values.noise, values.colourfulness)

Each value is a measurement, not a decision. curu-vision gives no threshold, no verdict and no combined score. Sort your pictures by a value. Or, find the limits that are correct for your pictures.

The values in the table come from a test on 2026-10-27 with three sample photos (cats, a tiger and a football match). The test applied a Gaussian blur, a motion blur of 15 px, and changes of brightness and contrast. The image signals guide gives all the results.

The measurement of detail, directionality, blur_direction and noise uses a smaller copy of the picture. Its longer side is at most 512 px (SignalsConfig(analysis_side=512)). Thus one picture gives almost the same values at different resolutions. A smaller copy has less noise, so compare noise only between pictures of similar size. For a copy smaller than 3 px on a side, these four values are None.

The inputs can be ImageBytes (the bytes of a file), ImageArray (an RGB array) or a Frame from the video module. The image signals guide gives the exact definitions.

Tag a picture or a frame

A tag is one thing that a picture shows, as the model writes it, such as cat, outdoors or blue_background. The tagger is WD14, a model that gives a score from 0 to 1 for each of 9,083 tags. Download it first with python -m curu_vision fetch wd-v1-4-swinv2-tagger-v2.

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.tagging import load_tagger, select_tags

tagger = load_tagger()  # loads the model files that the fetch command downloaded
(tags,) = tagger.tag([ImageBytes(Path("cats.jpg").read_bytes())])
for tag in select_tags(tags, general=0.35, character=0.85):
    print(tag.name, tag.category, round(tag.score, 2))  # cat general 0.93, and more

tags.scores holds the score of each tag in tags.labels. select_tags gives the tags with a score above your thresholds, highest first. You give each threshold, because curu-vision has no default. The model's author uses 0.35 for general tags and 0.85 for characters.

What the values mean in practice:

A frame from the video module is an input too: tagger.tag(frames.frames). Its time goes into the provenance. A tag name is as the model gives it, with underscores. If no score is above your thresholds, the result is an empty list. That is a result, not an error. The tagging guide gives the full rules and more measured values.

Ask a vision-language model about a picture

A vision-language model (VLM) reads pictures and text, and writes text. Use it to describe a picture, to answer a question about it, or to fill a form that you define. This example asks a local Ollama server for the two main colours of a picture. First, start Ollama. Then run ollama pull qwen3-vl:2b-instruct.

from pathlib import Path

from pydantic import BaseModel

from curu_vision.embedding import ImageBytes
from curu_vision.vlm import BlockingVlm, OllamaConfig, VlmRequest


class Colours(BaseModel):  # the form that the reply must fill
    main_colour: str
    second_colour: str


config = OllamaConfig(model="qwen3-vl:2b-instruct", temperature=0.1, num_ctx=4096, attempts=3)
picture = ImageBytes(Path("photo.jpg").read_bytes())
with BlockingVlm(config) as vlm:  # reads the model's digest and capabilities from the server
    request = VlmRequest(prompt="Name the two main colours.", images=(picture,), output=Colours)
    result = vlm.ask(request)
print(result.value)  # main_colour='green' second_colour='blue', for example
print(result.images[0].source_size, result.images[0].sent_size)  # (640, 480) (640, 480)
print(result.usage.prompt_tokens)  # about 1,050 for one picture on Ollama 0.40.11

You give the temperature, because a server's default temperature can make a model add detail that is not in the picture. Each request sends the picture again as a new JPEG, with its longer side at most 1,024 px and no metadata. output is a pydantic class (a subclass of BaseModel). The reply must fill it, else curu-vision sends the request again. It sends it at most attempts times in total. Without output, result.text is the reply text and result.value is None.

What the values mean in practice:

In async code, use async with open_vlm(config) as vlm: and await vlm.ask(request). For a llama.cpp server or vLLM, use OpenAICompatibleConfig. For a cloud model, use CloudConfig with the cloud extra. The VLM guide gives each setting, each server and each error.

Search by meaning

An embedding is a list of 768 numbers (a vector) that represents the meaning of a text or a picture. Texts and pictures with similar meanings have vectors that are near each other. So you can search pictures with a text, or texts with a picture.

from pathlib import Path

from curu_vision.embedding import Document, ImageBytes, Query, load_embedder

embedder = load_embedder()  # one shared instance for each configuration, safe for threads
query, doc, image = embedder.embed(
    [
        Query("cats sleeping on a couch"),
        Document("Two cats asleep on a pink blanket.", title="Cats"),
        ImageBytes(Path("cats.jpg").read_bytes()),
    ]
)
print(query.vector @ doc.vector, query.vector @ image.vector)  # cosine similarities
print(image.provenance)  # the model, its revision and file hashes, the preprocessing, ...

What the results mean:

• Each vector has a length of 1, so a.vector @ b.vector is the cosine similarity. A higher value means a nearer meaning.
• Use the scores to sort results for one query. They are not probabilities. In a test with three photos, a text and the photo that it describes gave 0.70 to 0.75. A text and a different photo gave 0.52 to 0.61.
• Compare scores only between pairs of the same kind. Two photos of different animals gave 0.71, which is near the score of a match between a text and a photo.
• Give a search text as a Query and a text to find as a Document, with an optional title. The model adds a different prompt to each, and the prompt is part of the vector space.
• Give a picture as ImageBytes (the bytes of a file) or ImageArray (a uint8 array). curu-vision never reads a path string as a picture.

EmbeddingConfig sets how the model runs:

• dimension: 768, 512, 256 or 128 numbers. A shorter vector has a length of 1 again.
• image_token_budget and video_token_budget: 70, 140, 280, 560 or 1120 (280 and 140 are the defaults). A larger budget keeps more detail of each picture and takes more time. These two values have the largest effect on speed and quality.
• providers: the default is ("cpu",). ("cuda",) uses an NVIDIA GPU, and ("coreml",) uses Core ML on an Apple computer. A provider that is not there is an error.

Search video, sound and scenes by meaning

The model also makes one vector for a video clip, a sound or a full scene. First, get the sound track as 16 kHz audio with ffmpeg:

ffmpeg -i episode.mkv -vn -ac 1 -ar 16000 -c:a pcm_s16le episode.wav

import wave

import numpy as np

from curu_vision.embedding import Audio, Combined, Query, load_embedder
from curu_vision.media import VideoSource, load_media

media = load_media()
embedder = load_embedder()
source = VideoSource(path="episode.mkv")
shot = media.shots(source).shots[0]
clip = media.shot_frames(source, shot, rate=1.0)  # one frame each second

with wave.open("episode.wav", "rb") as file:  # 16 kHz, 16-bit
    samples = np.frombuffer(file.readframes(file.getnframes()), np.int16)
first, last = int(shot.start * 16_000), int(shot.end * 16_000)
sound = Audio(samples[first:last], sample_rate=16_000, start=shot.start)

query, video, scene = embedder.embed(
    [
        Query("a door bell rings in a busy coffee shop"),
        clip.to_video(),
        Combined((clip.to_video(), *sound.split())),  # picture and sound together
    ]
)
print(query.vector @ video.vector, query.vector @ scene.vector)
print(video.provenance.video.frame_times)  # the source times of the frames that the model used

What the inputs and results mean:

• For a Video, the model uses one frame each second and at most 32 frames, spread over the clip. It selects frames by their own times.
• Give one shot or one scene as a clip. To cut a long video into clips, use media.shots.
• An Audio input must be 16 kHz and from 10 ms to 30 s long. Other rates and longer audio fail with a named error. curu-vision never resamples audio, and only Audio.split() cuts it.
• Audio.split() cuts audio into parts of 30 s or less, each with its start time.
• A Video takes (time, picture) pairs from any decoder. curu-vision reads a generator one time, so make a new Video each time that you use it.
• Combined makes one vector from its parts, in their order. This is the scene vector, which a catalogue can store. About 150 s of sound fits with 32 frames.

The embedding guide gives all the input rules.

Make a transcript

A transcript gives each spoken word with its start and end time. First, download a transcription model (see Download the models). Then get 16 kHz audio with ffmpeg:

ffmpeg -i interview.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le speech.wav

import wave

import numpy as np

from curu_vision.transcription import Audio, Transcript, TranscriptionConfig, load_transcriber

with wave.open("speech.wav", "rb") as file:  # 16 kHz, 16-bit
    samples = np.frombuffer(file.readframes(file.getnframes()), np.int16)

transcriber = load_transcriber()  # Parakeet on the CPU
# load_transcriber(TranscriptionConfig(engine="moss", device="cuda"))  # with speakers, on a GPU
(result,) = transcriber.transcribe([Audio(samples, 16_000, start=0.0)])
if isinstance(result, Transcript):
    print(result.text)
    for word in result.words:
        print(f"{word.start:8.2f} {word.end:8.2f} {word.text}")  # seconds in the source

What the results mean:

• Word times are in seconds in the source. Audio(start=...) is the time of the audio's first sample in the source.
• result.segments holds the segments: the words of one speaker turn. With MOSS, each segment has a speaker label, such as S01. With Parakeet, the speaker is None.
• result.word_timing is "measured" when the engine measured each word's time (Parakeet). It is "spread" when the engine spread the words over their segment (MOSS).
• A transcript with no words means that the engine found no speech. A failure is an ItemFailure with a named error, never an empty transcript.
• Audio of 0.08 s or more is correct, with no maximum length. The engine transcribes long audio in windows and joins them.

The engines:

A missing extra or device is an error when the engine loads. curu-vision never uses a different engine instead. The transcription guide tells how the engine joins long audio and what the other fields mean.

Read a video from bytes, a stream or a URL

A VideoSource is a local path, a URL, the bytes of a file or a stream:

from pathlib import Path

from curu_vision.media import VideoSource, load_media

media = load_media()
data = Path("episode.mkv").read_bytes()
print(media.frames_at(VideoSource(data=data), [12.0]).frames[0].time)  # same as the path

with Path("episode.mkv").open("rb") as file:  # curu-vision reads a stream one time, from the start
    frames = media.frames_every(VideoSource(stream=file), rate=1.0, end=20.0)
print(len(frames.frames))
# VideoSource(url="https://example.com/clip.mp4") reads any URL that ffmpeg can open

• Bytes and streams never go to the disk. Bytes give the same results as their path.
• A stream cannot go back, so curu-vision reads it from the start, one time. Give end to frames_every for a stream.
• An MP4 or MOV file with its index after its frames cannot come from a stream. It fails as NeedsSeekableInput: give its bytes or its path instead.

Process many files in one call

from curu_vision.embedding import ItemFailure
from curu_vision.media import VideoSource, load_media

media = load_media()
results = media.shots_many([VideoSource(path="episode.mkv"), VideoSource(path="broken.mkv")])
for result in results:
    if isinstance(result, ItemFailure):
        print(result.index, type(result.error).__name__, result.error)  # 1 NotAVideo ...
    else:
        print(len(result.shots), "shots")

• A batch call gives one result for each input, in order. An input that fails gives an ItemFailure, and the other inputs continue.
• probe_many and shots_many use a pool of threads. Its size is MediaConfig(workers=...), and the default is the CPU quota.
• signals.compute, tagger.tag, embedder.embed and transcriber.transcribe also take a list. In a test on 2026-10-26 on an x86-64 computer, 4 threads gave the image signals of about 300 pictures of 1,024 × 1,024 px each second.
• The video and image signals calls release the Python lock (the GIL). Thus calls from different threads run at the same time.

Handle errors

curu-vision never hides an error as an empty result. All errors are in curu_vision.errors.

from curu_vision.errors import InputError, TransientError
from curu_vision.media import MediaConfig, VideoSource, load_media

media = load_media(MediaConfig(idle_timeout=120.0, probe_timeout=30.0, wall_timeout=None))
try:
    media.probe(VideoSource(path="broken.mkv"))
except InputError as error:  # the file is bad: correct it or skip it
    print(type(error).__name__, error)  # NotAVideo ...
except TransientError as error:  # the source or the computer failed: try again later
    print(type(error).__name__, error)

A failure of the model, the computer or the runtime stops the full call. There are two exceptions. A batch of the video module gives a TransientError as an ItemFailure. In a transcript batch, InvalidModelOutput or InsufficientMemory for one item gives an ItemFailure for that item only. curu-vision never uses a different model or engine instead.

A video that is cut off after some frames is not an error. You get the frames that decoded, and decoded_until gives the time where they stop.

MediaConfig sets the time limits of each child process of ffmpeg or ffprobe. The example shows the default values. The idle limit stops a child process that reads or writes no byte for that time. Thus a long decode that continues to make progress does not stop. The wall limit, if you set it, limits the full time of each call. The video guide lists each error.

Record what made a result

Each result has a provenance field. It tells exactly what made the result, so you can store it with the result. When a method changes, its version changes, and you can find old results.

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.signals import load_signals

(result,) = load_signals().compute([ImageBytes(Path("photo.jpg").read_bytes())])
record = result.provenance
print(record.hash_method, record.hash_version, record.measure_version, record.package_version)

• An embedding names the model, its revision, the SHA-256 of each file, the package version, the preprocessing version and the configuration. For video and audio, it also gives the times of the frames and samples that the model used.
• A video result names the method and its version, each value in effect, the ffmpeg and ffprobe versions and the identity of the source.
• An image signals result names the hash method and version and the measurement version. It also gives each parameter, the decode version, the package version and the two picture sizes.
• A transcript names the engine, the runtime, the device, the model, its revision and the SHA-256 of each file. It also gives the package and preprocessing versions and the audio parts that the engine used.
• A tag result names the model, its revision, and the SHA-256 of the model file and of the tag list. It also gives the preprocessing, decode, package and ONNX Runtime versions, the picture size, and the time of a frame.
• A vision-language model result names the server, its version, the model and its digest, or says that the server gave no digest. It also gives the model name in the server's reply and each setting that was sent. It gives the version of the picture preparation, the package version and the time of a frame.

Limits

• curu-vision is a library. It does not store results, run a pipeline or keep a queue. Its only command downloads models.
• It does not decode audio. Your program gives it 16 kHz samples, for example from ffmpeg.
• pHash does not find copies of smooth gradients or noise fields. In the tests, copies of these moved 4 to 32 bits.
• pHash does not find copies of pictures smaller than 32 px on a side. A 9 × 8 px picture moved 28 bits at half size.
• pHash does not find cropped or mirrored copies (see Find copies of a picture).
• The shot finder can find false cuts inside slow dissolves and other gradual transitions.
• HDR video from a stream source fails as UnsupportedSource. Give it as bytes, a path or a URL.
• A call decodes one video stream: the first, or the stream that video_stream selects.
• curu-vision has no prompts and no rules for captions or judgements. Your program gives the prompt, the form of the reply and the check.
• curu-vision does not pull, start or download a vision-language model. A llama.cpp server or vLLM gives no digest, so a result from it cannot prove which model file it used.
• Some model servers ignore the context size: Ollama's MLX runner sent a request of 10,105 tokens with a context of 4,096. Look at result.usage.prompt_tokens.
• The tagger knows only the 9,083 tags of its model. A tag that is not in that list has no score, not a score of 0.
• The tagger learned from illustrations (Danbooru tags). On photos it gives tags such as realistic, and some tags are wrong: a tiger photo got signature.
• A tag score is not calibrated: a score of 0.9 does not mean that the tag is correct 9 times in 10. Find the thresholds that are correct for your pictures.

Technical notes

• Build. The package includes a Rust extension, curu_vision._core. maturin, a build tool for Rust extensions, builds it one time for Python 3.12 and later (abi3, the stable Python ABI). uv builds it from the release tag. Each video operation runs ffmpeg or ffprobe as a child process from the Rust core, with limits on its time.
• Decoder. curu-vision uses the system ffmpeg and ffprobe (ADR-009).
• ffmpeg on arm64. Releases before 8.1 convert frames to RGB with too much error on arm64, so load_media() requires 8.1 there (ADR-010).
• Hashes. The hashes are equal to imagehash 4.3.2's phash and dhash on Pillow 13.0.0 with the x86-64 wheels of scipy 1.18.1. The installed versions of these packages have no effect. Rust code does Pillow's luma, Pillow 13's LANCZOS resize and scipy's DCT, bit for bit. f"{h:016x}" is imagehash's str(hash).
• Hashes on arm64. The arm64 wheels of scipy fuse multiply-adds in this DCT. Thus imagehash on arm64 can give different bits for block graphics. curu-vision gives the x86-64 values on every platform.
• Hashes and shots on a Mac. The DCTs take their cosines and sines from glibc's constants, not from the maths library of the platform. Under hash-1, a Mac gave different hashes for some block graphics. Under shots-2, a Mac can give different shots for some videos. Compute each hash whose hash_version is hash-1 again. Compute each shot list whose method_version is shots-2 again.
• Shots. The shot method is shots-3. It finds cuts, fades and dissolves. Its name and all its parameters are in the provenance of each result.
• Embedding. The embedding model is EmbeddingGemma 2, on ONNX Runtime, with no PyTorch when it runs (ADR-002).
• Transcription engines. Parakeet-TDT-0.6b-v3 runs on the CPU with ONNX Runtime, and on Apple silicon with MLX. MOSS-Transcribe-Diarize runs on NVIDIA GPUs (ADR-006). PyTorch and MLX are optional extras (ADR-007).
• Tagging. The tagger is WD14 SwinV2 v2 at a pinned revision, on ONNX Runtime, on the CPU by default. Its input and its scores are equal to its author's reference tagger code. The model reads BGR pixels from 0 to 255, padded to a white square of 448 px.
• Vision-language models. curu-vision reaches Ollama through Ollama's own /api/chat, because Ollama's OpenAI-compatible endpoint ignores think and num_ctx. Other servers use /v1/chat/completions, and cloud models use PydanticAI (ADR-014). The tests use Ollama 0.40.13 with qwen3-vl:2b-instruct and qwen3-vl:2b.
• Type check. The type check runs in a temporary environment with the reference group and the gpu extra (ADR-008).
• API contracts. Each module has an API contract. It gives the exact types, rules and errors of each call:
  • Embedding
  • Video and audio inputs for the embedding
  • Transcription
  • Video probe, frames and shots
  • Image hashes and measurements
  • Tags
  • Vision-language models
• Rules and decisions. The rules for each change are in .specify/memory/constitution.md, and the decisions are in project-management/ADRs/.
• The curu packages. How curu-vision fits with the other curu packages.

Develop

uv sync
uv tool install beacon-framework                  # one time: the beacon command
uv run ruff check && uv run ruff format --check
uv run --isolated --group reference --extra gpu ty check   # in a temporary environment
cargo fmt --check --manifest-path crates/curu-vision-core/Cargo.toml
# Python from uv: pyo3 builds against it (clippy, test); the test binary links uv's
# libpython and finds it at run time (on Linux the bare `cargo test` fails with `unable to
# find library -lpython3.14` or cannot load libpython3.14.so.1.0)
export PYO3_PYTHON="$(uv run python -c 'import sys; print(sys.executable)')"
LIBDIR="$("$PYO3_PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')"
cargo clippy --manifest-path crates/curu-vision-core/Cargo.toml -- -D warnings
RUSTFLAGS="-L $LIBDIR -C link-arg=-Wl,-rpath,$LIBDIR" \
  cargo test --manifest-path crates/curu-vision-core/Cargo.toml
uv run pytest                                     # hermetic: no network, no weights
uv sync --group reference                         # real weights vs the reference, one file a process
for f in tests/integration/test_*.py; do uv run pytest -m integration -s "$f"; done
uv run pytest -m apple                            # the MLX path, on Apple silicon
uv run pytest -m live                             # a running Ollama 0.40.13 (see below)
CURU_VISION_EVAL_SET=/path/to/set uv run pytest -m evaluation  # your labelled or golden set
CURU_VISION_EVAL_SET=/path/to/golden-v1 CURU_VISION_SHOT_SET=/path/to/rai \
  uv run pytest -m evaluation tests/evaluation/test_shots.py   # shot boundaries
beacon docs check                                 # the writing standard, docs/WRITING.md

The hermetic tier needs ffmpeg 6.0 or later, and 8.1 or later on arm64. It also needs the beacon command on the path. Its HDR tests need the zscale filter (libzimg), and they skip without it, as on Homebrew's ffmpeg 8 and 9. To run every gate on Linux, run tests/containers/matrix.sh (Docker). It uses Ubuntu's ffmpeg 6.1.1 (not on arm64) and the 8.1.3 and 9.0.1 releases, each in its own container. See tests/containers/README.md.

The live tier needs Ollama 0.40.13 with qwen3-vl:2b-instruct and qwen3-vl:2b at the digests that the tests name. CURU_VISION_OLLAMA_URL gives its address, else http://localhost:11434. The README example test needs Ollama at http://localhost:11434. A live test fails, and does not skip, when its server is not there. tests/live/test_vlm_openai.py also needs a llama.cpp server. Start it in a second terminal. Then run the test with its address:

llama-server -hf ggml-org/Qwen3-VL-2B-Instruct-GGUF:Q8_0 --ctx-size 4096 --jinja --port 8089
CURU_VISION_OPENAI_URL=http://localhost:8089 uv run pytest -m live tests/live/test_vlm_openai.py

To find whether each ffmpeg build gives the shots of a video the same frame numbers, use tools/shot_frames.py. The section "Measure the shot frame numbers on each ffmpeg build" of tests/containers/README.md gives the procedure.

Write README.md, docs/ and AGENTS.md to the writing standard. AGENTS.md gives the rules for each change.

Glossary

• Analysis size: the smaller copy of a picture that some measurements use. Its longer side is at most 512 px.
• BGR: the order blue, green, red of the three colour values of a pixel. Most files use RGB.
• Boundary: the place where one shot stops and the next shot starts. It is a cut or a gradual transition.
• Category: the kind of a tag: general, character or rating.
• Clipped: a pixel at the end of the brightness range, black or white, that has lost its detail.
• Context: the most tokens that a model reads and writes in one request. Ollama calls it num_ctx.
• Cosine similarity: a measure of how near two vectors are, from -1 to 1. Higher is nearer.
• CPU quota: the number of CPU threads that the container or the computer lets the process use.
• Cut: a sudden change from one shot to the next.
• Danbooru: a website of illustrations whose tags WD14 learned.
• DCT (discrete cosine transform): a calculation that changes a picture into frequencies. pHash uses it.
• dHash (difference hash): a 64-bit hash from the brightness differences of adjacent pixels.
• Digest: the SHA-256 of a model as a model server reports it.
• Edge: a line where the brightness of a picture changes quickly. Blur makes edges weaker.
• Embedding: a vector that represents the meaning of a text, a picture, a video or a sound.
• Event loop: the part of Python's asyncio that runs async code.
• Extra: an optional part of the package that adds dependencies, such as gpu or mlx.
• Gaussian blur: a blur that is equal in all directions. Its size σ is in pixels.
• glibc: the GNU C library, the usual C library on Linux. It gives the same cosines and sines on x86-64 and arm64.
• GIL: Python's global interpreter lock. A call that releases it lets other Python threads run at the same time.
• Gradual transition: a change from one shot to the next over a number of frames, such as a fade or a dissolve.
• Hamming distance: the number of bits that are different between two hashes.
• HDR: high dynamic range video. curu-vision converts it to standard RGB frames.
• ItemFailure: the result for one input that failed in a batch. It holds the input's index and the error.
• LANCZOS: a method to resize a picture.
• JSON schema: a description of the fields and types that a JSON value must have.
• Label: one tag of the model's list, with its category.
• Laplacian: a calculation that finds edges. detail is the variance of the Laplacian of the luma.
• llama.cpp server: a model server that gives an OpenAI-compatible endpoint.
• Luma: the brightness of a pixel, from 0 (black) to 255 (white).
• Model server: a program that runs a model and answers requests over HTTP, such as Ollama.
• Ollama: a model server for local models.
• ONNX Runtime: the library that runs a model file in the ONNX format.
• OpenAI-compatible endpoint: /v1/chat/completions, the request form that many model servers accept.
• pHash (perceptual hash): a 64-bit hash from the large shapes of a picture. Copies have near hashes.
• PydanticAI: a Python library that sends requests to cloud models.
• Presentation time: the time in the source at which a frame shows on the screen.
• Provenance: the record of what made a result: the method, its version, the model and the values in effect.
• Runner: the program inside Ollama that runs a model, such as llama.cpp or MLX.
• Scene: one shot or a few shots together.
• Segment: the words of one speaker turn in a transcript.
• Shot: a continuous run of frames from one camera take.
• Tag: one thing that a picture shows, as the model writes it, such as cat. A tag has a name, a category and a score.
• Temperature: how random a model's reply is. Lower gives replies that change less.
• Threshold: the score that a tag must be above to be selected. You choose it.
• Token budget: the number of tokens that the model uses for one picture. A token is one small part of the input. More tokens keep more detail.
• Vector: a list of numbers. An embedding is a vector.
• vLLM: a model server that gives an OpenAI-compatible endpoint.
• VLM (vision-language model): a model that reads pictures and text, and writes text.
• WD14: the family of tagging models that SmilingWolf trained on Danbooru tags.
• Window: one part of long audio that the engine transcribes in one call.