curu-vision

What is this

curu-vision is a Python library for programs that work with video, pictures and sound. It gets frames from a video and finds the scenes in it. It finds copies of a picture and measures blur, exposure and noise. It gives tags for pictures, makes vectors for search by meaning and makes transcripts with timed words. It also finds and describes sounds, and asks a vision-language model about a picture. It finds faces and persons in pictures, and gives a vector for each of them. It gives the probability of yes for each yes-or-no question about a picture. The other curu packages use it, so each of these operations has one implementation. Only the model download, a video URL and a vision-language model on a model server or a cloud provider use the network.

What you can do

Install

You need these on the computer that installs the package:

• Python 3.12 or later.
• A Rust toolchain (rustup). The package builds a Rust extension when uv installs it.
• ffmpeg and ffprobe 6.0 or later on the path, and 8.1 or later on arm64. Install them with apt install ffmpeg or brew install ffmpeg. On arm64, Ubuntu 24.04's ffmpeg 6.1 is too old, and Homebrew's ffmpeg is correct. curu-vision does not include or download a decoder.

Add the package to your project by its release tag:

# pyproject.toml
[project]
dependencies = ["curu-vision"]

[tool.uv.sources]
curu-vision = { git = "https://github.com/darth-veitcher/curu-vision", tag = "v0.15.0" }

Two optional extras add transcription engines. Use curu-vision[mlx] for Parakeet on Apple silicon. Use curu-vision[gpu] for MOSS, which needs PyTorch and transformers. The sound event and audio description models also need curu-vision[gpu], on the CPU too. Your project chooses the CUDA build of PyTorch from its own package index.

The yes-or-no questions have two engines. On a computer with a CUDA device, the transformers engine needs curu-vision[gpu]. On other computers, the gguf engine needs llama-server from llama.cpp on the path, for example from brew install llama.cpp.

The extra curu-vision[cloud] adds cloud vision-language models through PydanticAI. You need it only when you name a cloud model. A local model on Ollama or on a llama.cpp server needs no extra.

Download the models

The video module and the image signals module need no model. Tags, faces and persons, search by meaning, transcripts, sounds and yes-or-no questions need model files, which are not in the package. Download each model one time before you use it:

python -m curu_vision fetch embeddinggemma-2      # search by meaning, about 3 GB
python -m curu_vision fetch embeddinggemma-2 --modality text --modality image  # 1.8 GB, no audio
python -m curu_vision fetch parakeet-tdt-0.6b-v3      # transcripts on the CPU, about 2.6 GB
python -m curu_vision fetch parakeet-tdt-0.6b-v3-mlx  # transcripts on Apple silicon, mlx extra
python -m curu_vision fetch moss-transcribe-diarize   # transcripts with speakers, gpu extra
python -m curu_vision fetch wd-v1-4-swinv2-tagger-v2  # tags, about 455 MB
python -m curu_vision fetch d1-3b             # yes-or-no questions on CUDA, about 6.3 GB
python -m curu_vision fetch d1-3b-gguf-q8     # yes-or-no questions with llama-server, 3.5 GB
python -m curu_vision fetch ast-audioset              # sound events, gpu extra, about 346 MB
python -m curu_vision fetch whisper-small-audio-captioning  # descriptions, gpu extra, about 968 MB
python -m curu_vision fetch yunet-2023mar      # face boxes, about 0.2 MB
python -m curu_vision fetch sface-2021dec      # face vectors, about 39 MB
python -m curu_vision fetch rtdetr-v2-r18      # person boxes, gpu extra, about 81 MB
python -m curu_vision fetch dinov3-vits16      # appearance vectors, gpu extra, 86 MB, gated
python -m curu_vision fetch sam3               # person masks and tracks, gpu extra, 3.4 GB, gated
python -m curu_vision fetch insightface-buffalo-l  # other face models, non-commercial only

The download is the only operation that goes to the network for curu-vision itself. Each file has a pinned revision, size and SHA-256, and the command verifies each file. If a download stops, run the same command again. The files go to CURU_VISION_CACHE_DIR, else $XDG_CACHE_HOME/curu-vision, else ~/.cache/curu-vision. When curu-vision loads a model, it never downloads it: a missing model is an error (ModelNotAvailable).

Two of the models are gated: Meta must approve your Hugging Face account on the model page before the download. Ask for access at https://huggingface.co/facebook/dinov3-vits16-pretrain-lvd1689m and https://huggingface.co/facebook/sam3. Then put a token of that account in HF_TOKEN.

A vision-language model is not one of these files. It runs on a model server, and you get it with that server, for example ollama pull qwen3-vl:2b-instruct. curu-vision does not pull or start a model.

Quickstart

This example finds the shots in a video, and then measures the frame at the middle of each shot. It uses the video module and the image signals module. It needs ffmpeg, and it needs no model.

from curu_vision.media import VideoSource, load_media
from curu_vision.signals import load_signals

media = load_media()  # finds ffmpeg and ffprobe and checks their versions, one time
signals = load_signals()  # needs no model
source = VideoSource(path="episode.mkv")

print(media.probe(source).duration)  # the length of the video, in seconds
shots = media.shots(source)
middles = media.frames_at(source, [(shot.start + shot.end) / 2 for shot in shots.shots])
for shot, result in zip(shots.shots, signals.compute(middles.frames)):
    detail, phash = result.measurements.detail, result.hashes.phash
    print(shot.index, shot.start, shot.end, round(detail), f"{phash:016x}")

The README tests run this example on a generated video of 36 s with three shots of 12 s. It prints this:

36.0
0 0.0 12.0 3402 b2bfd084314c1f8f
1 12.0 24.0 3093 93de15d12d245573
2 24.0 36.0 2152 9b932b369705b368

Your video gives different values. The generated frames are small blocks of colour with sharp edges and a little noise. Thus their detail is usually higher than the 1,200 to 2,200 of the sharp test photos in Find blurred, dark, bright or noisy pictures.

What the results mean:

• The first line is the length of the video in seconds.
• Each line after it is one shot: its index, its start and its end in seconds, then two values of the frame at its middle.
• The fourth value, detail, measures fine detail and sharp edges. A lower value usually means a more blurred frame. See Find blurred, dark, bright or noisy pictures.
• The last value is the pHash, a 64-bit number from the large shapes of the frame. Copies of one picture have almost the same pHash. See Find copies of a picture.
• signals.compute takes the frames from the video module as they are. These frames are already decoded, so they never give an ItemFailure.

The task sections that follow tell more about each step.

Get frames at given times

from curu_vision.media import VideoSource, load_media

media = load_media()  # finds ffmpeg and ffprobe and checks their versions, one time
source = VideoSource(path="episode.mkv")

info = media.probe(source)
print(info.duration, info.video[0].width, info.video[0].height)

frames = media.frames_at(source, [12.0, 30.5], size=(448, 252))
for frame in frames.frames:
    print(frame.time, frame.image.shape)  # the frame's own time, and (252, 448, 3)

every = media.frames_every(source, rate=1.0, start=10.0, end=20.0)
print([frame.time for frame in every.frames])  # one frame each second from 10 s to 19 s

What the results mean:

• probe reads the properties of the file: its streams, duration, frame rates, rotation, HDR and container. It does not decode the frames.
• frames_at gives one frame for each time, in the order of the times. Each frame is the frame on the screen at that time: the last frame that starts at or before it.
• frame.time is the frame's own presentation time in the source, in seconds. It can be a little before the time that you gave. This is also correct for a variable frame rate.
• frame.image is an RGB uint8 array of shape (height, width, 3). It is upright, with the display aspect ratio. size=(width, height) scales it, and with no size it has the display size.
• frames_every gives the frames on the screen at start, start + 1/rate and so on, before end. With no start and end, it starts at the first frame and stops at the end of the video.
• A time outside the video raises TimeOutOfRange, and its message gives the valid range.

The video guide gives the full rules.

Find the scenes in a video

A shot is a continuous run of frames from one camera take. A video is a sequence of shots, and a scene is one shot or a few shots together. media.shots finds the shots. curu-vision does not join shots into scenes.

from curu_vision.media import VideoSource, load_media

media = load_media()
source = VideoSource(path="episode.mkv")

shots = media.shots(source)
for shot in shots.shots:
    print(shot.index, shot.start, shot.end, shot.boundary.kind)  # "start", "cut" or "gradual"

picked = media.shot_frames(source, shots.shots[0], count=3, size=(448, 252))
print([frame.time for frame in picked.frames])  # three frames from the first shot

What the results mean:

• The shots cover the full video with no gap. Each shot ends where the next shot starts, and the last shot ends at the end of the last frame. Times are in seconds in the source.
• shot.boundary.kind tells how the shot starts. It is "start" for the first shot, "cut" for a sudden change and "gradual" for a fade or a dissolve.
• For a gradual boundary, boundary.start to boundary.end is the transition. The new shot starts at the first frame of the transition.
• boundary.score is the change of colours at the boundary, from 0 (the same colours) to 2 (no colours in common). Most cuts have a score of 0.4 or more. A gradual boundary has a score of 0.6 or more. A high score does not always make a cut. Use the score to sort boundaries. It is not a probability.
• shot_frames gives frames from inside the shot only, with its first and last frame. Give count (2 or more, at equal time intervals) or rate (frames each second). With neither, you get every frame of the shot.

Accuracy, measured on public test sets, and speed:

• Cuts are accurate. On 11 BBC Planet Earth episodes, 97 % of the cuts found were real cuts, and it found 95 % of the real cuts.
• Gradual boundaries are less accurate. The RAI set is a public set of Italian television videos with labelled shot boundaries. On RAI videos 6 to 10, 39 % of the gradual boundaries found were real, and it found 85 % of the real ones. On all ten RAI videos, the values were 58 % and 72 %.
• It reads every frame. In a test on 2026-10-25 with a generated 720p video, one call ran about 29 times faster than real time, with 4 CPU threads.

Find copies of a picture

A perceptual hash (pHash) is a 64-bit number that comes from the large shapes of a picture. Two copies of one picture have almost the same bits. The Hamming distance is the number of bits that are different, from 0 to 64.

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.signals import hamming, load_signals

signals = load_signals()  # needs no model and reads no file
a, b = signals.compute(
    [ImageBytes(Path("photo.jpg").read_bytes()), ImageBytes(Path("copy.jpg").read_bytes())]
)
print(hamming(a.hashes.phash, b.hashes.phash))  # 0 to 64: small for a copy
print(f"{a.hashes.phash:016x}")  # the hash as 16 hexadecimal digits

What the distance means in practice:

Frames of a video and of a copy at half size and lower quality were at most 2 bits apart. A copy with 10 % cut from each side was 14 to 20 bits away, so a cropped copy can look unrelated. A mirrored copy was 30 to 32 bits away.

Other values:

• a.hashes.dhash is a second 64-bit hash, the difference hash (dHash). Compare it with hamming too.
• a.hashes.ahash is a third 64-bit hash, the average hash (aHash). It is None unless you use load_signals(SignalsConfig(ahash=True)), because it costs one more resize of each picture. It is the same as the average_hash of imagehash. In the test pictures, a JPEG copy at quality 75 and half size was at most 3 bits away. 98.9 % of the unrelated pairs were 20 bits or more apart, and the nearest pair was 10 bits apart. Compare an aHash only with an aHash. In the tests, a dark frame with a little noise also had aHash 0.
• a.hashes.flat is True for a picture of one colour. The hashes of all such pictures are at most 1 bit apart, so they all match each other. Their pHash is 8000000000000000, or 0 for black. Their aHash is 0.

pHash does not work well for all pictures. See Limits.

Find blurred, dark, bright or noisy pictures

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.signals import load_signals

signals = load_signals()
(result,) = signals.compute([ImageBytes(Path("photo.jpg").read_bytes())])
values = result.measurements
print(values.detail, values.directionality, values.blur_direction)  # blur
print(values.dark_clipped, values.bright_clipped, values.contrast_spread)  # exposure, contrast
print(values.noise, values.colourfulness)

Each value is a measurement, not a decision. curu-vision gives no threshold, no verdict and no combined score. Sort your pictures by a value. Or, find the limits that are correct for your pictures.

The values in the table come from a test on 2026-10-27 with three sample photos (cats, a tiger and a football match). The test applied a Gaussian blur, a motion blur of 15 px, and changes of brightness and contrast. The image signals guide gives all the results.

The measurement of detail, directionality, blur_direction and noise uses a smaller copy of the picture. Its longer side is at most 512 px (SignalsConfig(analysis_side=512)). Thus one picture gives almost the same values at different resolutions. A smaller copy has less noise, so compare noise only between pictures of similar size. For a copy smaller than 3 px on a side, these four values are None.

The inputs can be ImageBytes (the bytes of a file), ImageArray (an RGB array) or a Frame from the video module. The image signals guide gives the exact definitions.

Tag a picture or a frame

A tag is one thing that a picture shows, as the model writes it, such as cat, outdoors or blue_background. The tagger is WD14, a model that gives a score from 0 to 1 for each of 9,083 tags. Download it first with python -m curu_vision fetch wd-v1-4-swinv2-tagger-v2.

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.tagging import load_tagger, select_tags

tagger = load_tagger()  # loads the model files that the fetch command downloaded
(tags,) = tagger.tag([ImageBytes(Path("cats.jpg").read_bytes())])
for tag in select_tags(tags, general=0.35, character=0.85):
    print(tag.name, tag.category, round(tag.score, 2))  # cat general 0.93, and more

tags.scores holds the score of each tag in tags.labels. select_tags gives the tags with a score above your thresholds, highest first. You give each threshold, because curu-vision has no default. The model's author uses 0.35 for general tags and 0.85 for characters.

What the values mean in practice:

A frame from the video module is an input too: tagger.tag(frames.frames). Its time goes into the provenance. A tag name is as the model gives it, with underscores. If no score is above your thresholds, the result is an empty list. That is a result, not an error. The tagging guide gives the full rules and more measured values.

Find faces and persons

A detection is one face or one person that a model finds in a picture, with a box and a score. An identity embedding is a vector for one detection. Two vectors of the same person are near each other, so you can group the faces of one person (see the glossary). Download the models first (see Download the models). The person models need the gpu extra.

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.entities import load_face_detector, load_face_embedder, select_detections
from curu_vision.entities import load_appearance_embedder, load_person_detector

picture = ImageBytes(Path("crew.jpg").read_bytes())
(faces,) = load_face_detector().detect([picture])
faces = select_detections(faces, 0.9)  # your threshold: curu-vision has no default
(found,) = load_face_embedder().embed([picture], [faces])
for face in found.embeddings:
    print(face.detection.box, round(face.detection.score, 2))  # Box(left=..., ...) 0.93
first, second = found.embeddings[:2]
print(round(float(first.vector @ second.vector), 2))  # two different people: 0.25

(persons,) = load_person_detector().detect([picture])
(looks,) = load_appearance_embedder().embed([picture], [persons])
print(len(persons.detections), looks.embeddings[0].vector.shape)  # 6 (384,)

detect gives the detections of each picture, highest score first. embed takes the same pictures and their detections, and gives one vector for each detection. Each vector has length 1, so a.vector @ b.vector is the cosine similarity of two detections.

What the values mean in practice, measured on three public-domain crew photographs and their mirrored copies:

curu-vision gives scores and similarities, never a verdict. A person must measure which threshold is correct for your pictures. measured_threshold gives a measured threshold, and it gives None until the first measurement. The faces and persons guide gives the procedure, the masks and tracks of persons in a shot, and the other face models.

The vectors are identity data. curu-vision gives them to your program only: it never logs them, sends them to a server or writes them to a file. It never names a person. Keep the vectors apart from your other data, and remove them when a person asks.

Ask a vision-language model about a picture

A vision-language model (VLM) reads pictures and text, and writes text. Use it to describe a picture, to answer a question about it, or to fill a form that you define. This example asks a local Ollama server for the two main colours of a picture. First, start Ollama. Then run ollama pull qwen3-vl:2b-instruct.

from pathlib import Path

from pydantic import BaseModel

from curu_vision.embedding import ImageBytes
from curu_vision.vlm import BlockingVlm, OllamaConfig, VlmRequest


class Colours(BaseModel):  # the form that the reply must fill
    main_colour: str
    second_colour: str


config = OllamaConfig(model="qwen3-vl:2b-instruct", temperature=0.1, num_ctx=4096, attempts=3)
picture = ImageBytes(Path("photo.jpg").read_bytes())
with BlockingVlm(config) as vlm:  # reads the model's digest and capabilities from the server
    request = VlmRequest(prompt="Name the two main colours.", images=(picture,), output=Colours)
    result = vlm.ask(request)
print(result.value)  # main_colour='green' second_colour='blue', for example
print(result.images[0].source_size, result.images[0].sent_size)  # (640, 480) (640, 480)
print(result.usage.prompt_tokens)  # about 1,050 for one picture on Ollama 0.40.11

You give the temperature, because a server's default temperature can make a model add detail that is not in the picture. Each request sends the picture again as a new JPEG, with its longer side at most 1,024 px and no metadata. output is a pydantic class (a subclass of BaseModel). The reply must fill it, else curu-vision sends the request again. It sends it at most attempts times in total. Without output, result.text is the reply text and result.value is None.

What the values mean in practice:

In async code, use async with open_vlm(config) as vlm: and await vlm.ask(request). For a llama.cpp server or vLLM, use OpenAICompatibleConfig. For a cloud model, use CloudConfig with the cloud extra. The VLM guide gives each setting, each server and each error.

Ask yes-or-no questions about a picture

Use this task to sort or check pictures with questions in words, such as "Is at least one person visible?". The model is LiquidAI d1-3B. For each question, it gives the probability of yes, a number from 0 to 1. Download the model first: python -m curu_vision fetch d1-3b-gguf-q8, or d1-3b on a computer with a CUDA device.

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.questions import Item, load_questions

picture = ImageBytes(Path("photo.jpg").read_bytes())
item = Item(
    pictures=[picture],
    questions={
        "person": "Is at least one person visible?",
        "sharp": "Is the picture sharp and in focus?",
    },
)
with load_questions() as questions:  # transformers with a CUDA device, else llama-server
    (answers,) = questions.ask([item])
for name, probability in answers.probabilities.items():
    print(name, round(probability, 3))
print(answers.provenance.engine, answers.provenance.fingerprints["person"])

An item holds the pictures of one thing, such as some frames of one clip, and its named questions. The model reads all the pictures of an item together. ask takes a list of items and gives one result for each item, in order. Give chunk=4 to ask at most 4 questions in one model call. With no chunk, all the questions of an item go in one call.

What the values mean in practice:

curu-vision gives no threshold and no verdict. You choose the threshold for each question, on your own pictures, because the words of a question change its numbers. With no engine named, the transformers engine runs when PyTorch finds a CUDA device. Else the gguf engine runs. The questions guide gives the speed of each engine, each setting and each error.

Search by meaning

An embedding is a list of 768 numbers (a vector) that represents the meaning of a text or a picture. Texts and pictures with similar meanings have vectors that are near each other. So you can search pictures with a text, or texts with a picture.

from pathlib import Path

from curu_vision.embedding import Document, ImageBytes, Query, load_embedder

embedder = load_embedder()  # one shared instance for each configuration, safe for threads
query, doc, image = embedder.embed(
    [
        Query("cats sleeping on a couch"),
        Document("Two cats asleep on a pink blanket.", title="Cats"),
        ImageBytes(Path("cats.jpg").read_bytes()),
    ]
)
print(query.vector @ doc.vector, query.vector @ image.vector)  # cosine similarities
print(image.provenance)  # the model, its revision and file hashes, the preprocessing, ...

What the results mean:

• Each vector has a length of 1, so a.vector @ b.vector is the cosine similarity. A higher value means a nearer meaning.
• Use the scores to sort results for one query. They are not probabilities. In a test with three photos, a text and the photo that it describes gave 0.70 to 0.75. A text and a different photo gave 0.52 to 0.61.
• Compare scores only between pairs of the same kind. Two photos of different animals gave 0.71, which is near the score of a match between a text and a photo.
• Give a search text as a Query and a text to find as a Document, with an optional title. The model adds a different prompt to each, and the prompt is part of the vector space.
• Give a picture as ImageBytes (the bytes of a file) or ImageArray (a uint8 array). curu-vision never reads a path string as a picture.

EmbeddingConfig sets how the model runs:

• dimension: 768, 512, 256 or 128 numbers. A shorter vector has a length of 1 again.
• image_token_budget and video_token_budget: 70, 140, 280, 560 or 1120 (280 and 140 are the defaults). A larger budget keeps more detail of each picture and takes more time. These two values have the largest effect on speed and quality.
• providers: the default is ("cpu",). ("cuda",) uses an NVIDIA GPU, and ("coreml",) uses Core ML on an Apple computer. A provider that is not there is an error.

Search video, sound and scenes by meaning

The model also makes one vector for a video clip, a sound or a full scene. First, get the sound track as 16 kHz audio with ffmpeg:

ffmpeg -i episode.mkv -vn -ac 1 -ar 16000 -c:a pcm_s16le episode.wav

import wave

import numpy as np

from curu_vision.embedding import Audio, Combined, Query, load_embedder
from curu_vision.media import VideoSource, load_media

media = load_media()
embedder = load_embedder()
source = VideoSource(path="episode.mkv")
shot = media.shots(source).shots[0]
clip = media.shot_frames(source, shot, rate=1.0)  # one frame each second

with wave.open("episode.wav", "rb") as file:  # 16 kHz, 16-bit
    samples = np.frombuffer(file.readframes(file.getnframes()), np.int16)
first, last = int(shot.start * 16_000), int(shot.end * 16_000)
sound = Audio(samples[first:last], sample_rate=16_000, start=shot.start)

query, video, scene = embedder.embed(
    [
        Query("a door bell rings in a busy coffee shop"),
        clip.to_video(),
        Combined((clip.to_video(), *sound.split())),  # picture and sound together
    ]
)
print(query.vector @ video.vector, query.vector @ scene.vector)
print(video.provenance.video.frame_times)  # the source times of the frames that the model used

What the inputs and results mean:

• For a Video, the model uses one frame each second and at most 32 frames, spread over the clip. It selects frames by their own times.
• Give one shot or one scene as a clip. To cut a long video into clips, use media.shots.
• An Audio input must be 16 kHz and from 10 ms to 30 s long. Other rates and longer audio fail with a named error. curu-vision never resamples audio, and only Audio.split() cuts it.
• Audio.split() cuts audio into parts of 30 s or less, each with its start time.
• A Video takes (time, picture) pairs from any decoder. curu-vision reads a generator one time, so make a new Video each time that you use it.
• Combined makes one vector from its parts, in their order. This is the scene vector, which a catalogue can store. About 150 s of sound fits with 32 frames.

The embedding guide gives all the input rules.

Make a transcript

A transcript gives each spoken word with its start and end time. First, download a transcription model (see Download the models). Then get 16 kHz audio with ffmpeg:

ffmpeg -i interview.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le speech.wav

import wave

import numpy as np

from curu_vision.transcription import Audio, Transcript, TranscriptionConfig, load_transcriber

with wave.open("speech.wav", "rb") as file:  # 16 kHz, 16-bit
    samples = np.frombuffer(file.readframes(file.getnframes()), np.int16)

transcriber = load_transcriber()  # Parakeet on the CPU
# load_transcriber(TranscriptionConfig(engine="moss", device="cuda"))  # with speakers, on a GPU
(result,) = transcriber.transcribe([Audio(samples, 16_000, start=0.0)])
if isinstance(result, Transcript):
    print(result.text)
    for word in result.words:
        print(f"{word.start:8.2f} {word.end:8.2f} {word.text}")  # seconds in the source

What the results mean:

• Word times are in seconds in the source. Audio(start=...) is the time of the audio's first sample in the source.
• result.segments holds the segments: the words of one speaker turn. With MOSS, each segment has a speaker label, such as S01. With Parakeet, the speaker is None.
• result.word_timing is "measured" when the engine measured each word's time (Parakeet). It is "spread" when the engine spread the words over their segment (MOSS).
• A transcript with no words means that the engine found no speech. A failure is an ItemFailure with a named error, never an empty transcript.
• Audio of 0.08 s or more is correct, with no maximum length. The engine transcribes long audio in windows and joins them.

The engines:

A missing extra or device is an error when the engine loads. curu-vision never uses a different engine instead. The transcription guide tells how the engine joins long audio and what the other fields mean.

Find sounds in a soundtrack

A sound event is a class of sound, such as Speech, Music, Dog or Laughter, from the 527 classes of AudioSet. The sound event model gives each class a score in each window of 10.255 s. The description model writes one sentence about each window of 30 s. Both need the gpu extra. Download them first (see Download the models). Get 16 kHz audio as in Make a transcript.

import wave

import numpy as np

from curu_vision.audio_descriptions import load_describer, measured_categories, select_claims
from curu_vision.sound_events import Audio, load_sound_tagger, measured_thresholds
from curu_vision.sound_events import select_sound_events

with wave.open("speech.wav", "rb") as file:  # 16 kHz, 16-bit
    audio = Audio(np.frombuffer(file.readframes(file.getnframes()), np.int16), 16_000)

(events,) = load_sound_tagger().tag([audio])
for window in events.windows:
    if not window.silent:
        best = np.argsort(-window.scores)[:3]
        print(window.start, window.end, [events.classes[i] for i in best])  # Speech, ...
print(select_sound_events(events, measured_thresholds()))  # measured classes only

(described,) = load_describer().describe([audio])
for description in described.descriptions:
    print(description.start, description.end, description.text)  # A man speaks
print(select_claims(described, measured_categories()))  # measured claims only

What the results mean:

• window.scores[i] is the score of the class events.classes[i], from 0 to 1. Higher means more evidence. It is not a probability that a person checked. A 440 Hz tone gave Sine wave 0.88, and most other classes stay near 0.
• A description is the model's sentence, and description.claims holds its parts, each with a category such as music, speech or animal. A description never gives the spoken words.
• A window whose every 100 ms part is below -70 dBFS is silent (see the glossary). It has no scores and no text, because both models invent sounds in silence.
• A person must measure how often each class or category is correct, by ear. measured_thresholds() and measured_categories() give only the measured ones. The tables are empty until the first measurement, so the two selections give empty lists.

The sound guide gives the categories, the failures and the procedure to measure the precision on your own audio.

Read a video from bytes, a stream or a URL

A VideoSource is a local path, a URL, the bytes of a file or a stream:

from pathlib import Path

from curu_vision.media import VideoSource, load_media

media = load_media()
data = Path("episode.mkv").read_bytes()
print(media.frames_at(VideoSource(data=data), [12.0]).frames[0].time)  # same as the path

with Path("episode.mkv").open("rb") as file:  # curu-vision reads a stream one time, from the start
    frames = media.frames_every(VideoSource(stream=file), rate=1.0, end=20.0)
print(len(frames.frames))
# VideoSource(url="https://example.com/clip.mp4") reads any URL that ffmpeg can open

• Bytes and streams never go to the disk. Bytes give the same results as their path.
• A stream cannot go back, so curu-vision reads it from the start, one time. Give end to frames_every for a stream.
• An MP4 or MOV file with its index after its frames cannot come from a stream. It fails as NeedsSeekableInput: give its bytes or its path instead.

Process many files in one call

from curu_vision.embedding import ItemFailure
from curu_vision.media import VideoSource, load_media

media = load_media()
results = media.shots_many([VideoSource(path="episode.mkv"), VideoSource(path="broken.mkv")])
for result in results:
    if isinstance(result, ItemFailure):
        print(result.index, type(result.error).__name__, result.error)  # 1 NotAVideo ...
    else:
        print(len(result.shots), "shots")

• A batch call gives one result for each input, in order. An input that fails gives an ItemFailure, and the other inputs continue.
• probe_many and shots_many use a pool of threads. Its size is MediaConfig(workers=...), and the default is the CPU quota.
• MediaConfig(hwaccel="videotoolbox") or hwaccel="cuda" decodes the video on a hardware decoder, with the same frames and shots. On a Mac, shots on 1080p at a high bit rate took less than half the time.
• signals.compute, tagger.tag, embedder.embed and transcriber.transcribe also take a list. In a test on 2026-10-26 on an x86-64 computer, 4 threads gave the image signals of about 300 pictures of 1,024 × 1,024 px each second.
• The video and image signals calls release the Python lock (the GIL). Thus calls from different threads run at the same time.

Handle errors

curu-vision never hides an error as an empty result. All errors are in curu_vision.errors.

from curu_vision.errors import InputError, TransientError
from curu_vision.media import MediaConfig, VideoSource, load_media

media = load_media(MediaConfig(idle_timeout=120.0, probe_timeout=30.0, wall_timeout=None))
try:
    media.probe(VideoSource(path="broken.mkv"))
except InputError as error:  # the file is bad: correct it or skip it
    print(type(error).__name__, error)  # NotAVideo ...
except TransientError as error:  # the source or the computer failed: try again later
    print(type(error).__name__, error)

A failure of the model, the computer or the runtime stops the full call. There are three exceptions. A batch of the video module gives a TransientError as an ItemFailure. In a transcript batch, InvalidModelOutput or InsufficientMemory for one item gives an ItemFailure for that item only. In a description batch, InvalidModelOutput gives an ItemFailure for that item only. curu-vision never uses a different model or engine instead.

A video that is cut off after some frames is not an error. You get the frames that decoded, and decoded_until gives the time where they stop.

MediaConfig sets the time limits of each child process of ffmpeg or ffprobe. The example shows the default values. The idle limit stops a child process that reads or writes no byte for that time. Thus a long decode that continues to make progress does not stop. The wall limit, if you set it, limits the full time of each call. The video guide lists each error.

Record what made a result

Each result has a provenance field. It tells exactly what made the result, so you can store it with the result. When a method changes, its version changes, and you can find old results.

from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.signals import load_signals

(result,) = load_signals().compute([ImageBytes(Path("photo.jpg").read_bytes())])
record = result.provenance
print(record.hash_method, record.hash_version, record.measure_version, record.package_version)

• An embedding names the model, its revision, the SHA-256 of each file, the package version, the preprocessing version and the configuration. For video and audio, it also gives the times of the frames and samples that the model used.
• A video result names the method and its version, each value in effect, the ffmpeg and ffprobe versions and the identity of the source.
• An image signals result names the hash method and version and the measurement version. It also gives each parameter, the decode version, the package version and the two picture sizes.
• A transcript names the engine, the runtime, the device, the model, its revision and the SHA-256 of each file. It also gives the package and preprocessing versions and the audio parts that the engine used.
• A sound event or description result names the model, its revision and the SHA-256 of each file. It also gives the PyTorch and transformers versions, the device, the threads, the package and method versions and the window length. A description also gives the style prefix and the claims version.
• A tag result names the model, its revision, and the SHA-256 of the model file and of the tag list. It also gives the preprocessing, decode, package and ONNX Runtime versions, the picture size, and the time of a frame.
• A vision-language model result names the server, its version, the model and its digest, or says that the server gave no digest. It also gives the model name in the server's reply and each setting that was sent. It gives the version of the picture preparation, the package version and the time of a frame.
• A yes-or-no answer names the model, its revision, the SHA-256 of each file, the engine, its runtime and its number format. It also gives the version of the picture preparation, the package version and a fingerprint of each question text.
• A face or person result names the model, its revision, the SHA-256 of each file and the licence of the weights. It also gives the runtime and its versions, the device and the number format. It gives the floor of the detector, the preprocessing and package versions, the picture size and the time of a frame. An embedding also names the detector of the detections that it embeds.

Limits

• curu-vision is a library. It does not store results, run a pipeline or keep a queue. Its only command downloads models.
• It does not decode audio. Your program gives it 16 kHz samples, for example from ffmpeg.
• pHash does not find copies of smooth gradients or noise fields. In the tests, copies of these moved 4 to 32 bits.
• The aHash of a dark frame with a little noise can be 0, the same as a picture of one colour. To find the difference between such frames, use flat and the measurements.
• pHash does not find copies of pictures smaller than 32 px on a side. A 9 × 8 px picture moved 28 bits at half size.
• pHash does not find cropped or mirrored copies (see Find copies of a picture).
• The shot finder can find false cuts inside slow dissolves and other gradual transitions.
• HDR video from a stream source fails as UnsupportedSource. Give it as bytes, a path or a URL.
• A call decodes one video stream: the first, or the stream that video_stream selects.
• curu-vision has no prompts and no rules for captions or judgements. Your program gives the prompt, the form of the reply and the check.
• curu-vision does not pull, start or download a vision-language model. A llama.cpp server or vLLM gives no digest, so a result from it cannot prove which model file it used.
• Some model servers ignore the context size: Ollama's MLX runner sent a request of 10,105 tokens with a context of 4,096. Look at result.usage.prompt_tokens.
• The tagger knows only the 9,083 tags of its model. A tag that is not in that list has no score, not a score of 0.
• The tagger learned from illustrations (Danbooru tags). On photos it gives tags such as realistic, and some tags are wrong: a tiger photo got signature.
• A tag score is not calibrated: a score of 0.9 does not mean that the tag is correct 9 times in 10. Find the thresholds that are correct for your pictures.
• A probability of yes is not calibrated either. The two engines give slightly different numbers, so check a threshold again when you change the engine.
• The transformers engine runs only on CUDA. On a CPU it is too slow: about 2 minutes for each item on an Apple M1. The gguf engine took 15 to 27 seconds for one call on an Apple M1 that ran other work at the same time.
• The questions take only yes-or-no questions. For a free-text answer, use a vision-language model.
• A sound event score is not calibrated either. Use only the classes and claim categories that a person measured by ear on audio like yours.
• The description model can describe a sound that is not there, and it writes English only. Its licence, CC BY-NC 4.0, does not let you use it for commercial work.
• The face detector sees each picture at 640 px on its longer side. It can miss a face that is only a few pixels high in a large picture.
• An appearance vector is weaker than a face vector: two different people in similar clothes can be near each other. Use it with the faces.
• The person masks and tracks need a CUDA GPU for a usable speed. A track ends at the end of its shot: the same person in the next shot gets a new track id.
• curu-vision never names a person and never compares a vector with a list of known people.
• The InsightFace face models are for non-commercial research only. The DINOv3 and SAM 3 licences do not let you use them for military, weapons, nuclear or espionage work.

Technical notes

• Build. The package includes a Rust extension, curu_vision._core. maturin, a build tool for Rust extensions, builds it one time for Python 3.12 and later (abi3, the stable Python ABI). uv builds it from the release tag. Each video operation runs ffmpeg or ffprobe as a child process from the Rust core, with limits on its time.
• Decoder. curu-vision uses the system ffmpeg and ffprobe (ADR-009).
• ffmpeg on arm64. Releases before 8.1 convert frames to RGB with too much error on arm64, so load_media() requires 8.1 there (ADR-010).
• Hashes. The hashes are equal to imagehash 4.3.2's phash, dhash and average_hash on Pillow 13.0.0 with the x86-64 wheels of scipy 1.18.1. The installed versions of these packages have no effect. Rust code does Pillow's luma, Pillow 13's LANCZOS resize and scipy's DCT, bit for bit. f"{h:016x}" is imagehash's str(hash).
• Hashes on arm64. The arm64 wheels of scipy fuse multiply-adds in this DCT. Thus imagehash on arm64 can give different bits for block graphics. curu-vision gives the x86-64 values on every platform.
• Hashes and shots on a Mac. The DCTs take their cosines and sines from glibc's constants, not from the maths library of the platform. Under hash-1, a Mac gave different hashes for some block graphics. Under shots-2, a Mac can give different shots for some videos. Compute each hash whose hash_version is hash-1 again. Compute each shot list whose method_version is shots-2 again.
• Hash version. The hash version is hash-2, with the aHash on or off. The aHash changes no hash-2 rule. The provenance parameter ahash tells if it is on.
• Shots. The shot method is shots-3. It finds cuts, fades and dissolves. Its name and all its parameters are in the provenance of each result.
• Embedding. The embedding model is EmbeddingGemma 2, on ONNX Runtime, with no PyTorch when it runs (ADR-002).
• Transcription engines. Parakeet-TDT-0.6b-v3 runs on the CPU with ONNX Runtime, and on Apple silicon with MLX. MOSS-Transcribe-Diarize runs on NVIDIA GPUs (ADR-006). PyTorch and MLX are optional extras (ADR-007).
• Tagging. The tagger is WD14 SwinV2 v2 at a pinned revision, on ONNX Runtime, on the CPU by default. Its input and its scores are equal to its author's reference tagger code. The model reads BGR pixels from 0 to 255, padded to a white square of 448 px.
• Sounds. The sound event model is AST, and the description model is MU-NLPC's Whisper audio captioner, each at a pinned revision, on PyTorch. On an Apple M1, PyTorch ran AST 1.8 to 3.4 times as fast as ONNX Runtime. The description model gets the style prefix audiocaps > caption: before each answer. It takes the most probable token at each step (greedy decoding), to 100 tokens, as its model card says (ADR-007).
• Vision-language models. curu-vision reaches Ollama through Ollama's own /api/chat, because Ollama's OpenAI-compatible endpoint ignores think and num_ctx. Other servers use /v1/chat/completions, and cloud models use PydanticAI (ADR-014). The tests use Ollama 0.40.13 with qwen3-vl:2b-instruct and qwen3-vl:2b.
• Yes-or-no questions. The model is LiquidAI d1-3B at a pinned revision. The transformers engine runs it in bf16 on CUDA with the model's own code (trust_remote_code). The gguf engine starts llama-server as a child process with the pinned Q8_0 files, and uses its /v1/systemone endpoint. The tests used llama.cpp build 11960 (ADR-015).
• Type check. The type check runs in a temporary environment with the reference group and the gpu extra (ADR-008).
• API contracts. Each module has an API contract. It gives the exact types, rules and errors of each call:
  • Embedding
  • Video and audio inputs for the embedding
  • Transcription
  • Video probe, frames and shots
  • Image hashes and measurements
  • Tags
  • Vision-language models
  • Yes-or-no questions
  • Sound events and audio descriptions
  • Faces and persons. The model choices are ADR-016 to ADR-019 in project-management/ADRs/. ADR-019 (SAM 3) is Accepted. ADR-016 to ADR-018 stay Proposed until a person measures the candidates on judged pictures.
• Rules and decisions. The rules for each change are in .specify/memory/constitution.md, and the decisions are in project-management/ADRs/.
• The curu packages. How curu-vision fits with the other curu packages.

Develop

uv sync
uv tool install beacon-framework                  # one time: the beacon command
uv run ruff check && uv run ruff format --check
uv run --isolated --group reference --extra gpu ty check   # in a temporary environment
cargo fmt --check --manifest-path crates/curu-vision-core/Cargo.toml
# Python from uv: pyo3 builds against it (clippy, test); the test binary links uv's
# libpython and finds it at run time (on Linux the bare `cargo test` fails with `unable to
# find library -lpython3.14` or cannot load libpython3.14.so.1.0)
export PYO3_PYTHON="$(uv run python -c 'import sys; print(sys.executable)')"
LIBDIR="$("$PYO3_PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')"
cargo clippy --manifest-path crates/curu-vision-core/Cargo.toml -- -D warnings
RUSTFLAGS="-L $LIBDIR -C link-arg=-Wl,-rpath,$LIBDIR" \
  cargo test --manifest-path crates/curu-vision-core/Cargo.toml
uv run pytest                                     # hermetic: no network, no weights
uv sync --group reference                         # real weights vs the reference, one file a process
for f in tests/integration/test_*.py; do uv run pytest -m integration -s "$f"; done
uv run pytest -m apple                            # the MLX path, on Apple silicon
uv run pytest -m live                             # a running Ollama 0.40.13 (see below)
CURU_VISION_EVAL_SET=/path/to/set uv run pytest -m evaluation  # your labelled or golden set
CURU_VISION_EVAL_SET=/path/to/golden-v1 CURU_VISION_SHOT_SET=/path/to/rai \
  uv run pytest -m evaluation tests/evaluation/test_shots.py   # shot boundaries
beacon docs check                                 # the writing standard, docs/WRITING.md

The hermetic tier needs ffmpeg 6.0 or later, and 8.1 or later on arm64. It also needs the beacon command on the path. Its HDR tests need the zscale filter (libzimg), and they skip without it, as on Homebrew's ffmpeg 8 and 9. To run every gate on Linux, run tests/containers/matrix.sh (Docker). It uses Ubuntu's ffmpeg 6.1.1 (not on arm64) and the 8.1.3 and 9.0.1 releases, each in its own container. See tests/containers/README.md.

The live tier needs Ollama 0.40.13 with qwen3-vl:2b-instruct and qwen3-vl:2b at the digests that the tests name. CURU_VISION_OLLAMA_URL gives its address, else http://localhost:11434. The README example test needs Ollama at http://localhost:11434. A live test fails, and does not skip, when its server is not there. tests/live/test_vlm_openai.py also needs a llama.cpp server. Start it in a second terminal. Then run the test with its address:

llama-server -hf ggml-org/Qwen3-VL-2B-Instruct-GGUF:Q8_0 --ctx-size 4096 --jinja --port 8089
CURU_VISION_OPENAI_URL=http://localhost:8089 uv run pytest -m live tests/live/test_vlm_openai.py

To find whether each ffmpeg build gives the shots of a video the same frame numbers, use tools/shot_frames.py. The section "Measure the shot frame numbers on each ffmpeg build" of tests/containers/README.md gives the procedure.

Write README.md, docs/ and AGENTS.md to the writing standard. AGENTS.md gives the rules for each change.

Glossary

• aHash (average hash): a 64-bit hash that compares each pixel of an 8 × 8 copy of a picture with the mean brightness.
• Analysis size: the smaller copy of a picture that some measurements use. Its longer side is at most 512 px.
• Appearance vector: the identity embedding of a whole person, from DINOv3. It is for a person whose face does not show, such as a character in a costume.
• AudioSet: Google's list of 527 classes of sound, such as Speech, Music and Dog.
• Audio description: one sentence that a model writes about the sounds in a window.
• bf16: a 16-bit form of floating-point numbers that a GPU calculates quickly. It keeps less precision than 32 bits.
• BGR: the order blue, green, red of the three colour values of a pixel. Most files use RGB.
• Boundary: the place where one shot stops and the next shot starts. It is a cut or a gradual transition.
• Category: the kind of a tag (general, character or rating), or the kind of a claim (such as music, speech or animal).
• Claim: one part of an audio description, such as "a dog barks", with its category.
• Box: the rectangle around one detection, in pixels of the upright picture: left, top, right and bottom.
• Clipped: a pixel at the end of the brightness range, black or white, that has lost its detail.
• Context: the most tokens that a model reads and writes in one request. Ollama calls it num_ctx.
• Cosine similarity: a measure of how near two vectors are, from -1 to 1. Higher is nearer.
• CPU quota: the number of CPU threads that the container or the computer lets the process use.
• CUDA: NVIDIA's system to run calculations on a GPU.
• Cut: a sudden change from one shot to the next.
• Danbooru: a website of illustrations whose tags WD14 learned.
• dBFS: decibels below full scale, the level of a sound. 0 dBFS is the loudest sample, and -70 dBFS is almost silent.
• DCT (discrete cosine transform): a calculation that changes a picture into frequencies. pHash uses it.
• dHash (difference hash): a 64-bit hash from the brightness differences of adjacent pixels.
• Digest: the SHA-256 of a model as a model server reports it.
• Detection: one face or one person that a model finds in a picture, with a box and a score.
• Edge: a line where the brightness of a picture changes quickly. Blur makes edges weaker.
• Embedding: a vector that represents the meaning of a text, a picture, a video or a sound.
• Event loop: the part of Python's asyncio that runs async code.
• Extra: an optional part of the package that adds dependencies, such as gpu or mlx.
• Fingerprint: a short hash of a question text. A change of one character changes it.
• Floor: the score at or below which a detector removes a candidate. It is 0.5 for the face detectors and 0.3 for RT-DETRv2 and SAM 3.
• Gaussian blur: a blur that is equal in all directions. Its size σ is in pixels.
• GGUF: the file format of a model for llama.cpp. Q8_0 is a GGUF with 8 bits for each weight.
• Gated model: a model whose download needs the approval of its publisher on the model page, and a token of the approved account in HF_TOKEN.
• glibc: the GNU C library, the usual C library on Linux. It gives the same cosines and sines on x86-64 and arm64.
• GIL: Python's global interpreter lock. A call that releases it lets other Python threads run at the same time.
• Gradual transition: a change from one shot to the next over a number of frames, such as a fade or a dissolve.
• Hamming distance: the number of bits that are different between two hashes.
• HDR: high dynamic range video. curu-vision converts it to standard RGB frames.
• Identity data: the vectors that tell one person from another. curu-vision gives them to your program only.
• Identity embedding: a vector of length 1 for one face or one person. Two vectors of the same person have a high cosine similarity.
• Item: the pictures of one thing and its named questions, which ask answers together.
• ItemFailure: the result for one input that failed in a batch. It holds the input's index and the error.
• LANCZOS: a method to resize a picture.
• JSON schema: a description of the fields and types that a JSON value must have.
• Label: one tag of the model's list, with its category.
• Laplacian: a calculation that finds edges. detail is the variance of the Laplacian of the luma.
• Landmarks: five points of a face: the two eyes, the nose and the two corners of the mouth.
• llama.cpp server: a model server that gives an OpenAI-compatible endpoint. Its program is llama-server.
• Luma: the brightness of a pixel, from 0 (black) to 255 (white).
• Mask: the pixels of one person in a picture, as True and False values.
• Model server: a program that runs a model and answers requests over HTTP, such as Ollama.
• Ollama: a model server for local models.
• ONNX Runtime: the library that runs a model file in the ONNX format.
• OpenAI-compatible endpoint: /v1/chat/completions, the request form that many model servers accept.
• pHash (perceptual hash): a 64-bit hash from the large shapes of a picture. Copies have near hashes.
• PydanticAI: a Python library that sends requests to cloud models.
• Precision: the part of a model's outputs of one class that a person found correct.
• Presentation time: the time in the source at which a frame shows on the screen.
• Probability of yes: the number from 0 to 1 that d1-3B gives for a yes-or-no question. It is the model's own number, not a verdict.
• Provenance: the record of what made a result: the method, its version, the model and the values in effect.
• Runner: the program inside Ollama that runs a model, such as llama.cpp or MLX.
• Scene: one shot or a few shots together.
• Segment: the words of one speaker turn in a transcript.
• Shot: a continuous run of frames from one camera take.
• Silent window: a window of audio whose every 100 ms part is below -70 dBFS. No model sees it.
• Sound event: a class of sound that a model finds in a window, with a score.
• Style prefix: the text audiocaps > caption: that the description model gets before each answer, so that it writes a sentence.
• SAM 3: Meta's model that finds the pixels of each thing that a text names, such as person. It works on a picture and on the frames of a shot.
• Tag: one thing that a picture shows, as the model writes it, such as cat. A tag has a name, a category and a score.
• Track: the masks of one person in the frames of one shot, with one track id.
• Temperature: how random a model's reply is. Lower gives replies that change less.
• Threshold: the score that a tag, a sound event or a detection must be above to be selected. For two vectors, it is the similarity that they must be above. You choose it.
• Token budget: the number of tokens that the model uses for one picture. A token is one small part of the input. More tokens keep more detail.
• Vector: a list of numbers. An embedding is a vector.
• vLLM: a model server that gives an OpenAI-compatible endpoint.
• VLM (vision-language model): a model that reads pictures and text, and writes text.
• WD14: the family of tagging models that SmilingWolf trained on Danbooru tags.
• Window: one part of the audio that a model reads in one call. It is 10.255 s for sound events and 30 s for descriptions.