Durchblick

Durchblick is a middleware for camera streams and perception results. It moves frames between programs in shared memory and in GPU memory, and it uses the message types of ROS 2 without needing ROS.

Why Durchblick

A 1080p colour frame has 6 MB. A publisher that sends it to 16 programs over TCP spends 26 ms of CPU time per frame. Durchblick writes the frame once into shared memory and sends each reader only the location of the frame. This takes about 1 ms for any number of readers.

• Each host stores a frame once. The publisher writes the frame into a ring in shared memory, and all readers on that host work on this one copy. Rings
• Frames stay on the GPU. A GPU producer and a GPU consumer exchange frames in GPU memory, without a copy to the CPU. Frames on the GPU
• A publisher works only on request. It encodes a JPEG only while a subscriber asks for JPEG, and it writes a ring only while a reader has it open. Transports
• Latency stays constant under load. Every stream keeps only its newest frame, so a slow consumer skips frames and never receives old ones. Latest value
• Every consumer uses the same stream. The subscriber chooses the transport. The topic, the message type and the API are the same on one host and across the network, in Python and in C++, on a robot and in a simulation. Streams and topics

Requirements

Linux on x86-64, Python 3.10 to 3.13, and one Zenoh router per PC (xblick router, or zenohd 1.10). An NVIDIA GPU is needed only for frames in GPU memory. C++ programs need CMake and a C++20 compiler.

Install

pip install "xblick[rerun,record]"                      # wheels for Linux x86-64 with glibc 2.28 or newer
xblick router                                           # the router, once per PC (or zenohd, below)

xblick router runs the Zenoh router from the package. A robot's setup may use zenohd instead (zenohd --no-multicast-scouting -l tcp/0.0.0.0:7447), which is not part of the pip packages. It comes from Eclipse Zenoh.

To build from a checkout of this repository, run pip install "./xblick[rerun,record]". The build needs CMake and a C++20 compiler.

Extras and C++: install.

Use

xblick test-camera       # publish a moving test pattern
xblick cam /dev/video0   # a webcam as cam/video0/color (needs xblick[cam], not yet tested on a device)
xblick list              # show every stream
xblick hz test/camera/color
xblick rerun             # show every stream and result in the Rerun viewer
xblick view test/camera/color  # show a stream in VLC or a browser

import xblick as db

session = db.open_session()
images = db.ImageSubscriber(session, "test/camera/color", transport="auto")
frame = images.latest(timeout=1.0)        # the newest frame as a NumPy array

On the publisher's host, transport="auto" reads the frame from a ring. On another host, it receives a JPEG over the network. Your code is the same in both cases.

Continue with your first stream. Without a command, xblick opens a terminal interface with all streams, nodes and parameters.

Write a node

A node reads the streams of a camera and publishes its results below that camera, at <camera>/<node>/<name>. The tutorial your first node builds a complete node in 40 lines. The writing nodes guide lists what Durchblick gives every node: logging, diagnostics, frames as tensors, streams paired by stamp, rate limits and parameters.

Three rules matter most.

1. Stamp each result with the capture time of its input.
2. Name your CUDA stream when you publish an array from GPU memory.
3. Fail with a clear message. Do not continue with a default that hides a wrong configuration.

Nodes

Each perception method is a node in a repository of its own. Every node works on any camera, whether the frames come from a robot, a simulation or a recording.

The node reference has the data flow between the nodes, what each node publishes, and the GPU memory and load of each.

Speed

Measured on an RTX PRO 6000 and an AMD EPYC 9335. Making a node fast explains the method, the changes that helped in several nodes, and the changes that did not help.

Documentation

Coding agents read AGENTS.md and the project skills in .agents/skills.

Status

Version 0.5.3. Durchblick is in active development, and a new minor version can change the protocol. All programs of one setup speak the same protocol. We now released that version. The changelog states which versions do. Programs of 0.5 do not work with programs of 0.3 or 0.4. The nodes are tested in simulation and on recordings. Tests on robots are in progress.

Origin

Durchblick was started at the Computer Vision for Human-Computer Interaction Lab (cv:hci) at KIT and is developed further in cooperation with the High Performance Humanoid Technologies Lab (H2T) at KIT.