ataraxis-video-system

Interfaces with a wide range of cameras to flexibly record visual stream data as video files.

Detailed Description

This library abstracts all necessary steps for acquiring and saving video data. During each runtime, it interfaces with one or more cameras to grab the raw frames and encodes them as video files stored in the non-volatile memory. The library is specifically designed for working with multiple cameras at the same time and supports fine-tuning the acquisition and saving parameters to precisely balance the resultant video quality and real-time throughput for a wide range of applications. This library is part of the Ataraxis framework for AI-assisted scientific hardware control.

Features

• Supports Windows, Linux, and macOS, with the GenICam camera interface covering Windows, Linux, and Apple Silicon.
• Uses OpenCV or GenICam (Harvesters) to interface with a wide range of consumer, industrial, and scientific cameras.
• Uses FFMPEG to efficiently encode acquired data as videos in real time using CPU or GPU.
• Exposes encoder, preset, pixel format, and quantization parameters for tuning quality against throughput.
• Supports inspecting, modifying, saving, and loading GenICam camera configurations for reproducible setups.
• Provides a log data processing pipeline for extracting frame acquisition timestamps from runtime log archives, with post-processing tools for frame timing analysis and frame drop detection.
• Generates camera manifest files that tag DataLogger output directories with source-to-name mappings, enabling downstream tools to identify which log archives were produced by ataraxis-video-system.
• Includes an MCP server for AI agent integration (compatible with Claude Desktop and other MCP clients).
• Apache 2.0 License.

Table of Contents

• Dependencies
• Installation
• Usage
  • OS Support Status
  • Quickstart
  • Data Logging
  • Log Processing
  • CLI Commands
  • MCP Server
  • Using GenICam Compatible Cameras
  • GenICam Configuration
• API Documentation
• Developers
• Versioning
• Authors
• License
• Acknowledgments

Dependencies

• FFMPEG version n9.0.1. The installed FFMPEG must be available on the system’s path and callable from Python processes.
• A GenTL Producer (PDF) interface compatible with the Harvesters library if the target camera requires the 'harvesters' camera interface. It is recommended to use the CTI interface supplied by the camera’s vendor, if possible, as this typically ensures that the camera performs as advertised. If the camera-specific CTI file is not available, it is possible to instead use a general interface, such as MvImpactAcquire. This library has been tested using MvImpactAcquire version 2.9.2.

For users, all other library dependencies are installed automatically by all supported installation methods. For developers, see the Developers section for information on installing additional development dependencies.

Installation

Source

Note, installation from source is highly discouraged for anyone who is not an active project developer.

1. Download this repository to the local machine using the preferred method, such as git-cloning. Use one of the stable releases that include precompiled binary and source code distribution (sdist) wheels.
2. If the downloaded distribution is stored as a compressed archive, unpack it using the appropriate decompression tool.
3. cd to the root directory of the prepared project distribution.
4. Run pip install . to install the project and its dependencies.

pip

Use the following command to install the library and all of its dependencies via pip: pip install ataraxis-video-system

Usage

OS Support Status

While this library works on all major operating systems, it is largely up to the maintainers of the low-level library components (OpenCV, Harvesters, FFMPEG) to ensure that the operation is smooth on each supported OS.

Linux

This library was primarily written on and for Linux systems. It is extensively tested on Linux and performs well under all test conditions. Linux users rarely encounter issues specific to this library.

Windows

The library is mostly stable on Windows systems, but requires additional setup to ensure smooth operation. First, FFMPEG has to be updated to the latest stable version, as older versions may have a drastically reduced encoding speed even with hardware acceleration. Some of OpenCV’s advanced features also have to be disabled to support smooth runtimes on the Windows platform. The library disables the MSMF HW transformations automatically, exporting OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS=0 when it is imported, so that feature requires no user action. For any remaining camera-specific feature, information on which features to disable is readily available from OpenCV’s Windows community.

macOS

macOS mostly works as expected except for live frame displaying, which does not work for modern macOS devices. The issue is due to the OS restriction on drawing certain GUI elements outside the main thread of the application. The restriction interferes with the library, as it displays the acquired frames from the same process that interfaces with the camera to minimize the visual lag between grabbing and displaying the frame. The restriction is imposed by the operating system and applies to every application.

Note, the GenICam ('harvesters') camera interface covers Apple Silicon Macs running Python 3.12 or 3.13, which is the only combination the genicam distribution publishes a macOS wheel for. That wheel targets macOS 13 and later. On an Intel Mac, or on any Mac running Python 3.14, the library declares neither harvesters nor genicam, and every entry point that reaches GenICam hardware aborts with an error naming the limitation. Use the 'opencv' camera interface there, or drive GenICam cameras from a Linux or Windows host. The macOS wheel ships only up to genicam 1.5.0, so a Mac resolves one release behind the other platforms. Every other library feature, including video encoding and log processing, works on every Mac under every supported Python version.

Quickstart

Consult the API documentation for all available VideoSystem configuration parameters.

Most library functionality is accessible through the VideoSystem class:

import tempfile
from pathlib import Path

import numpy as np
import polars as pl
from ataraxis_time import PrecisionTimer
from ataraxis_data_structures import DataLogger, assemble_log_archives
from ataraxis_base_utilities import console, LogLevel

from ataraxis_video_system import VideoSystem, VideoEncoders, CameraInterfaces, run_log_processing_pipeline

# Since the VideoSystem and DataLogger classes use multiprocessing under-the-hood, the runtime must be protected by the
# __main__ guard.
if __name__ == "__main__":
    # Enables the console module to communicate the example's runtime progress via the terminal.
    console.enable()

    # Specifies the directory where to save the acquired video frames and timestamps.
    tempdir = tempfile.TemporaryDirectory()  # Creates a temporary directory for illustration purposes
    output_directory = Path(tempdir.name)

    # The DataLogger is used to save frame acquisition timestamps to disk as uncompressed .npy files.
    logger = DataLogger(output_directory=output_directory, instance_name="webcam")

    # The DataLogger has to be started before it can save any log entries.
    logger.start()

    # The VideoSystem requires a unique system_id, a DataLogger instance, a human-readable name, and an output
    # directory. The system_id identifies this camera's log entries. The name is written to the camera manifest
    # file alongside the system_id to support downstream log processing discovery.
    vs = VideoSystem(
        system_id=np.uint8(101),
        data_logger=logger,
        name="webcam",
        output_directory=output_directory,
        camera_interface=CameraInterfaces.OPENCV,  # OpenCV interface for webcameras
        display_frame_rate=15,  # Displays the acquired data at a rate of 15 frames per second
        color=False,  # Acquires images in MONOCHROME mode
        video_encoder=VideoEncoders.H264,  # Uses H264 CPU video encoder.
        quantization_parameter=25,  # Increments the default qp parameter to reflect using the H264 encoder.
    )

    # Calling this method arms the video system and starts frame acquisition. However, the frames are not initially
    # saved to disk.
    vs.start()
    console.echo(f"VideoSystem: Started", level=LogLevel.SUCCESS)

    console.echo(f"Acquiring frames without saving...")
    timer = PrecisionTimer("s")
    timer.delay(delay=5, block=False)  # During this delay, camera frames are displayed to the user but are not saved

    # Begins saving frames to disk as an MP4 video file.
    console.echo(f"Saving the acquired frames to disk...")
    vs.start_frame_saving()
    timer.delay(delay=5, block=False)  # Records frames for 5 seconds, generating ~150 frames
    vs.stop_frame_saving()

    # Frame acquisition can be started and stopped as needed, although all frames are written to the same output
    # video file.

    # Stops the VideoSystem runtime and releases all resources.
    vs.stop()
    console.echo(f"VideoSystem: Stopped", level=LogLevel.SUCCESS)

    # Stops the DataLogger and assembles all logged data into a single .npz archive file. This step is required to be
    # able to extract the timestamps for further analysis.
    logger.stop()
    console.echo(f"Assembling the frame timestamp log archive...")
    assemble_log_archives(remove_sources=True, log_directory=logger.output_directory, verbose=True)

    # Runs the log processing pipeline to extract frame acquisition timestamps from the assembled log archive. The
    # pipeline writes results as Feather files in a camera_timestamps/ subdirectory under the specified output
    # directory. Source IDs are validated against the camera_manifest.yaml auto-generated by the VideoSystem.
    console.echo(f"Extracting frame acquisition timestamps from the assembled log archive...")
    processed_directory = output_directory / "processed"
    run_log_processing_pipeline(
        log_directory=logger.output_directory,
        output_directory=processed_directory,
        source_ids=["101"],
    )

    # Reads the processed timestamps from the output Feather file and computes the camera frame rate. The pipeline
    # writes output files into the camera_timestamps/ subdirectory.
    timestamps_directory = processed_directory / "camera_timestamps"
    dataframe = pl.read_ipc(source=timestamps_directory / "camera_101_timestamps.feather")
    timestamp_array = dataframe["frame_time_us"].to_numpy()
    time_diffs = np.diff(timestamp_array)
    fps = 1 / (np.mean(time_diffs) / 1e6)
    console.echo(
        message=(
            f"According to the extracted timestamps, the interfaced camera had an acquisition frame rate of "
            f"approximately {fps:.2f} frames / second."
        ),
        level=LogLevel.SUCCESS,
    )

    # Cleans up the temporary directory before shutting the runtime down.
    tempdir.cleanup()

Data Logging

This library relies on the DataLogger class to save frame acquisition timestamps to disk during runtime. Each saved frame’s acquisition timestamp is serialized and saved as an uncompressed .npy file.

The DataLogger instance used by the VideoSystem instances may be shared by multiple other Ataraxis assets that generate log entries, such as MicroControllerInterface instances. To support using the same logger instance for multiple concurrently active sources, each source has to use a unique identifier value (system id) when sending data to the logger instance.

Camera Manifest

Each VideoSystem instance automatically writes a camera_manifest.yaml file into the DataLogger output directory during initialization. The manifest associates the system_id with the human-readable name provided to the VideoSystem constructor. When multiple VideoSystem instances share the same DataLogger, each instance appends its entry to the same manifest file. The manifest is required by the log processing pipeline to identify which .npz archives were produced by ataraxis-video-system and to resolve source IDs for processing.

Log Format

Each frame’s acquisition timestamp is logged as a one-dimensional numpy uint8 array, saved as an .npy file. Inside the array, the data is organized in the following order:

1. The uint8 id of the data source (video system instance). The ID occupies the first byte of each log entry.
2. The uint64 timestamp that specifies the number of microseconds elapsed since the acquisition of the onset timestamp (see below). The timestamp occupies 8 bytes following the ID byte. This value communicates when each saved camera frame has been acquired.

Note, timestamps are generated at frame acquisition but are only submitted to the logger when the corresponding frame is saved to disk. Therefore, the timestamps always match the order in which the saved frames appear in the video file.

Onset Timestamp

Each VideoSystem generates an onset timestamp as part of its start() method runtime. This log entry uses a modified data order and stores the current UTC time, accurate to microseconds, as the total number of microseconds elapsed since the UTC epoch onset. All further log entries for the same source use the timestamp section of their payloads to communicate the number of microseconds elapsed since the onset timestamp acquisition.

The onset log entry uses the following data organization order:

1. The uint8 id of the data source (video system instance).
2. The uint64 value 0 that occupies 8 bytes following the source id. A 'timestamp' value of 0 universally indicates that the log entry stores the onset timestamp.
3. The uint64 value that stores the number of microseconds elapsed since the UTC epoch onset. This value specifies the current time when the onset timestamp was generated.

Working with VideoSystem Logs

See the quickstart example above for a demonstration on how to assemble and parse the frame acquisition log archives generated by the VideoSystem instance at runtime.

Note, the parsed frame acquisition timestamps are stored as a contiguous numpy uint64 array that matches the order in which the frames were saved to disk as an .mp4 file. The Feather output files in the camera_timestamps/ subdirectory carry the same values as a Polars DataFrame column. Each timestamp is given as the number of microseconds elapsed since the UTC epoch onset.

Log Processing

This library includes a log data processing pipeline for extracting frame acquisition timestamps from the .npz log archives generated by VideoSystem instances at runtime. The pipeline reads archives produced by the DataLogger, extracts timestamps for each saved frame, and writes the results as Polars DataFrames in Apache Feather (IPC) format.

The pipeline uses the camera manifest to identify which .npz archives were produced by ataraxis-video-system. A camera_manifest.yaml file must be present in the log directory for processing to succeed. When source IDs are not explicitly provided, the pipeline resolves all registered source IDs from the manifest automatically. When source IDs are provided explicitly, they are validated against the manifest to prevent accidental processing of non-video log archives.

One recording writes one VideoSystem to one DataLogger, so exactly one camera manifest is supported per invocation. A log directory tree holding several manifests, or archives written by several DataLogger instances, spans several recordings and is rejected with a diagnostic naming the topology it detected.

Processing is split across two entry points that share their job resolution but not their sizing or execution. The axvs process CLI command and the run_log_processing_pipeline() function target a single recording and run its archives one at a time in the calling process, which suits manual runs and small recordings. The MCP server log processing tools orchestrate batches spanning many recordings, admitting jobs against a core budget and a memory budget and running them in one shared process pool. Both write a YAML-based processing tracker that manages job lifecycle (scheduled, running, succeeded, or failed), and both write every output file into a camera_timestamps/ subdirectory under the specified output directory.

Each job targets exactly one log archive. On the batch path, every job is sized from its own archive. An archive holding fewer than 35000 data messages takes one worker, and every larger archive takes the declared stage width of eight cores. The stage emits these two shapes and nothing between them, and the batch's core budget bounds the width only at admission. On the sequential path a positive --workers value reaches every job verbatim, while the default resolves each job's width from its own archive exactly as the batch path does. The job resolution and the memory model are exported as callable functions, so an external scheduler derives the same figures this library dispatches with.

CLI Commands

This library provides the axvs CLI that exposes the following commands:

Use axvs --help or axvs COMMAND --help for detailed usage information. A command whose execution fails reports the reason through the console at the error level and exits zero, so a shell driving the CLI reads the reported message rather than an interpreter traceback. A malformed invocation exits 2 instead, whether Click rejects the option itself or the command body raises a usage error.

Note, a script chaining commands with set -e or && therefore continues past a failed command. The console writes to the standard error stream, so such a script captures that stream to decide whether the command succeeded.

MCP Server

This library provides an MCP server that exposes camera discovery, configuration, video recording, camera manifest management, and log data processing functionality for AI agent integration.

Starting the Server

Start the MCP server using the CLI:

axvs mcp

Available Tools

Client Registration

MCP server registration and Claude Code skill assets for this library are distributed through the ataraxis marketplace as part of the video plugin. Install the plugin from the marketplace to automatically register the MCP server with compatible clients and make all associated skills available.

Using GenICam Compatible Cameras

This library supports all cameras compatible with the GenICam standard, which includes most GigE+ scientific and machine vision cameras.

Note, this interface is available on Windows, Linux, and Apple Silicon Macs running Python 3.12 or 3.13. See the macOS section for the limitation that leaves it out on the remaining Macs.

Note, before using the library with a GenICam camera, it must be provided with the path to the .cti GenTL Producer Interface file. Use the axvs cti set CLI command to configure the library to use the .cti file provided by the camera vendor (preferred) or a general .cti file, such as the one listed under Dependencies. This command only needs to be called once, as the library remembers and reuses the provided .cti file for all future runtimes. The AXVS_CTI_PATH environment variable takes precedence over the persisted path, which redirects a single runtime to an alternative Producer without changing the path stored for future runtimes.

GenICam Configuration

GenICam-compatible cameras expose many configurable parameters (exposure time, gain, resolution, trigger mode, etc.) through the GenICam standard. This library provides tools to inspect, modify, save, and load these parameters via the CLI (axvs configure) or the MCP server. Calling code reads the live configuration of a camera through the read_camera_configuration() function, which opens and closes its own connection.

The axvs configure read command lists all writable nodes on a connected camera, or displays detailed metadata for a specific node (type, current value, access mode, valid range, step increment, enumeration entries, unit, and description). The axvs configure write command sets a single node to a new value, with automatic type conversion for integer, float, and boolean nodes. String and enumeration values are written as supplied. The command reads the node back over the same connection and reports the value the camera holds, because a node that advertises ReadWrite access can still round the write to its step increment or reject it.

To support reproducible configurations, the axvs configure dump command saves all current camera parameters to a human-readable YAML file, tagged with the camera model and serial number. The axvs configure load command restores a saved configuration onto a camera, with optional --strict mode that aborts on camera identity mismatches instead of issuing a warning. Saved configuration files can also be edited manually before loading.

The axvs configure group parses the options its subcommands share, so they must be given before the subcommand name. Every subcommand requires -c/--camera-index to name the camera it operates on, and axvs configure SUBCOMMAND --help stays reachable without it. The group also excludes a default set of vendor nodes (CustomerIDKey, CustomerValueKey, and TestPattern) from the read, dump, and load operations, because some cameras report them as writable and then reject the write at the hardware level. Pass -b/--blacklisted-node to replace that set, or --no-blacklist to disable the filtering entirely. The two are mutually exclusive. An explicitly named node passed to axvs configure write is always written.

Note, configurations are independent of video capture sessions. The camera is configured first, and the VideoSystem respects the active configuration for every node it does not set itself. Supplying frame_width, frame_height, or frame_rate to the VideoSystem constructor overwrites the corresponding nodes for that session. Every configuration command opens and closes its own connection, so this workflow depends on the camera retaining node state across a device close. A camera that discards that state serves its previous values to the VideoSystem, and it needs the configuration saved to a camera UserSet instead. Reading the nodes back in a separate command shows whether the camera retained the values.

API Documentation

See the API documentation for the detailed description of the methods and classes exposed by components of this library.

Note, the API documentation also includes the details about the axvs CLI interface exposed by this library.

Developers

This section provides installation, dependency, and build-system instructions for the developers that want to modify the source code of this library.

Installing the Project

Note, this installation method requires mamba version 2.3.2 or above. Currently, all automation pipelines require that mamba is installed through the miniforge3 installer.

1. Download this repository to the local machine using the preferred method, such as git-cloning.
2. If the downloaded distribution is stored as a compressed archive, unpack it using the appropriate decompression tool.
3. cd to the root directory of the prepared project distribution.
4. Install the core development dependencies into the base mamba environment via the mamba install tox uv tox-uv command.
5. Use the tox -e create command to create the project-specific development environment followed by tox -e install command to install the project into that environment as a library.

Additional Dependencies

In addition to installing the project and all user dependencies, install the following dependencies:

1. Python distributions, one for each version supported by the developed project. Currently, this library supports the three latest stable versions. It is recommended to use a tool like pyenv to install and manage the required versions.

Development Automation

This project uses tox for development automation. The following tox environments are available:

Run any environment using tox -e ENVIRONMENT. For example, tox -e lint.

Note, all pull requests for this project have to successfully complete the tox task before being merged. To expedite the task's runtime, use the tox --parallel command to run some tasks in parallel.

AI-Assisted Development

Claude Code skills and other AI development assets for this project are distributed through the ataraxis marketplace across two plugins:

• video plugin: Provides MCP server registration, video-specific skills for camera setup, pipeline orchestration, log processing, and post-recording verification. Install this plugin to register the axvs mcp server with compatible MCP clients and make all video workflow skills available.
• automation plugin: Provides shared development skills that enforce Ataraxis framework coding conventions (Python style, README style, commit messages, pyproject.toml, tox configuration) and general-purpose codebase exploration tools.

Install both plugins from the marketplace to make all associated skills and development tools available to compatible AI coding agents.

Automation Troubleshooting

Many packages used in tox automation pipelines (uv, mypy, ruff) and tox itself may experience runtime failures. In most cases, this is related to their caching behavior. If an unintelligible error is encountered with any of the automation components, deleting the corresponding cache directories (.tox, .ruff_cache, .mypy_cache, etc.) manually or via a CLI command typically resolves the issue.

Versioning

This project uses semantic versioning. See the tags on this repository for the available project releases.

Authors

• Ivan Kondratyev (Inkaros)
• Jacob Groner (Jgroner11)
• Natalie Yeung

License

This project is licensed under the Apache 2.0 License: see the LICENSE file for details.

Acknowledgments

• All Sun lab members for providing the inspiration and comments during the development of this library.
• The creators of all other dependencies and projects listed in the pyproject.toml file.