GStreamer Python ML

Run any model with ease inside any GStreamer pipeline.

Pure Python ML elements for upstream GStreamer 1.24 or later: one element per task, or bring your own model and run it with the base classes and matching engine.

Write the pipeline in gst-launch format and run it with pyml-launch, with media backend set to either GStreamer or glass2glass.

Features

• Video
  • object detection and zero-shot detection
  • tracking
  • pose estimation
  • depth estimation
  • zero-shot classification with CLIP or SigLIP
  • segmentation with SAM2
  • OCR
  • face detection and recognition
  • optical flow
  • super-resolution
  • action recognition
  • anomaly detection
  • captioning and vision-language models
  • embeddings and video search
• Audio
  • voice activity detection
  • transcription and translation
  • speech separation
  • text to speech
  • audio classification with CLAP
• Text
  • local and remote LLMs
  • incident digests
  • text to image
• Around them
  • alerts over webhooks and MQTT
  • clip recording
  • metadata sink and replay
  • Kafka sink
  • MCP server for agents
• Engines
  • PyTorch by default
  • ONNX Runtime, TensorRT through ONNX Runtime, OpenVINO, LiteRT, TensorFlow
  • Apache TVM, tinygrad, Apple MLX, ExecuTorch, llama.cpp, JAX
  • MiGraphX, IREE, NCNN, Renesas DRP-AI
  • CI runs every engine that installs from PyPI on the CPU and checks its output against PyTorch

Install

Ubuntu 24.04 or later:

sudo apt install -y python3-pip python3-venv \
    gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps \
    gstreamer1.0-plugins-good gstreamer1.0-plugins-bad \
    gir1.2-gst-plugins-bad-1.0 python3-gst-1.0 gstreamer1.0-python3-plugin-loader \
    libcairo2 libcairo2-dev git

Then a venv on the system Python, which is the one GStreamer's plugin loader embeds. uv sync installs PyTorch, with CUDA on Linux, and the yolo extra adds the YOLO models the quick start uses.

curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/collabora/gst-python-ml.git && cd gst-python-ml
uv venv --python /usr/bin/python3 --system-site-packages
uv sync --extra yolo
export GST_PLUGIN_PATH=$PWD/plugins:$GST_PLUGIN_PATH
gst-inspect-1.0 python

The last line lists the pyml_* elements. INSTALL.md covers Fedora, Windows, Docker, the feature extras, the other engines and custom plugins.

Quick start

Paths are relative to the checkout directory. Change device=cuda to device=cpu on a machine without a GPU.

Track people with YOLO:

python pyml-launch.py filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvertscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda track=True ! pyml_overlay ! videoconvert ! autovideosink

Estimate depth with Depth Anything V2:

python pyml-launch.py filesrc location=data/people.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_depth model-name=depth-anything/Depth-Anything-V2-Small-hf device=cuda ! videoconvert ! autovideosink sync=false

Transcribe Korean speech and translate it to English. Transcripts are logged at GStreamer info level:

GST_DEBUG=python:4 python pyml-launch.py filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! fakesink

Record a clip around every person detection:

python pyml-launch.py filesrc location=data/people.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda ! pyml_alert rules='{"class":"person","min_score":0.7}' cooldown=8 draw-alert=false ! pyml_alertrecorder location=alert-%s.webm seconds-before=2 seconds-after=3 ! pyml_overlay ! videoconvert ! autovideosink sync=false

Replay a recorded detection stream through the tracker and overlay, with no model and no GPU:

python pyml-launch.py filesrc location=data/people.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_metareplay location=data/people.jsonl ! pyml_tracker tracker-type=sort ! pyml_overlay ! videoconvert ! autovideosink sync=false

PIPELINES.md describes every pipeline, one or two per element, how to run them under glass2glass, and how to run the MCP server.

Blog post

Unleashing gst-python-ml: Python-powered ML analytics for GStreamer pipelines

Combining GStreamer with machine learning frameworks into video analytics pipelines, from object detection to video captioning.