roboml 0.4.0

Machine learning models optimized for robotics experimentation and deployment

RoboML is an aggregator package for quickly deploying open-source ML models for robots. It supports three main use cases:

• Rapid deployment of general-purpose models: Wraps around popular ML libraries like 🤗 Transformers, allowing fast deployment of models through scalable server endpoints.
• Deploy detection models with tracking: Supports deployment of detection models from 🤗 Transformers (RT-DETR, DETR, Grounding DINO, etc.) with optional tracking integration.
• Aggregate robot-specific models from the robotics community: Intended as a platform for community-contributed multimodal models, usable in planning and control, especially with ROS components. See EmbodiedAgents.

Models And Wrappers

Model Class	Description	Default Checkpoint / Resource	Key Init Parameters
TransformersLLM	General-purpose large language model (LLM) from 🤗 Transformers	Qwen/Qwen3-0.6B	name, checkpoint, quantization, init_timeout
TransformersMLLM	Multimodal vision-language model (MLLM) from 🤗 Transformers	Qwen/Qwen2.5-VL-3B-Instruct	name, checkpoint, quantization, init_timeout
RoboBrain2	Embodied planning + multimodal reasoning via RoboBrain 2.0	BAAI/RoboBrain2.0-3B	name, checkpoint, init_timeout
Whisper	Multilingual speech-to-text (ASR) from OpenAI Whisper	small.en (checkpoint list)	name, checkpoint, compute_type, init_timeout
TransformersTTS	Text-to-speech via 🤗 Transformers (Bark, VITS, SpeechT5, SeamlessM4T, etc.)	suno/bark-small	name, checkpoint, voice, vocoder_checkpoint, init_timeout
VisionModel	Detection + tracking via 🤗 Transformers	PekingU/rtdetr_r50vd_coco_o365	name, checkpoint, setup_trackers, tracking_distance_threshold, num_trackers, init_timeout

Installation

RoboML has been tested on Ubuntu 20.04 and later. A GPU with CUDA 12.1+ is recommended. If you encounter problems, please open an issue.

pip install roboml

From Source

git clone https://github.com/automatika-robotics/roboml.git && cd roboml
virtualenv venv && source venv/bin/activate
pip install pip-tools
pip install .

Vision Model Support

VisionModel uses HuggingFace Transformers for object detection and tracking. It works out of the box with any HuggingFace object detection model (RT-DETR, DETR, Grounding DINO, YOLOS, etc.). Object tracking via ByteTrack is included.

If ffmpeg or libGL is missing:

sudo apt-get update && apt-get install ffmpeg libsm6 libxext6

Docker Build (Recommended)

Jetson users are especially encouraged to use Docker.

• Install Docker Desktop
• Install the NVIDIA Container Toolkit

git clone https://github.com/automatika-robotics/roboml.git && cd roboml

# Build container image
docker build --tag=automatika:roboml .
# For Jetson boards:
docker build --tag=automatika:roboml -f Dockerfile.Jetson .

# Run HTTP server
docker run --runtime=nvidia --gpus all --rm -p 8000:8000 automatika:roboml roboml
# Or run RESP server
docker run --runtime=nvidia --gpus all --rm -p 6379:6379 automatika:roboml roboml-resp

• (Optional) Mount your cache dir to persist downloaded models:

  -v ~/.cache:/root/.cache
  

Servers

RoboML uses Ray Serve to host models as scalable apps across various environments.

WebSocket Endpoint

WebSocket endpoints are exposed for streaming use cases (e.g., STT/TTS).

Experimental RESP Server

For ultra-low latency in robotics, RoboML also includes a RESP-based server compatible with any Redis client. RESP (see spec) is a lightweight, binary-safe protocol. Combined with msgpack instead of JSON, it enables very fast I/O, ideal for binary data like images, audio, or video.

This work is inspired by @hansonkd’s Tino project.

Usage

Run the HTTP server:

roboml

Run the RESP server:

roboml-resp

Example usage in ROS clients is documented in ROS Agents.

Running Tests

Install dev dependencies:

pip install ".[dev]"

Run tests from the project root:

python -m pytest

Copyright

Unless otherwise specified, all code is © 2024 Automatika Robotics. RoboML is released under the MIT License. See LICENSE for details.

Contributions

ROS Agents is developed in collaboration between Automatika Robotics and Inria. Community contributions are welcome!