roboml 0.5.0

RoboML is an aggregator package for quickly deploying open-source ML models for robots. It supports three main use cases:

• Rapid deployment of general-purpose models: Wraps around popular ML libraries like 🤗 Transformers, allowing fast deployment of models through scalable server endpoints.
• Deploy detection models with tracking: Supports deployment of detection models from 🤗 Transformers (RT-DETR, DETR, Grounding DINO, etc.) with optional tracking integration.
• Aggregate robot-specific models from the robotics community: Intended as a platform for community-contributed multimodal models, usable in planning and control, especially with ROS components. See EmbodiedAgents.

Models And Wrappers

Installation

RoboML has been tested on Ubuntu 20.04 and later. A GPU with CUDA 12.1+ is recommended. If you encounter problems, please open an issue.

pip install roboml

From Source

git clone https://github.com/automatika-robotics/roboml.git && cd roboml
virtualenv venv && source venv/bin/activate
pip install pip-tools
pip install .

Model Checkpoints from ModelScope

By default, RoboML downloads model checkpoints from the HuggingFace Hub. However, users can pull checkpoints from ModelScope (魔搭社区) instead. First install the optional dependency (the Docker images below already include it):

pip install "roboml[modelscope]"

Then either set the source globally when starting the server:

ROBOML_SOURCE=modelscope roboml
# or with Docker: docker run -e ROBOML_SOURCE=modelscope ...

or per model, by passing source: "modelscope" in the body of the model's /initialize call. An explicit source parameter always overrides the environment variable.

ModelScope checkpoints are cached in ~/.cache/modelscope (configurable via MODELSCOPE_CACHE), so the cache mount shown in the Docker section covers them as well.

Most default checkpoints are available on ModelScope under the same IDs they have on the HuggingFace Hub, and Whisper size aliases such as small.en are resolved automatically. If a checkpoint cannot be found on ModelScope, model initialization fails with a descriptive error that suggests a suitable alternative checkpoint whenever one is known.

Some checkpoints are restricted on ModelScope (e.g. BAAI/RoboBrain2.0-*) and cannot be downloaded anonymously. For these, set the MODELSCOPE_API_TOKEN environment variable with an access token from your ModelScope account.

Models that only exist on HuggingFace Hub can still be reached by routing HF downloads through a mirror, e.g. HF_ENDPOINT=https://hf-mirror.com (a community-run proxy; must be set before the server starts). This works independently of ROBOML_SOURCE, since every model node can choose its own source.

Vision Model Support

VisionModel uses HuggingFace Transformers for object detection and tracking. It works out of the box with any HuggingFace object detection model (RT-DETR, DETR, Grounding DINO, YOLOS, etc.). Object tracking via ByteTrack is included.

If ffmpeg or libGL is missing:

sudo apt-get update && apt-get install ffmpeg libsm6 libxext6

Docker Build (Recommended)

Jetson users are especially encouraged to use Docker.

• Install Docker Desktop
• Install the NVIDIA Container Toolkit

git clone https://github.com/automatika-robotics/roboml.git && cd roboml

# Build container image
docker build --tag=automatika:roboml .
# For Jetson boards:
docker build --tag=automatika:roboml -f Dockerfile.Jetson .

# Run HTTP server
docker run --runtime=nvidia --gpus all --rm -p 8000:8000 automatika:roboml roboml
# Or run RESP server
docker run --runtime=nvidia --gpus all --rm -p 6379:6379 automatika:roboml roboml-resp

• (Optional) Mount your cache dir to persist downloaded models:

  -v ~/.cache:/root/.cache
  

Servers

RoboML uses Ray Serve to host models as scalable apps across various environments.

WebSocket Endpoint

WebSocket endpoints are exposed for streaming use cases (e.g., STT/TTS).

Experimental RESP Server

For ultra-low latency in robotics, RoboML also includes a RESP-based server compatible with any Redis client. RESP (see spec) is a lightweight, binary-safe protocol. Combined with msgpack instead of JSON, it enables very fast I/O, ideal for binary data like images, audio, or video.

This work is inspired by @hansonkd’s Tino project.

Usage

Run the HTTP server:

roboml

Run the RESP server:

roboml-resp

Example usage in ROS clients is documented in ROS Agents.

Running Tests

Install dev dependencies:

pip install ".[dev]"

Run tests from the project root:

python -m pytest

Copyright

Unless otherwise specified, all code is © 2024 Automatika Robotics. RoboML is released under the MIT License. See LICENSE for details.

Contributions

ROS Agents is developed in collaboration between Automatika Robotics and Inria. Community contributions are welcome!