rai_core 2.11.2

Core functionality for RAI framework

RAI

RAI is a flexible AI agent framework to develop and deploy Embodied AI features for your robots.

---

---

Installation

1. Install RAI:

pip install rai-core

2. Initialize the global configuration file:

rai-config-init

3. Optionally install ROS 2 dependencies:

sudo apt install ros-${ROS_DISTRO}-rai-interfaces

For more visit robotecai.github.io/rai for the latest documentation, setup guide and tutorials.

🎯 Overview

Category	Description	Features
🤖 Multi-Agent Systems	Empowering robotics with advanced AI capabilities	• Seamlessly integrate Gen AI capabilities into your robots • Enable sophisticated agent-based architectures
🔄 Robot Intelligence	Enhancing robotic systems with smart features	• Add natural human-robot interaction capabilities • Bring flexible problem-solving to your existing stack • Provide ready-to-use AI features out of the box
🌟 Multi-Modal Interaction	Supporting diverse interaction capabilities	• Handle diverse data types natively • Enable rich sensory integration • Process multiple input/output modalities simultaneously

RAI framework

• rai core: Core functionality for multi-agent system, human-robot interaction and multi-modalities.
• rai whoami: Tool to extract and synthesize robot embodiment information from a structured directory of documentation, images, and URDFs.
• rai_asr: Speech-to-text models and tools.
• rai_tts: Text-to-speech models and tools.
• rai_sim: Package for connecting RAI to simulation environments.
• rai_bench: Benchmarking suite for RAI. Test agents, models, tools, simulators, etc.
• rai_perception: Perception module for open-set detection models and tools.
• rai_nomad: Integration with NoMaD for navigation.
• rai_finetune: Finetune LLMs on your embodied data.

Simulation demos

Try RAI yourself with these demos:

Application	Robot	Description	Docs Link
Mission and obstacle reasoning in orchards	Autonomous tractor	In a beautiful scene of a virtual orchard, RAI goes beyond obstacle detection to analyze best course of action for a given unexpected situation.	link
Manipulation tasks with natural language	Robot Arm (Franka Panda)	Complete flexible manipulation tasks thanks to RAI and Grounded SAM 2	link
Autonomous mobile robot demo	Husarion ROSbot XL	Demonstrate RAI's interaction with an autonomous mobile robot platform for navigation and control	link
Speech-to-speech interaction with autonomous taxi	Simulated car	Demonstrate RAI's speech-to-speech interaction capabilities for specifying destinations to an autonomous taxi in awsim with autoware environment	link

Community

Embodied AI Community Group

RAI is one of the main projects in focus of the Embodied AI Community Group. If you would like to join the next meeting, look for it in the ROS Community Calendar.

Publicity

• A talk about RAI at ROSCon 2024.

RAI Q&A

Please take a look at Q&A.

Developer Resources

See our documentation for a deeper dive into RAI, including instructions on creating a configuration specifically for your robot.

Contributing

You are welcome to contribute to RAI! Please see our Contribution Guide.