OpenTau - Train VLA models with state-of-the-art techniques by Tensor

At Tensor, we are pushing the frontier of large foundation models for physical AI. In robot learning, a vision-language-action (VLA) model is a multimodal foundation model that integrates vision, language, and action. Today, VLA represents the leading approach for embodied AI, spanning autonomous driving, robot manipulation, and navigation.

OpenTau is Tensor’s open-source training toolchain for frontier VLA models—designed to make training reproducible, accessible, and scalable. At Tensor, we believe in open research and reproducible progress for the robotics community. By open-sourcing our training toolchain, we aim to expand knowledge sharing and accelerate scientific progress that others can reproduce.

Whether you use the official OpenPi codebase or LeRobot’s reimplementation, you may still be missing key components. OpenTau implements these key capabilities in one place:

• Co-training on an adjustable mixture of heterogeneous datasets
• Discrete actions for fast VLM convergence in $\pi_{0.5}$
• Knowledge insulation between the VLM backbone and the action expert
• Dropout in the VLM to reduce overfitting
• Authentic $\pi_{0.6}$ policy with a Gemma 3 4B backbone, 448×448 vision, and 5-step flow matching
• Hierarchical $\pi_{0.7}$ policy: high-level planner + low-level controller with a SpaceTime SigLIP video encoder on a Gemma 3 backbone
• cosmos3 policy: the $\pi_{0.5}$ flow-matching recipe on a frozen Qwen3-VL-32B (NVIDIA Cosmos3-Super reasoner) backbone with a custom sub-1B action expert — plus a cosmos3_nano sibling on the smaller Qwen3-VL-8B (Cosmos3-Nano reasoner) backbone
• A reinforcement learning pipeline described in $\pi^*_{0.6}$
• And more...

OpenTau ($\tau$) is a tool developed by Tensor to bridge this gap, and we also use it internally to train our proprietary in-house models. Our goal is to help you train VLAs on any dataset while fully leveraging state-of-the-art techniques. We plan to continuously upgrade this repository to keep pace with the state of the art in the robotics community.

Quick Start

If you are familiar with LeRobot, getting started with OpenTau is very easy. Because OpenTau is a fork of the popular LeRobot repository, any LeRobot-compliant policy and dataset can be used directly with OpenTau. Check out our documentation to get started quickly. We provide a quick start guide to help you get started with OpenTau.

For using local notebooks to train and evaluate models, find the notebooks at notebooks/pi05_training.ipynb and notebooks/pi05_evaluation_only.ipynb.

For using the Google Colab notebooks to train and evaluate models, find the colab notebooks here: pi05_training and pi05_evaluation_only respectively.

To spin up a $\pi_{0.6}$ training run, start from configs/examples/pi06_training_config.json. The policy is selected by setting "type": "pi06" and differs from $\pi_{0.5}$ in its Gemma 3 4B backbone, 448×448 image resolution, ~860M-parameter action expert, and 5-step flow-matching default.

$\pi_{0.7}$ splits the model into a high-level planner that proposes subgoals and a low-level controller that executes them. The current implementation pairs a Gemma 3 backbone with a SpaceTime SigLIP video encoder so the controller can attend over temporal context, and the two stages train independently. To train the low-level controller, start from configs/examples/pi07_low_level_libero.json (select via "type": "pi07_low_level"); the high-level planner is registered as "type": "pi07_high_level".

cosmos3 reuses the $\pi_{0.5}$ flow-matching recipe on a frozen Qwen3-VL-32B backbone (NVIDIA's Cosmos3-Super reasoner) paired with a custom sub-1B action expert. Select it with "type": "cosmos3" and start from configs/examples/cosmos3_training_config.json; see the model documentation for the design adjustments. cosmos3_nano is the same architecture on the smaller frozen Qwen3-VL-8B backbone (the Cosmos3-Nano reasoner, which keeps the identical 8-KV-head × 128-head-dim interface the expert reads) — select it with "type": "cosmos3_nano" and start from configs/examples/cosmos3_nano_training_config.json.

Training Diagnostics

OpenTau ships three drop-in scripts under src/opentau/scripts/ to help you figure out where a training run is spending its time. Each reads the same TrainPipelineConfig as opentau-train, so they reproduce your exact model / dataset / batch size — no reconfiguration needed.

A one-command example — see where your training time is going:

accelerate launch \
    --config_file configs/examples/accelerate_ddp_config.yaml \
    src/opentau/scripts/profile_step.py \
    --config_path=<your_training_config.json>

Full tutorial with annotated example output and env-var knobs: docs/tutorials/benchmarking. A worked example investigation that used these tools to find and fix a 2.9× throughput regression is tracked in issue #177.

Checkpoints

We provide fully functioning $\pi_{0.5}$ checkpoints trained with high success rates. We plan to release more models in the near future.

A published training run lives in one repo with each saved step as a git tag and main at the highest step, so a checkpoint path can pin an exact step — --policy.path=TensorAuto/<repo>@6000 — while a bare repo id gives the latest. The suffix works for --policy.path, --config_path and policy.pretrained_path; see the training tutorial.

Acknowledgements

This project builds on the $\pi$ series of papers and many other open-source efforts—especially LeRobot—for re-implementing the $\pi$ models and helping standardize training infrastructure. OpenTau extends these foundations to provide a more accessible, comprehensive toolchain for training vision-language-action agents.