neon-vla 0.1.5

The Idea

A child watches a ball roll off a table and reaches out to catch it. She doesn't look at a photograph — she sees the motion. The arc, the acceleration, the moment it leaves the edge. She predicts the future from the flow of time.

Every robot today is blind to this. State-of-the-art Vision-Language-Action models look at the world through frozen snapshots. They see where things are, but not where things are going. It's like trying to catch that ball with your eyes closed between blinks.

Neon's insight is one sentence:

Video foundation models already understand motion — we just connect them to robot bodies.

Models like Qwen2.5-Omni and Cosmos-Reason2 have watched millions of hours of video. They've learned that cups fall when pushed, that doors swing on hinges, that hands reach before they grasp. This temporal understanding — physics, dynamics, cause and effect — is exactly what a robot needs. It's sitting there, pre-trained, waiting.

So we do something radical in its simplicity. We take a 7-billion-parameter video model, freeze it entirely, and train a tiny action decoder on top — just 6 million parameters, 0.08% of the total — that translates the video model's rich temporal understanding into 29 joint commands for a humanoid body, 16 timesteps into the future.

The video model sees. The decoder acts.

pip install neon-vla

How It Works

graph LR
    CAM["📹 Camera"] --> VB["Video Backbone<br/><b>7B frozen</b><br/>Qwen2.5-Omni / Cosmos"]
    MIC["🎤 Voice"] --> VB
    PROP["🦾 Joints"] --> PE["Proprio Encoder"]
    LIDAR["📡 LiDAR"] --> LE["PointCloud Encoder"]
    EEF["🤲 EEF State"] --> EE["EEF Encoder"]
    VB --> FUS["Feature Fusion"]
    PE --> FUS
    LE --> FUS
    EE --> FUS
    FUS --> AH["Action Heads<br/><b>~6M trainable</b><br/>Parameter Golf v2"]
    AH --> ACT["🤖 29 DoF × 16 steps"]
    AH --> SPEECH["🔊 Speech Out"]
    
    style VB fill:#0097a7,color:#fff,stroke:#0097a7
    style AH fill:#e65100,color:#fff,stroke:#e65100
    style FUS fill:#333,color:#fff

graph TD
    subgraph "Inputs (6 modalities)"
        CAM["📹 Camera Frames"]
        VID["🎬 Video Frames"]
        MIC["🎤 Audio (16kHz)"]
        TXT["📝 Language"]
        PROP["🦾 Joint States (29 DoF)"]
        LID["📡 LiDAR Point Cloud (N×4)"]
        EEF["🤲 EEF State (14 DoF)"]
    end

    subgraph "Neon VLA"
        VB["Video Backbone<br/>Qwen2.5-Omni / Cosmos-Reason2<br/><i>frozen, 3-7B</i>"]
        AE["Whisper Audio Encoder<br/><i>frozen, 39M</i>"]
        PE["Proprio Encoder<br/><i>trainable MLP</i>"]
        LE["PointCloud Encoder<br/><i>trainable PointNet-style</i>"]
        EE["EEF Encoder<br/><i>trainable MLP</i>"]
        FUS["Feature Fusion<br/>Linear + ReLU²"]
        AH["Action Heads<br/>Parameter Golf v2<br/><i>trainable, ~6M</i>"]
        SH["Speech Head<br/>PersonaPlex TTS"]
    end

    subgraph "Outputs"
        ARM["Arms (14 DoF)"]
        LOCO["Locomotion (vx, vy, ω)"]
        HEAD["Head (2 DoF)"]
        VOICE["🔊 Voice"]
    end

    CAM --> VB
    VID --> VB
    TXT --> VB
    MIC --> AE
    PROP --> PE
    LID --> LE
    EEF --> EE
    VB --> FUS
    AE --> FUS
    PE --> FUS
    LE --> FUS
    EE --> FUS
    FUS --> AH
    FUS --> SH
    AH --> ARM
    AH --> LOCO
    AH --> HEAD
    SH --> VOICE

    style VB fill:#0097a7,color:#fff
    style AH fill:#e65100,color:#fff
    style FUS fill:#333,color:#fff

Why video models, not image models?

Quick Start

As a VLA model

from neon.model.neon_vla import NeonVLA, NeonConfig

model = NeonVLA(NeonConfig(control_mode="arms_only"))
model.load_backbone()

# Full omni-modal prediction
output = model.predict(
    image=camera_frame,                     # 📹 what the robot sees
    instruction="Pick up the red cup",      # 📝 what you want
    proprioception=joint_states,            # 🦾 where the robot is
    audio=voice_waveform,                   # 🎤 spoken command (16kHz)
    lidar=point_cloud,                      # 📡 spatial awareness (N×4)
    eef_state=ee_positions,                 # 🤲 hand positions (14-DOF)
    speak=True,                             # 🔊 robot narrates its action
)

output.actions      # → (16, 17) — 16 timesteps × 17 joints
output.upper_body   # → (16, 14) — arm positions
output.locomotion   # → (16, 3)  — velocity commands (vx, vy, ω)
output.speech_path  # → "/tmp/neon_speech_xyz.wav"

As a strands-robots policy (plug-and-play)

# Direct usage
from neon import NeonPolicy
policy = NeonPolicy(host="192.168.123.10", port=8300)
actions = policy.get_actions_sync(obs, "pick up the red cup")

# Via strands-robots (auto-discovered on install)
from strands_robots.policies import create_policy
policy = create_policy("neon", host="robot-ip", port=8300)

# Smart resolution from HuggingFace model ID
policy = create_policy("cagataydev/neon-g1-v1-dev")

Run the inference server

# On the robot (Jetson Orin / any CUDA machine)
neon-serve --model cagataydev/neon-g1-v1-dev --port 8300

# The server accepts ALL modalities via HTTP:
curl -X POST http://robot:8300/predict \
  -H "Content-Type: application/json" \
  -d '{"image_base64": "...", "instruction": "pick up the cup", "proprioception": [...]}'

# Check what modalities the model supports:
curl http://robot:8300/health
# → {"modalities": {"camera": true, "audio": true, "lidar": false, ...}}

The Action Decoder — Parameter Golf v2

Our decoder heads come from a competition to build the smallest working language model. Every trick matters when your entire trainable model fits in 25 megabytes:

Data Soup — A Thousand Bodies, Unified

A robot built for a specific body usually needs data from that body. We break this with relative actions — displacements in the gripper's local frame. The same reaching motion produces the same numbers whether performed by a Franka, an SO-100, or our G1 humanoid.

# Same physical motion = same numbers, any robot
rel_xyz = prev_rotm.T @ (curr_xyz - prev_xyz)    # Position delta
rel_rot = rotm2euler(prev_rotm.T @ curr_rotm)     # Rotation delta
action  = [rel_xyz, rel_rot, gripper_state]        # Cross-embodiment!

Seven source types, one training stream:

graph LR
    LR["🤖 LeRobot<br/>Bridge + DROID"] --> MIX
    AG["🦾 Agibot-World<br/>Bimanual 1M+"] --> MIX
    COS["🌌 Cosmos DreamGen<br/>Synthetic"] --> MIX
    S4D["📸 Stereo4D<br/>Kitchen depth"] --> MIX
    VC["🗣️ Voice Commands<br/>50K instructions"] --> MIX
    TEL["🎮 G1 Teleoperation<br/>LiDAR + EEF + Audio"] --> MIX
    DR["💭 GR00T-Dreams<br/>Humanoid demos"] --> MIX

    MIX["Data Soup 🥣"] --> TRAIN["NeonTrainer<br/>All 6 modalities"]

    style MIX fill:#e65100,color:#fff,stroke:#e65100

G1 Humanoid — 29 Degrees of Freedom

graph TD
    G1["Unitree G1<br/>29 DoF"] --> LA["Left Arm · 7"]
    G1 --> RA["Right Arm · 7"]
    G1 --> T["Torso · 1"]
    G1 --> H["Head · 2"]
    G1 --> LL["Left Leg · 6"]
    G1 --> RL["Right Leg · 6"]

    style G1 fill:#e65100,color:#fff,stroke:#e65100
    style LA fill:#00695c,color:#fff
    style RA fill:#00695c,color:#fff
    style LL fill:#0097a7,color:#fff
    style RL fill:#0097a7,color:#fff

Training

Eight presets, from a laptop GPU to a cloud A100:

# Train on HuggingFace Jobs (recommended)
hf jobs uv run --flavor a100-large --secrets HF_TOKEN --timeout 8h -- \
    python -m neon.training.train --backbone Qwen/Qwen2.5-Omni-7B --mode arms_only

# Or locally
from neon.training.config import default_arms_only_config
from neon.training.train import train
train(default_arms_only_config())

By The Numbers

Project Structure

neon/
├── neon/
│   ├── model/
│   │   ├── neon_vla.py          # Complete VLA pipeline + PointCloudEncoder + EEFEncoder
│   │   ├── action_heads.py      # Parameter Golf v2 decoders (ReLU², soft-cap)
│   │   ├── video_backbone.py    # Qwen / Cosmos adapter (3B-8B)
│   │   └── audio.py             # Whisper encoder + PersonaPlex TTS
│   ├── data/
│   │   ├── action_space.py      # G1 29-DoF joint definitions + normalization
│   │   ├── data_soup.py         # 7-source data mixing (NeonEpisode w/ all modalities)
│   │   └── relative_actions.py  # Cosmos-style relative EE actions
│   ├── training/
│   │   ├── config.py            # TrainConfig + 8 presets (incl. g1_omnimodal)
│   │   └── train.py             # NeonTrainer (omni-modal collation + loss)
│   ├── inference/
│   │   ├── server.py            # HTTP inference server (all 6 modalities)
│   │   └── g1_controller.py     # Unitree SDK interface
│   ├── streams/
│   │   ├── channels.py          # Typed data channels (Camera, Joint, LiDAR, Audio, Text, ToolCall)
│   │   ├── recorder.py          # StreamRecorder → LeRobot dataset
│   │   └── session.py           # StreamSession — full robot loop
│   ├── dashboard/
│   │   └── bridge.py            # WebSocket dashboard (camera, joints, LiDAR viz)
│   └── policy.py                # NeonPolicy — strands-robots integration (HTTP/ZMQ)
├── tests/                       # 168 tests (all CPU, no GPU needed)
├── paper/                       # LaTeX white paper + soul manifesto
├── video/                       # Remotion explainer video source
└── docs/                        # MkDocs Material site

strands-robots Integration

Neon ships as a first-class strands-robots policy. On pip install neon-vla, it auto-registers and is immediately discoverable:

from strands_robots.policies import create_policy

# The NeonPolicy bridges VLA inference (5-10 Hz) to robot control (50 Hz)
# via RTC action queue with temporal blending
policy = create_policy("neon", host="192.168.123.10", port=8300)

# Full omni-modal observation
obs = {
    "observation.images.front": camera_frame,   # (H, W, 3) uint8
    "observation.state": joint_positions,        # (17,) float32
    "observation.audio": voice_waveform,         # (16000,) float32
    "observation.lidar": point_cloud,            # (4096, 4) float32
    "observation.eef_state": ee_state,           # (14,) float32
}
actions = policy.get_actions_sync(obs, "pick up the red cup")

Three blend schedules: linear, step, exponential. The policy auto-discovers server capabilities via /health.

Video

The explainer video is built with Remotion — code-as-video, version-controlled, reproducible:

cd video
npm install
npx remotion studio          # Preview in browser
npx remotion render src/index.ts NeonExplainer out/neon-explainer.mp4

Papers

PDFs attached to every GitHub release.

Related Work

• GR00T N1 — Architecture reference for humanoid VLA
• GR00T-WholeBodyControl — RL whole-body policies (learnings adopted in Neon)
• Cosmos-Predict2.5 — Relative actions, world model reasoning
• OmniVLA — Omni-modal VLA reference
• MicroGPT Parameter Golf — Source of action head optimizations
• Strands Agents — Agent framework for robot integration
• strands-robots — Robot SDK (NeonPolicy integrates via entry-point)

Citation

@software{neon2026,
  title   = {Neon: Open-Source Vision-Language-Action Model for Humanoid Whole-Body Control},
  author  = {Cali, Cagatay},
  year    = {2026},
  url     = {https://github.com/cagataycali/neon},
  license = {MIT}
}