Runway Robotics Python SDK

A Python SDK for interacting with Runway's General World Model (GWM-1) and video generation APIs, featuring action-conditioned world simulation and video-to-video transformation for robotics applications.

Features

• World Model — Generate predicted video sequences from robot actions and camera frames
• Image-to-Video — Generate video from a single reference image using text prompts
• Video-to-Video — Transform footage using text prompts and style references

Requirements

• Python 3.10+
• A Runway API key

Installation

pip install runway-robotics-sdk

Then set your API key as an environment variable:

export RUNWAYML_API_SECRET=your_api_key_here

Quick Start

All SDK functionality is accessed through the RunwayRobotics client:

from runway_robotics_sdk import RunwayRobotics

client = RunwayRobotics()

The client reads your API key from RUNWAYML_API_SECRET by default. You can also pass it explicitly:

client = RunwayRobotics(api_key="your_api_key_here")

Robotics World Model

Generate predicted video sequences from initial camera frames and robot action trajectories using client.world_model.create().

Single-View Mode

world = client.world_model.create(
    base_view=Image.open("data/base_view.jpg"),
)

actions = load_your_action_sequence()

for i in range(0, len(actions), world.chunk_size):
    obs = world.step(actions[i : i + world.chunk_size])

world.save("outputs/trajectory.mp4")

Multi-View Mode

world = client.world_model.create(
    base_view=Image.open("data/base_view.jpg"),
    wrist_view=Image.open("data/wrist_view.jpg"),
)

actions = load_your_action_sequence()

for i in range(0, len(actions), world.chunk_size):
    obs = world.step(actions[i : i + world.chunk_size])
    # obs.base_image  — latest base camera frame
    # obs.wrist_image — latest wrist camera frame

world.save("outputs/trajectory.mp4")

Action Format

Each action is a 7-element list:

Actions are processed in chunks of world.chunk_size (17) frames. Shorter sequences are automatically padded to the chunk size.

Supported Robot Arms

The current model works well with the following robot arms:

API Reference

client.world_model.create()

Returns: A WorldModelEnv instance.

WorldModelEnv Methods:

WorldModelEnv Properties:

Image-to-Video

Generate video from a single reference image via client.image_to_video.generate(). Provide a starting frame and an optional text prompt describing the desired motion or scene.

Basic Usage

result = client.image_to_video.generate(
    image="robot_workspace.png",
    prompt="The robot arm reaches forward and grasps the red cup",
    show_progress=True,
)

# Save the raw video to disk
result.save("outputs/generated.mp4")

# Or access decoded frames directly
frames = result.frames  # np.ndarray of shape (T, H, W, C)

Input Formats

The image parameter accepts multiple formats:

API Reference

client.image_to_video.generate()

Returns: A GenerationResult with .bytes (raw video), .frames (lazy-decoded np.ndarray of shape (T, H, W, C)), and .save(path).

Video-to-Video

Transform videos using text prompts via client.video_to_video.generate(). Common use cases include:

• Scene modification — Add or change objects on a workspace surface
• Lighting conditions — Simulate different lighting environments
• Material appearance — Change object textures, colors, or materials

Basic Usage

result = client.video_to_video.generate(
    video="rollout.mp4",
    prompt="Make the lighting in the scene dark as if the power was out in the warehouse.",
    show_progress=True,
)

result.save("outputs/transformed.mp4")
frames = result.frames  # np.ndarray of shape (T, H, W, C)

Input Formats

The video parameter accepts multiple formats:

Style References

Guide the transformation with a reference image:

result = client.video_to_video.generate(
    video="rollout.mp4",
    prompt="Transform the scene to match the reference lighting",
    references=[Image.open("reference.jpg")],
)
result.save("outputs/transformed.mp4")

API Reference

client.video_to_video.generate()

Returns: A GenerationResult with .bytes (raw video), .frames (lazy-decoded np.ndarray of shape (T, H, W, C)), and .save(path).

Error Handling

The SDK raises the following exceptions:

All exceptions can be imported directly from runway_robotics_sdk:

from runway_robotics_sdk import GenerationError, AuthenticationError

try:
    obs = world.step(actions)
except GenerationError as e:
    print(f"Simulation failed: {e}")
except AuthenticationError:
    print("Check your API key")

Examples

uv run python examples/world_model.py       # World model action rollout (multi-view)
uv run python examples/image_to_video.py     # Generate rollout video from an image
uv run python examples/video_to_video.py     # Transform a rollout video with a text prompt

Interactive Web Demo

Launch the Gradio-based demo for interactive world model simulation:

pip install runway-robotics-sdk[demo]
uv run python demo/app.py

Open http://localhost:7880 to upload camera frames, provide action sequences, and generate predicted trajectories.

Contributing

Clone the repository and install development dependencies:

git clone https://github.com/runwayml/robotics-sdk-python.git
cd robotics-sdk-python
uv sync --all-extras
uv run lefthook install

Git hooks (via Lefthook) run automatically:

• Pre-commit — ruff format and ruff check --fix on staged files (auto-fixes and re-stages)
• Pre-push — mypy type checking and pytest with coverage

You can also run checks manually:

uv run ruff format .          # Format code
uv run ruff check --fix .     # Lint (with auto-fix)
uv run mypy runway_robotics_sdk/ demo/ examples/  # Type check
uv run pytest                  # Run tests

All checks are also enforced via GitHub Actions CI.

Data Attribution

Example data in this repository uses the DROID (Distributed Robot Interaction Dataset), licensed under CC BY 4.0. See CITATION.md for full citation details.

License

This project is licensed under the Apache License 2.0. See LICENSE for details.