hyperdrone

Drone simulation stack for RLtools: raytracing renderer, L2F multirotor dynamics, and environment setup under one Python package.

Native components are JIT-compiled per set of compile-time constants (resolution, camera count, drone count, ...) into a persistent cache; later uses load the cached library directly. render and dynamics are peers that never import each other — the coupling surface is a plain DLPack tensor of packed camera bases, device-resident on OptiX + CUDA.

Notebooks

Notebook	Open in Colab
HyperDrone
Benchmark
Custom
Orbiter

Install

From an rl-tools checkout (development):

.venv/bin/pip install -e interface/python/hyperdrone

From an sdist (self-contained — bundles the rl-tools headers and raytracing sources, no checkout needed):

pip install hyperdrone-<version>.tar.gz

Install the dependencies used by the packaged examples with:

pip install "hyperdrone[examples]"

Requirements: CMake >= 3.24, a C++17 compiler, and git for dependency downloads. GLB loading uses the Assimp 6.0.4 release archive with a pinned SHA-256, shared by render and env builds. Cached archives are reused offline and verified by CMake before extraction. CUDA + OptiX driver is required for the OptiX render backend and cuda dynamics, Vulkan dev + glslang for the VULKAN backend. The WEBGPU backend fetches a hash-pinned wgpu-native prebuilt at first build (no extra system packages). Nothing beyond the build toolchain for GENERIC + cpu.

Environment

Each component and variant gets its own build tree (render-optix-*, dynamics-cuda-*, ...); switching never invalidates another's cache. All configure/build steps run under a per-tree file lock, so many worker processes can share one cache safely. CMake FetchContent sources and native build state live in <cache>/.dependencies; the native build does not write generated files into an editable checkout or installed package.

Whenever a renderer is successfully allocated, RLtools reports the selected backend on stderr, for example #rl_tools::rendering::raytracing: backend=metal. The Renderer.backend property provides the same lowercase name programmatically.

Rendering

import math, numpy as np
from hyperdrone import render
from hyperdrone.examples.data import procthor_scene_path

scene = render.load_scene(procthor_scene_path(), fidelity="high")

renderer = render.Renderer(width=320, height=240, num_cameras=4, output="rgbd", fidelity="high")
renderer.init(scene)

camera = renderer.camera(position=(0, 0, 1.5), look_at=(1, 0, 1.5), fov=math.radians(80))
renderer.set_cameras(np.repeat(camera[None], 4, axis=0))
renderer.render()

rgb   = renderer.frame()    # (4, 240, 320, 4) uint8
depth = renderer.depth()    # (4, 240, 320) float32

The renderer operates in the L2F FLU frame: +X forward, +Y left, +Z up; GLB assets are swizzled at load time. Procedural scenes (render.Scene() + render.Object / render.Mesh / render.SceneLight), segmentation output, overlays (per-camera dynamic content via AssetPool + spawn/attach), motion blur, and anti-aliasing follow the same API as before under hyperdrone.render.*.

Converted ProcTHOR scenes carry ambient (0.6, 0.6, 0.6) and a black background in scenes[].extras.rl_tools.environment (mode: "solid", ambient and background are nonnegative linear RGB triples). Scene loading imports these settings; importing objects or assemblies leaves the parent environment alone. Ambient affects high and veryhigh, approximating indirect illumination without light bounces. Override imported settings with scene.set_environment(...) before renderer.init(scene).

num_probes sizes the collision-probe pass (generate_probe_directions(), render("collision"), collisions(); one probe per camera by default). num_probes=0 compiles that pass out entirely, the counterpart of the probe-ray macro the C++ benchmark targets set: no probe buffers are allocated, the default render() target runs only the image channels, and the probe methods raise.

Zero-copy I/O (DLPack)

renderer.frame()                  # snapshot copy (safe to keep)
renderer.frame(copy=False)        # numpy view of the staging buffer (refreshed in place)
rgba = torch.from_dlpack(renderer.frame_dlpack())
# live uint8 (num_cameras, height, width, 4), zero-copy GPU tensor on OptiX
packed = torch.from_dlpack(renderer.frame_raw_dlpack())
# same memory as packed uint32 (num_cameras, height, width)

Inputs accept any DLPack producer. CUDA-resident camera input (OptiX): set_cameras dispatches on __dlpack_device__, so a CUDA tensor — a torch GPU tensor, a set pre-uploaded with hyperdrone.cuda.upload, or Sim.camera_bases() — is handed over device-to-device on the render stream, fully async; stream= takes the producer's cudaStream_t handle for event-ordered hand-off.

Dynamics

from hyperdrone import dynamics

sim = dynamics.Sim(num_drones=4096, model="x500_sim", device="cuda")
sim.reset(seed=0)                     # deterministic; host-sampled, identical on cpu/cuda
sim.step(actions)                     # (N, 4) float32 in [-1, 1], host or CUDA tensor
sim.state["position"]                 # zero-copy DLPack views, CUDA-resident on cuda
sim.state.numpy("position")           # host copy
sim.observe()                         # (N, 18): position, rotation matrix, velocities
sim.parameters["mass"] = masses       # per-drone runtime parameters (until the next reset)

cameras = sim.camera_bases(fov=math.radians(100), aspect=renderer.aspect)
renderer.set_cameras(cameras, stream=sim.stream)   # device-resident hand-off

num_drones and domain randomization are compile-time (the JIT key); the model preset (crazyflie, x500_real, x500_sim, mrs, ...), integration dt, and all physical parameters are runtime. Reward and termination live in the environment (hyperdrone.env), not the simulator. One process can use one dynamics variant (cpu or cuda).

RL environment

import numpy as np
from hyperdrone.env import EnvConfig, MultiEnvironment

config = EnvConfig(num_environments=1, instances=1024, cam_width=32, cam_height=32, task="target_frame")
env = MultiEnvironment("scenes/", config=config, seed=0)

mask = np.ones(env.total_instances, dtype=np.uint8)
env.reset(mask)                    # resample parameters + states where mask is set
env.render(mask)                   # render the FPV cameras (mask marks fresh episodes)
observations = env.observe()       # (total, observation_dim) float32
env.step(actions)                  # (total, action_dim) float32 in [-1, 1]
rewards, terminated = env.rewards(), env.terminated()
critic_input = env.observe_privileged()
env.rotate_scene()                 # deterministic scene rotation; reset all instances after
env.observation_layout             # named blocks: which channels/values mean what

reset(mask) samples parameters and states immediately. observe() renders as needed, so explicit render(mask) calls are optional. The training loop owns episode counters and chooses when to reset; the Gymnasium adapter below supplies same-step autoreset and time limits.

All environment semantics — reset, reward, termination, scene scheduling, observation composition — live on the C++ side (rl_tools::rl::environments::hyperdrone::MultiEnvironment<World>); the binding marshals tensors and nothing else, and a seeded rollout is pinned bit-exact against the C++ verbs by a golden test. The scenes argument is a directory of .glb scenes (sorted corpus) or a list of references — .glb paths, conta:HASH strings, or conta store manifest entries {"description": ..., "hash": ...} — used as the corpus in list order and partitioned across environments; conta references (scenes, drone_asset=, gate_asset=) resolve through hyperdrone.conta into the shared content-addressed cache, so a config pins the exact corpus on any machine. Configuration follows the C++ extension ladder: preset= names the platform ("crazyflie", "x500_fpv" — SELF_VISIBLE, pass a body/prop_* GLB via drone_asset= — and "x500_fpv_imu", the same platform stepping at IMU rate), task= names the wrapper ("target_frame", "moving_gate", "visual_inertial_localization"), n_agents= enables multi-agent, and EnvConfig(spec_header=...) pins an arbitrary C++ specification (a header defining hyperdrone_env_user::WORLD, hashed into the JIT key). A spec header can go beyond constants to a full user-authored task wrapper — verb overloads registering and moving entities, compiled reward, termination: hyperdrone/examples/Orbiter.ipynb walks through one, and its contract is pinned by tests/env/user_task_header.h. The render backend for the environment follows HYPERDRONE_ENV_BACKEND (default: the render backend selection).

Manual composition of Sim and Renderer (no MDP — rendering research, data generation) remains first-class: see python -m hyperdrone.examples.drone_flythrough for the wiring. FreeSpaceSampler lives in hyperdrone.render — it rides the renderer's collision probes and is deterministic given a seed.

Visual-inertial localization

task="visual_inertial_localization" on preset="x500_fpv_imu" is the localization benchmark: fixed-length synchronized episodes stepping at IMU rate (env.dt), a camera frame every env.frame_stride steps, and the IMU sample of each step from env.observe_imu() (named by env.observation_layout_imu: accelerometer, gyroscope, frame age, new-frame flag). The task is autonomous: the RAPTOR autopilot built into it flies a waypoint route (its checkpoint is pinned by content hash and fetched through conta on first use), so action_dim is 0 and step() takes no actions. The privileged observation carries the current waypoint next to the dynamics state; the drone's own body is not rendered, matching the C++ benchmark harness. Resets must be all-or-none per environment.

config = EnvConfig(instances=2, cam_width=160, cam_height=120, preset="x500_fpv_imu", task="visual_inertial_localization")
env = MultiEnvironment(scenes, config=config, seed=0)
env.reset(); env.render(np.ones(env.total_instances, dtype=np.uint8))
for step in range(env.episode_step_limit):
    if step > 0: env.render()
    if step % env.frame_stride == 0:
        frames = env.frames()          # the estimator's camera frame
    env.step()
    imu = env.observe_imu()            # the estimator's IMU sample for this step
    truth = env.observe_privileged()   # position / rotation matrix / velocities / waypoint

python -m hyperdrone.examples.visual_inertial_localization runs an episode and scores IMU dead reckoning against the ground truth, the same protocol as the C++ demo.

Gymnasium

from hyperdrone.gym import VectorEnv   # pip install "hyperdrone[gym]"
env = VectorEnv("scenes/", config=EnvConfig(instances=1024), seed=0)
observations, infos = env.reset()
observations, rewards, terminations, truncations, infos = env.step(actions)

The adapter resets instances that terminate or reach episode_step_limit within the same step. Returned observations start the new episode for those instances; no final observation is exposed. truncations flags time limits that are not terminations. The core packages never import gymnasium.

End-to-end example: python -m hyperdrone.examples.drone_flythrough; renderer benchmark: python -m hyperdrone.examples.benchmark (flag-compatible with the C++ benchmark counterpart).

Publishing

From the repository root, build and upload a release with:

interface/python/hyperdrone/scripts/publish.sh

Set the release version in pyproject.toml and hyperdrone/__init__.py first. The script uses the repository's .venv (creating it if needed), installs the build and upload tools, builds a bundled source archive and a py3-none-any wheel from that archive, checks both with Twine, and uploads both to PyPI. Authenticate through your existing Twine configuration (~/.pypirc, keyring, or TWINE_PASSWORD), or enter a PyPI API token when prompted.

Use interface/python/hyperdrone/scripts/publish.sh --build-only to build and check without uploading. Each run keeps its two artifacts in a fresh interface/python/hyperdrone/dist/release.* directory, so older builds are never included in an upload. Native components remain JIT-compiled on first use.

Tests

HYPERDRONE_RENDER_BACKEND=GENERIC .venv/bin/python -m pytest interface/python/hyperdrone/tests -v

tests/jit exercises the build infrastructure against a toy component in seconds; tests/test_architecture.py enforces the import DAG (render and dynamics never see each other); scripts/test_sdist.sh is the self-containment rot guard (build sdist → clean venv → vendored-tree render).

For labelled ProcTHOR rendering, preserve GLB root instances and select the output meaning:

scene = render.load_scene(procthor_scene_path(), rgb=False, preserve_instances=True)
renderer = render.Renderer(width=512, height=512, output="segmentation",
                           semantic_segmentation=True)  # False selects instance IDs
renderer.init(scene)
# Set cameras and render as usual.
labels = renderer.segmentation()  # uint32; renderer.segmentation_mode identifies the mode
classes = scene.segmentation_classes  # {class_id: name}, taxonomy: scene.segmentation_taxonomy

Semantic IDs come from AI2THOR-Hab's taxonomy: unknown is 0 (including Ceiling, which has no dataset category); background is 0xFFFFFFFF in both modes. Instance IDs identify scene instances: use scene.instance_object(id) then scene.object_name(object_index) for static hits. Overlay instance IDs use the renderer's global slot layout. Semantic IDs must never be used as object indices. Loading remains welded by default for RGB; semantic rendering rejects a welded object containing multiple classes. Assemblies and asset-pool copies retain labels and the taxonomy. Composing conflicting taxonomies requires an explicit remapping before semantic rendering.

Assign labels before Renderer.init(): object.segmentation_class = id, scene.set_object_segmentation_class(index, id), assembly.set_segmentation_class(index, id), or pool.set_object_segmentation_class(asset, object_index, id) retains the current taxonomy and explicitly makes that object's class valid, including welded objects. Assignment rejects IDs absent from that taxonomy. object.assign_segmentation_class(id, taxonomy="custom", classes={0: "Unknown", 7: "Furniture"}) replaces the taxonomy; omitting taxonomy/classes selects unnamed numeric IDs.

Objects, scenes, assemblies and pools expose remap_segmentation_classes(mapping, taxonomy="custom", classes={...}); pools also expose remap_asset_segmentation_classes(asset, mapping, ...). Omitting taxonomy/classes selects unnamed numeric IDs. Mappings must cover every used class, and validation failures leave labels unchanged. Remapping preserves valid class boundaries and rejects mixed-class welded objects; explicitly assign one class first if collapsing their labels is intentional. Remap conflicting sources separately before composition. The C++ equivalents are device-first assign_segmentation_class and remap_segmentation_classes in rendering/segmentation_cpu.h, with a shared immutable target taxonomy (or nullptr for unnamed IDs).