gpui-python-runtime

Retained scene specifications for the GPUI Python wrapper.

Python declares UI and scene3d objects. Rust validates the scene, tracks stable ids, classifies dirty resources, and adapts supported nodes to gpui-d3rs 3D elements. Raw wgpu devices, queues, buffers, pipelines, and shaders remain private to the renderer.

Scene3D Scope

• Surfaces: row-major z grids with optional x/y axes, log axes, z range, labels, colormaps, wireframe mode, orbit cameras, and interactions.
• Lines: retained orbit camera state and CPU-projected Lines3DElement segments/line strips.
• Meshes and surfaces consume retained ArrayData geometry and scalar-field resources; materials, perspective cameras, and lights remain declarative scene objects while GPU state stays host-owned.

Audited Python Surface

python-surface.toml is the reviewed Python mapping registry. The generated python-rustdoc-inventory.json freezes the all-feature macOS public API of the current priority crates, gpui-d3rs and gpui-px, including inherent methods, fields, callable parameters, and return types. Run the freshness gate with:

just qa-python-rustdoc-inventory

The gate uses pinned nightly-2026-09-02 rustdoc JSON and rejects public Rust surface changes until the snapshot is reviewed and regenerated. Inventory coverage does not itself claim Python parity; stable v2 remains blocked until every inventory entry has a reviewed binding or exclusion classification.

JSON Schema Contract

Python-authored payloads are versioned at the JSON boundary:

• app IR uses schema_version: 1, exposed as PYTHON_APP_IR_SCHEMA_VERSION.
• Scene3D specs use schema_version: 1, exposed as SCENE3D_SPEC_SCHEMA_VERSION.

New Python emitters write the current schema version. Rust treats omitted schema_version fields as v1 so early examples and local scripts keep loading, but validation rejects unsupported future versions before rendering or reusing a cached spec.

Compatibility policy:

1. Additive optional fields may stay on v1 when Rust gives them safe defaults.
2. Renaming fields, removing fields, changing data-shape semantics, or changing renderer meaning requires a schema-version bump.
3. A schema-version bump must include compatibility tests for previous v1 payloads and a migration path before Python emitters start writing it.
4. Consumers should parse JSON, validate PythonAppIr, then parse and validate Scene3D specs through TypedSpecCache or validate_scene3d_spec_schema_version.

Resource Model

RetainedSceneCache fingerprints geometry, material, and camera state separately:

• unchanged scenes do no renderer work,
• camera-only changes update uniforms/state,
• color/material changes update small renderer state,
• data/mesh changes reupload affected geometry.

Gpui3DCache is available behind the gpui feature and keeps Surface3DElement / line camera state keyed by stable ids.

Tutorial

New to the Python API? python/tutorial/ walks through installing the gpui-toolkit wheel and building a native demo app (demo_app.py: text, metrics, a px line chart, and a button with a live on_action → patch loop), then points at the larger examples below.

Python Examples

The examples build JSON-serializable scene specs that the Rust runtime can validate and adapt to GPUI elements:

PYTHONPATH=python python examples/surface_dispersion.py
PYTHONPATH=python python examples/lines_orbit.py
PYTHONPATH=python python examples/mesh_scene.py

• surface_dispersion.py shows a log-frequency surface with orbit controls.
• lines_orbit.py shows line strips, axis references, and a shared orbit camera.
• mesh_scene.py shows the future lower-level scene shape with mesh, path, and light nodes.

The larger app-authored demos mirror the Rust showcase programs:

# Dump the native app IR without requiring a GPUI host.
GPUI_TOOLKIT_DUMP_IR=1 PYTHONPATH=python python examples/spinorama_demo.py
GPUI_TOOLKIT_DUMP_IR=1 PYTHONPATH=python python examples/surface3d_demo.py
GPUI_TOOLKIT_DUMP_IR=1 PYTHONPATH=python python examples/chart_gallery.py

# Without a host-related environment variable, each script prints JSON too.
PYTHONPATH=python python examples/spinorama_demo.py > /tmp/spinorama-python.json

• spinorama_demo.py contains CEA2034, horizontal SPL, vertical SPL, contour, and retained 3D surface sections using deterministic local data.
• surface3d_demo.py contains the sinc, spinorama-style, and saddle surface modes from the Rust surface3d_demo.
• chart_gallery.py covers scatter, line, area, heatmap, contour, isoline, bar, pie, donut, box-plot, and treemap declarations.

Replace the local data-builder functions with measurement or application data when adapting a demo; the chart and Scene3D declarations remain unchanged.

Python Package

The Python declarations are packaged as gpui-toolkit with the import package gpui_toolkit. Wheels contain a private PyO3 abi3-py310 extension for pure native computations, while Rust/GPUI remains the rendering host. The base wheel has no Python runtime dependencies.

python -m pip install gpui-toolkit
python -c "from gpui_toolkit import native; print(native.AVAILABLE)"

Build a platform wheel with maturin (rather than invoking Cargo's cdylib directly):

maturin build --manifest-path Cargo.toml --release

The installed extension is gpui_toolkit._native; public code should import gpui_toolkit.native or the corresponding gpui_toolkit.d3rs functions. Native d3rs array statistics, search, tick generation, numeric/array and RGB/HSL/Lab/HCL/Cubehelix interpolation, color operations, and linear_scale execute the corresponding gpui-d3rs implementation. Substantial array work releases the GIL. Command request objects remain available for deliberately host-owned execution.

The package version matches the Rust crate version. Update both pyproject.toml and Cargo.toml together when releasing a new Python-facing runtime.

Showcase Application

Run the Python-authored native GPUI showcase with retained 3D scenes and embedded gpui-px charts:

cargo run -p gpui-python-runtime --features showcase --bin gpui-python-showcase -- crates/gpui-toolkit/gpui-python-runtime/python/showcase.py
PYTHONPATH=crates/gpui-toolkit/gpui-python-runtime/python ./venv/bin/python crates/gpui-toolkit/gpui-python-runtime/python/showcase.py

The complete gpui-d3rs gallery can use the same generic host with no d3rs-showcase-specific Rust route:

cargo run -p gpui-python-runtime --features showcase --bin gpui-python-showcase -- crates/gpui-python-runtime/python/d3_showcase.py

The abi3 wheel publishes nullable Boolean, integer, floating-point, and UTF-8 Dataset columns as Arrow IPC without Python dependencies. Install the optional Arrow adapter when nested or temporal columns need PyArrow's richer conversion support:

python -m pip install 'gpui-toolkit[arrow]'

Declare every live Dataset and ArrayData object on the application:

app = App(sections=(...), resources=(events, spectrum_grid))

The base App.on_session_ready publishes initial binary generations and subscribes to later commits. Overrides must call super().on_session_ready(context) before issuing application-specific commands. Resource values never enter the JSON application snapshot.

The base wheel remains dependency-free. Source-only use without either the native extension or PyArrow can still dump declarations, while starting a live dataset app raises a typed transport error instead of displaying an unpopulated chart.

The installed host negotiates resource_mmap_frames and creates a private 0700 directory with a high-entropy session token. Python publishes each complete dataset or ArrayData generation as a 0600 file; the host validates the token, local filename, size, permissions, and checksum, maps it read-only, and unlinks it immediately on Unix. The retained mapping is consumed directly by Arrow/table/chart and dense-array readers. Bounded binary stdout frames are used only when mmap transport is unavailable. Dataset and mesh frames carry payload checksums. Per-frame acknowledgements drive shared Python in-flight byte accounting, typed rejection reporting, backpressure, and deterministic fallback cleanup. Native consumers acquire generation-scoped leases. Publishing a new generation keeps an older mapped generation alive only while a retained consumer still owns it; explicit drops cannot invalidate active owners, and the final release reclaims both the mapping and its byte-budget charge.

Resource-backed scatter, line, area, boxplot, bar, pie, and donut declarations execute bounded LOD sampling from Dataset/DatasetView or numeric ArrayData frames. heatmap, contour, and isoline consume only a two-dimensional ArrayData grid ([height, width]) and never flatten it into JSON. One projection, one stable sort, and one range(...) execute in the host. Tables, aggregate pipelines, resource-backed charts, and static export execute the typed DatasetView.filter AST, including comparisons, boolean composition, membership, and null checks. Projection is enforced by Python, Rust IR validation, and each consumer. Tables sort before extracting their visible window. Tables, resource-backed charts, and static chart export execute one group_by(...).aggregate(...) stage over Arrow batches, then sort and consume only the bounded aggregate window. Supported reducers are count, sum, mean, min, max, first, and last. Scatter and line charts sort on their bound x or y role. Chart sort-plus-range remains explicitly rejected until it can preserve full pipeline ordering. Tables retain primary-key selection, scatter/line charts emit keyed point selections and viewport changes, treemaps emit keyed selection, and surfaces emit retained camera viewport changes.

Scene3D surfaces, line-strip points, mesh vertices, triangle indices, and scalar fields accept ArrayData. Their declarations contain only resource descriptors; the host resolves the exact generation and validates shape and dtype before creating native geometry or GPU state.

The showcase app, sections, UI kit demos, chart data, and scene3d specs live in Python. Rust loads the JSON UI IR, then owns GPUI, retained 3D renderer state, chart widgets, and theme integration.

Scene2D surfaces and game input

gpui_toolkit.scene2d declares small native drawing surfaces with stable object IDs. A Scene2D has a logical view box, ordered rect/circle/line/path/ text nodes, small transform groups, solid or linear-gradient brushes, local clips, bounded primitive shadows, an optional background and GridSpec, semantic labels, and input settings. Transitions can interpolate transforms, opacity, selected colors, and path reveal, then emit a requested completion ID once. The native host contain-fits that view box in the parent layout bounds. Python can publish a full scene2d_replace or a consecutive-revision scene2d_patch; SessionContext.scene2d_patch(...) can include related HUD operations in the same revisioned context.patch message. The wire operation nests its delta under patch: {"op":"scene2d_patch","id":"board","patch":{"base_revision":1,"revision":2,"upsert":[],"remove":[]}}.

Pointer, focused-key, lifecycle, and elapsed-tick messages specialize to Scene2DEvent and Scene2DTick. Scene2DInputConfig.keyboard=True scopes key delivery to the focused drawing surface; game handlers still choose which keys they use. Setting App.serial_reducer=True runs actions, surface input, ticks, effect results, command results, and profiler callbacks in one ordered lane. The lane has bounded input admission, reserved lifecycle capacity, and clears held input after overflow. Reducers should publish model-derived UI and scene changes together before returning. Accessibility activation of a semantic button or grid cell uses the same typed scene event path.

Handler exceptions report action_failed with the event, node, action, and exception type/message. The exception description is limited to 1,024 characters. Python stderr receives the full traceback for diagnosis.

App.tick_interval can seed an elapsed-time schedule. A reducer can adjust or cancel it later with SessionContext.set_tick_interval(seconds_or_none). The host owns its monotonic timer, pauses delivery for inactive sections and inactive windows, and resets elapsed time when the app or section resumes. Focus loss clears held keys and contacts even when the active section stays the same. Reducers should use elapsed time with a small catch-up bound and request ticks only while work is active. Native transitions animate target properties, so Python does not need to send patches every frame.

The games_demo.py example uses geometric Scene2D boards for Zip, Queens, Sudoku, Tetris, Chess, and Othello (one game module per file under games_demo/). Sudoku keeps a large keypad and selected-cell preview next to its board; the other boards expose semantic grid cells without drawing a widget button for each cell. Game cue commands go through the optional GameCueAdapter boundary. It accepts an already initialized CueBackend, uses a bounded nonblocking queue, and deduplicates recent command IDs. With no backend it remains silent. The current sotf-daw checkout has no Python cue binding, so the demo does not claim audio playback is connected.

The drawing surface and its input contract use native GPUI elements. The host owns the OS window lifecycle and Python reducer transport. This checkout's Python launcher runs CPython in a desktop child process, so games_demo.py is not bundled into the mobile showcase binaries. Mobile apps use the native Rust GPUI surface path in the existing iOS and Android backends; this delivery does not package a mobile Python interpreter.

Native input development API

Enable the local file API when launching a host built from this checkout:

GPUI_TOOLKIT_DEV_API=/tmp/gpui-dev \
  python3 crates/gpui-python-runtime/python/examples/games_demo.py
python3 scripts/python_dev_api.py /tmp/gpui-dev '{"command":"select","section":"queens"}'
python3 scripts/python_dev_api.py /tmp/gpui-dev '{"command":"click","surface":"queens-board","x":40,"y":40}'
python3 scripts/python_dev_api.py /tmp/gpui-dev '{"command":"status"}'

The API is disabled by default. Each host needs its own directory; clients submit one request at a time. scripts/python_dev_api.py handles atomic files and response IDs. A successful input response means the input was sent; inspect subsequent status responses to check Python results.

Commands:

• status: current section, committed application revision and IR, surface errors, captured Python stderr, and window bounds for surfaces and buttons.
• select: select an application section.
• window_click: send mouse down/up at window coordinates x, y through GPUI's normal event dispatch, layout hit-testing, and Python callbacks.
• click: send mouse down/up to a surface at logical scene coordinates x, y. The retained native router computes hit IDs and grid cells.
• pointer: route one surface sample with phase (down, move, up, cancel), device (mouse, touch, pen), contact_id, x, and y.
• key: route a surface key with phase (down, up).

Run the live regression with a built host:

python3 scripts/python_games_smoke.py target/release/gpui-python-host

It opens a separate native window with temporary presentation state, clicks Queens, Zip, Sudoku, Chess, Othello, and Tetris through GPUI, checks scene patches and live ticks, exercises two touch contacts, changes the palette, and closes its host. The test fails on native session errors or Python stderr. Handler errors include the event, node, action, exception type, and exception message; the full traceback is captured in stderr and available through status.

Platform Notes

Scene2D primitives paint through native GPUI elements and follow each platform's GPUI renderer (Metal on Apple platforms and the existing wgpu backend on Android). Scene3D and the other GPU-backed visualizations use wgpu via gpui-d3rs:

• macOS/iOS: Metal,
• Linux: Vulkan where available,
• Windows: DirectX 12 or Vulkan depending adapter support,
• Android: Vulkan through the existing GPUI Android backend.

The Python API is intended to stay the same across platforms; only GPUI backend initialization should differ.