Agent DX Environment Python SDK

adx-environment is the high-level Python client for the frontend environment v1 API. It creates and manages environments, including reusable Snapshots and a paused-environment lifecycle. It also provides the gateway routes used for reverse tunnels.

from adx_environment import Environment

with Environment(image="python:3.12-slim", cpu=2000, memory=4096) as environment:
    environment.files.write("/tmp/hello.txt", "hello world")
    print(environment.commands.run("cat /tmp/hello.txt").stdout)

close() releases local SDK resources and leaves the remote environment alive. kill() deletes a non-detached remote environment as well. The context manager explicitly calls kill() on exit; garbage collection does not make a remote delete request. If the block raises, its original exception remains visible even when cleanup also fails.

Server compatibility

The new Environment backend supports basic lifecycle, on-demand pause/resume, reusable snapshots, grouped placement, idle deletion/pause and restart policy. It requires an image containing the release EXECD at the configured command path; generic python:3.12-slim below is only an illustrative image name.

The Environment backend supports S3 rootfs, S3/image mounts, entrypoint inheritance, creation and runtime network policy, extra_config, failover=True, independent resource limits, and per-environment data-plane security. Declared user ports use Ingress and Relay routing. Public local rootfs paths and host mounts are not a tenant-facing contract. upstream reverse tunnel uses the published /tunnel/{environment} route; the legacy /invoke fallback remains unavailable. See the current HTTP contract.

OOM diagnostics

Command results and background-command info preserve exit_signal, such as SIGKILL=9. EXECD does not observe OOM. A user process can be killed while the Environment stays Running; a signal alone is not an OOM classification. The current sandboxd API does not expose a separate user-process OOM reason.

If the execution backend itself is OOM-killed, environment.get_info().failure contains an EnvironmentFailure(reason="OomKilled", exit_code=..., message=...) after the terminal state is committed. A failed direct invocation can raise EnvironmentError with code="ENVIRONMENT_OOM", retry="never", outcome="terminal" and failure details. Before publication, the caller may receive OUTCOME_UNKNOWN; query get_info() instead of interpreting route errors as OOM. Successful calls do not perform an additional catalog query.

Working directory

Environment(cwd="/workspace/task") sends the constructor directory to the server. Execd creates the directory and any missing parents before reporting Ready; the SDK does not issue a separate filesystem initialization call. An existing directory and its contents are preserved. Initialization failure makes creation fail through the normal lifecycle cleanup path.

Commands without an explicit cwd use the constructor directory. A per-command commands.run(..., cwd="/other") override must already exist; create it explicitly with files.make_dir() when needed. A failed command does not stop Execd. The constructor setting does not change Execd's own working directory or an inherited OCI entrypoint's WORKDIR.

Image startup process

Set inherit_entrypoint=True on a fresh image-backed environment to start the image's effective Entrypoint and Cmd as its managed workload:

environment = Environment(image="example/worker:latest", inherit_entrypoint=True)
exit_code = environment.wait_entrypoint()
print(exit_code, environment.entrypoint_exit_info)

Creation fails when the image has no effective startup command or cannot start it. A managed process may exit successfully; its terminal status remains available through bounded polling while the environment and its other APIs remain usable. Configure workload probes when creation must wait for a service.

Workload probes

from adx_environment import Environment, ReadinessProbe, HealthcheckProbe, EnvironmentError

try:
    environment = Environment(
        image="example/server:latest",
        inherit_entrypoint=True,
        init_timeout=120,
        readiness_probe=ReadinessProbe.http_get(port=8080, path="/ready"),
        healthcheck_probe=HealthcheckProbe.http_get(
            port=8080, path="/health", failure_threshold=3,
        ),
    )
except EnvironmentError as error:
    print(error.probe_failure)
    raise

Both probe types provide http_get(port=..., path="/"), tcp_socket(port=...), and exec(command=[...]) factories. HTTP and TCP check environment loopback; the port need not be publicly exposed. HTTP requires a 2xx response and does not follow redirects. Exec uses argv, the image workload's user, working directory and effective environment; exit code zero succeeds. Use an explicit shell argv when shell syntax is needed.

Setting	ReadinessProbe	HealthcheckProbe
initial_delay_seconds	0	0, measured after readiness
period_seconds	2	10
timeout_seconds	1	2
failure_threshold	retries until initialization deadline	3 consecutive failures

Execd runs readiness once per startup/resume/restore and only then reports Running. Adxlet activates routes and publishes Running after that report. init_timeout bounds runtime startup plus readiness; by default the SDK's total create budget is scheduling plus initialization plus a 30-second buffer. Readiness failure raises EnvironmentError with probe_failure and is terminal for that create attempt. Initialization is transient runtime state, not a separate persisted Environment lifecycle state.

Healthcheck starts after readiness (or immediately after runtime startup when readiness is absent). A success resets the failure count. At the threshold, Execd reports a sticky failure; Adxlet retires the execution and applies the existing restart/failover policy. environment.get_info().failure is a EnvironmentFailure(reason="ProbeFailed", ...); its probe_failure dictionary contains the probe kind, message, consecutive failures and available exit/HTTP status. Backend OOM and exit failures retain their existing reason values. Internal probes do not extend the environment's user-activity idle window. Pause/resume and restore cancel in-flight checks and reset healthcheck counts. The SDK checks server capability before a probe-dependent create; unsupported API servers return an error before submission. Use the matching release of API Server, Adxlet and Execd.

Lifecycle overview

There are three deliberately different checkpoint paths:

Path	Public SDK API	Artifact and placement	When to use it
Reusable Snapshot	create_snapshot() then Environment.create()	New Environment identity; shared storage allows fresh placement, local-only pins the source node.	Independent clones from a prepared source.
Pause / resume	pause() then resume()	Same Environment ID; Coordinator first tries the owner, then schedules a compatible node after definitive resource rejection or owner loss.	Stop and resume one logical environment.
Failure recovery	failover=True	Same Environment and node; restores the latest unexpired checkpoint after unexpected backend exit.	Workloads that must recover execution state rather than cold-start.
Explicit reload	reload()	Same Environment; replaces a Running backend from its latest unexpired checkpoint.	Operator-requested reset to a known recovery point.

The SDK is a client-side validation, request-ID, attempt, and result-shaping layer. Adxlet owns lifecycle and checkpoint bytes; Coordinator owns placement, committed state and the snapshot catalog.

Reusable Snapshots

Create a reusable Snapshot from an open environment:

from adx_environment import Environment

source = Environment(image="python:3.12-slim", name="source")
snapshot = source.create_snapshot(name="python-ready")

# The source stays running. `snapshot` is a SnapshotInfo value.
clone = Environment.create(snapshot, name="worker-1")

clone.kill()
source.kill()

The public signature is:

Environment.create_snapshot(*, name=None, timeout_seconds=300) -> SnapshotInfo

name is optional, but if supplied it must be a nonblank string. timeout_seconds must be an integral, non-boolean value from 1 through 3600. The result is a frozen SnapshotInfo(snapshot_id: str, names: tuple[str, ...]). The Snapshot has no TTL: delete it explicitly when it is no longer needed.

same_snapshot = Environment.get_snapshot(snapshot.snapshot_id)
snapshots, next_page_token = Environment.list_snapshots(
    name="python-ready", page_size=20,
)
Environment.delete_snapshot(same_snapshot.snapshot_id)

The source must have a valid lifecycle identity. In particular, an Environment with an active reverse tunnel cannot create a reusable Snapshot. A successful Snapshot leaves the source running and publishes an immutable artifact; the server records READY metadata only after that artifact is available. The SDK does not expose the reusable-Snapshot request ID and generates a fresh identity for each call. Raw HTTP clients that retry an uncertain request must reuse it only for the same source and name; they must not reuse one request ID for different catalog content. The SDK keeps its local tunnel client active while the checkpoint request is in flight; expected reconnect noise during that scope is logged at debug level without changing the checkpoint result.

Create from Snapshot

Use Environment.create(snapshot_id, **kwargs) with either a SnapshotInfo or a Snapshot ID string. It performs the normal environment create request with a snapshotId, so the result is a new Environment handle that reaches RUNNING in the usual way.

clone = Environment.create(
    snapshot,
    name="worker-2",
    # Omitted resources inherit the snapshot; explicit values must match it.
)

The new Coordinator inherits omitted image/runtime/scalar resources and validates explicit values against the source geometry. It does not resize a restored VM. Environment overrides and placement constraints are carried into the new Environment; local-only snapshots require the source node. Shared snapshots use normal scheduling. The source briefly pauses during snapshot creation and resumes before success. A reusable snapshot is not consumed by cloning; deletion blocks new references and waits for existing references to be released. See storage and cloning.

Later dual-clone FC runs exposed a network failure, tracked in the investigation; successful earlier batches do not close that issue.

Pause and resume

Pause one open environment and later resume the same logical environment:

from adx_environment import RetentionPolicy, Environment

environment = Environment(image="python:3.12-slim")

paused = environment.pause(retention=RetentionPolicy.for_seconds(90_000), timeout_seconds=300)
print(paused.snapshot_id, paused.expires_at)

running = environment.resume()
print(running.route_address, running.port_mappings)

The public signatures are:

Environment.pause(retention=None, *, timeout_seconds=300) -> PauseResult
Environment.resume() -> ResumeResult

retention=None inherits pause_policy.retention; the default policy retains the restore point for 90,000 seconds. Use RetentionPolicy.for_seconds(seconds) for a positive integer TTL or RetentionPolicy.forever() to retain the point until explicit deletion or replacement. Finite retention begins after publication. Permanent pause returns expires_at=None; the stored record and RPC use an absent expiry, rather than a large timestamp. PauseResult also contains environment_id, snapshot_id, byte size, and state="paused".

timeout_seconds is keyword-only, integral and non-boolean, from 1 through 3600. It bounds the sandboxd capture RPC and sets the SDK's per-attempt HTTP wait with an extra 30-second transport allowance. It is not a total pause deadline: OBS publication follows capture and uses its own storage request timeout. Losing the HTTP reply does not cancel an accepted lifecycle operation; its outcome can remain unknown and the SDK retries the same operation identity.

resume() returns a frozen ResumeResult with environment_id, state, route_address, function_proxy_id, node_id, and port_mappings. The SDK requires the response to identify this environment, report "running", and include a route address and function-proxy ID. Resume uses Coordinator EnsureRunning, first admitting on the owning node. A shared checkpoint allows another compatible node after definitive NoCapacity or owner invalidation; an uncertain RPC result retains ownership and retries there. The selected node restores the checkpoint, rearms the runtime listener and commits Running. Route-cache convergence after that result is outside the resume success boundary.

Pausing validates the configured store's artifact and commits Paused after removing the old execution. Local-only files are immediately readable on the owning node; adxlet does not force filesystem writeback, so publication does not guarantee survival of a host crash or power loss. S3 publication waits for upload completion. A SQLite-only result is not cluster success. Public resume uses authoritative ownership and a valid checkpoint. Local-only points remain tied to the original node; shared points preserve the Environment ID across a new ownership generation. Node loss preserves Paused until demand, expiry or deletion.

Idle pause and traffic resume

from adx_environment import CheckpointStorage, IdleAction, PausePolicy, RetentionPolicy, Environment

environment = Environment(
    image="example/execd:latest",
    idle_timeout=60,
    idle_action=IdleAction.PAUSE,
    pause_policy=PausePolicy(retention=RetentionPolicy.forever(), timeout_seconds=300, storage=CheckpointStorage.SHARED),
)

idle_action is an IdleAction enum and defaults to IdleAction.DELETE; IdleAction.PAUSE requires a positive idle timeout and a checkpoint-capable runtime. CheckpointStorage.CONFIGURED uses the node backend, CheckpointStorage.LOCAL retains files on that node, and CheckpointStorage.SHARED requires complete publication to its configured object store. Configure an OBS S3-compatible endpoint on every eligible node, with the same alias/bucket/prefix. The shared setting also limits creation placement to nodes advertising shared checkpoint storage.

An automatically paused environment also supports environment.resume() explicitly. It uses the same restore intent as traffic-triggered wake-up, so concurrent manual and automatic requests do not start two runtimes. The checkpoint must still be valid and unexpired; local-only restore requires the owning node. The idle policy stays enabled after resume.

The existing idle window observes Execd client operations and Relay streams; health, metrics and command observation do not keep the environment busy. A detached background command can be checkpointed. Automatic pause atomically closes new admission and rechecks activity. New demand before capture cancels pause; demand once capture starts waits for publication and restores afterward.

Authorized business traffic through Ingress shares the manual resume entry for both automatic and manual pauses; no auto-resume opt-in is required. Concurrent first requests coalesce into one durable intent. Ingress waits for the committed Running route before sending application bytes; passive route subscriptions and health/metrics do not wake a paused environment. Requests already sent upstream are not replayed. Transient owner RPC failures retry on the same owner within the resume budget. A temporary wake failure before forwarding returns HTTP 503. A finite SDK HTTP request sends X-ADX-Processing-Timeout-Ms: authentication, route lookup, wake, route delivery, connection admission, forwarding and response streaming share one Ingress deadline. The SDK defaults to 300 seconds for a data request and adds a configurable communication margin (30 seconds by default) to its HTTP timeout. Without the header, wake and route delivery share a 300-second default; existing forwarding and streaming lifetimes remain unchanged. Negative invoke timeouts omit the header and retain unbounded application HTTP waits.

If readiness is still pending when the deadline expires before application forwarding, Ingress returns HTTP 503 with X-ADX-Error-Code: ENV_RESUMING, Retry-After: 1, and a JSON error containing code: ENV_RESUMING, retry: same_operation, outcome: not_started, requestForwarded: false, requestId, and environmentId. The durable restore continues independently. SDK retries retain the request ID and body; each allowed retry gets the same configured timeout plus communication margin. Retry count is configurable via ConnectionConfig.request_max_attempts / ADX_REQUEST_MAX_ATTEMPTS (default 3, including the first data-path attempt). Known-unsent connection failures and absent direct routes retain their existing single control-plane fallback; ENV_RESUMING never takes that fallback. Retry-After and jitter apply between demand retries. Exhaustion raises EnvironmentHTTPError with the same ENV_RESUMING contract; callers can choose to try again. No sticky Ingress routing is required: replicas share the Coordinator intent and independently subscribe to the committed route.

HTTP 504 after forwarding carries PROCESSING_DEADLINE_EXCEEDED, outcome: unknown, requestForwarded: true, and retry: never; it is not a safe demand retry. If response headers have already been delivered, deadline expiry ends the body stream instead. Existing idempotent invoke transport retries keep the same EXECD operation ID and never cross into a control-plane fallback after an unknown outcome. File bytes and seekable file bodies support safe demand retry. Consumed iterators are not replayed automatically and surface the structured error instead. Expired or unavailable checkpoints return HTTP 409 ENV_UNAVAILABLE, retry: never.

A timeout may leave the durable operation pending. If every node rejects admission, rejected nodes are excluded for a 30-second capacity round before reevaluation.

Recovery methods

failover=True restores from the latest unexpired checkpoint after an unexpected backend exit on the owning node. If no valid checkpoint exists, the Environment becomes Failed and is not recreated from the original image. Environment.reload() -> bool explicitly replaces a Running backend from that same recovery point. It returns True only after the replacement reaches Running and the authoritative result is committed. Automatic restart policy is a separate cold restart mechanism. Shared-checkpoint node-failure recovery is coordinated by Coordinator and is not selected by either public call.

Timeouts, attempts, and errors

For create_snapshot() and pause(), timeout_seconds defaults to the public ADX_GET_DEFAULT_TIMEOUT value of 300 seconds. It is the logical server checkpoint timeout sent as timeoutSeconds; each SDK HTTP attempt uses that logical value plus a 30-second transport buffer. resume() and reload() have no public logical-timeout body field and use the SDK default plus that buffer for each transport attempt. SDK argument errors are raised before any request; closed create_snapshot, pause, and resume handles raise RuntimeError.

The SDK generates request IDs; callers of Environment cannot set them. Its attempt rules intentionally differ by operation:

• Reusable Snapshot creation makes one SDK transport attempt only. A connection problem or gateway 502/503/504 produces EnvironmentError with an uncertain outcome, because the immutable Snapshot might already have committed. Reconcile it explicitly instead of assuming that another attempt is safe.
• Pause, resume, and reload make up to three transport attempts for transient transport or gateway failures, all with one internal request ID. If the final result is still uncertain, reconcile the instance state instead of changing the request under that identity.
• Create from Snapshot uses the normal create policy with up to three attempts and one create-* identity. When the caller omits name, the SDK derives a stable name from that identity, so another API Server replica receives the same Environment ID. An uncertain result must still be queried or retried with the same identity.
• Environment.delete() treats HTTP 404 as an idempotent success. HTTP 403 raises PermissionDenied with the request ID and is not retried; transient transport or gateway failures retain one request ID across bounded retries.

Structured server failures raise EnvironmentHTTPError, whose code, retry, outcome, request_id, operation_id, and environment_id fields implement the management error contract. Transport failures with an uncertain write result surface the same fields on EnvironmentError. Other malformed typed-result shapes can instead surface ValueError or TypeError while values are converted, or RuntimeError when resume portMappings is not an object. reload() translates EnvironmentError into False; it does not normalize those other programming/shape exceptions.

SDK versus raw HTTP

These are SDK semantics, not a substitute for the frontend REST contract. The SDK uses these paths internally:

POST /api/environment/v1/environments/{id}/snapshots
POST /api/environment/v1/environments                 # create from Snapshot: snapshotId
POST /api/environment/v1/environments/{id}/pause
POST /api/environment/v1/environments/{id}/resume
POST /api/environment/v1/environments/{id}/reload
PUT  /api/environment/v1/environments/{id}/network

Raw HTTP has intentionally different validation in a few places. A raw Snapshot request has name as its handler field: an omitted or empty name is accepted, while a supplied whitespace-only name is rejected. Its timeoutSeconds defaults to 300, is validated in the range 1 through 3600, converted to milliseconds, and forwarded as the checkpoint RPC logical timeout. Raw Pause accepts optional retention as {"kind":"forever"} or {"kind":"ttl","ttlSeconds":3600}; omission inherits the creation policy. Its timeoutSeconds is validated from 1 through 3600 and defaults to 300. Only Snapshot and Pause currently accept this caller-provided logical timeout body field. The SDK sends {} for resume and reload, and their handlers define no body fields. Raw HTTP requires a pattern-valid X-ADX-Request-ID header; the SDK generates that header internally.

Internal signal/POSIX/runtime SDK APIs are not part of this package or the new Environment protocol.

Other create options

Request one kind of whole device with type:model:count:

Environment(xpu="gpu:l20:1")
Environment(xpu="gpu:h100:2")
Environment(xpu="gpu::1")  # any GPU model
Environment(xpu="npu:ascend910b4:1")

The SDK accepts one gpu or npu request with a positive whole-device count. An empty model leaves the scheduler to select a model. Real device execution requires a suitable worker and is a separate pending validation gate; see the whole-device acceptance.

Temporary writable storage is specified in MiB:

Environment(storage_mb=153600, storage_limit_mb=153600)

storage_limit_mb=0 uses storage_mb, or the cluster default when storage_mb is omitted. A nonzero limit cannot be below storage_mb. The request controls scheduling while the limit is passed separately to sandboxd's writable-layer enforcement.

Rootfs and mounts

rootfs=S3Config(...) starts from an S3-compatible EROFS root filesystem. Mount supports image-backed read-only bind mounts and S3-backed bind or EROFS mounts. Credentials are sent to the control plane and sandboxd; callers should use scoped object-store credentials. EXECD is still supplied by the deployment's local runtime environment, so a custom rootfs does not need to bake in ADX.

The deployment Runtime Environment is the rootfs baseline. runtime= overrides only its isolation runtime. rootfs_readonly=True or False overrides only its read-only setting; leaving it as None inherits the deployment default. Passing image= or rootfs=S3Config(...) replaces the source atomically while omitted runtime/read-only fields continue to inherit. Runtime-only and read-only-only overrides keep the baseline source and do not add the bootstrap mount.

Network policy models

NetworkPolicy, NetworkRule, and PortRange are accepted at creation.

Deny-all policies retain platform rules for Execd and declared public TCP ports. Stateful mode uses connection tracking for replies; stateless mode matches those local service ports in both directions. When runtime control is configured, a separate rule preserves the node's exact control IPv4/TCP endpoint. Ordinary traffic remains subject to the user policy; see the node network contract. update_network_policy(policy) atomically replaces the complete runtime policy; passing None clears it. Adxlet reserves the EXECD control port and declared published ports with the highest rule priority so a user default-deny policy cannot cut the control route. User priorities must be in 1..UINT32_MAX-1. Enforcement is provided by sandboxd's network policy implementation.

Connection configuration

Pass an immutable ConnectionConfig to avoid process-global environment configuration:

from adx_environment import ConnectionConfig, Environment, resources

connection = ConnectionConfig(
    server_address="frontend.example.com:443",
    token="<token>",
    use_tls=True,
    gateway_address="gateway.example.com:443",
    gateway_use_tls=True,
    communication_margin_seconds=30,
    request_max_attempts=3,
)

with Environment(image="python:3.12-slim", connection=connection) as environment:
    print(environment.id)

nodes = resources(connection=connection)
Environment.delete("environment-id", connection=connection)

resources() reports scheduling capacity and remaining allocatable resources. Online nodes that are not accepting allocations remain in the result with status 1, zero CPU/Memory/Disk resources and no GPU/NPU resource entries. Existing allocations remain held; this view does not report actual utilization or the rated capacity of admission-disabled nodes.

Without a ConnectionConfig, the SDK reads ADX_SERVER_ADDRESS, ADX_TOKEN, ADX_TLS, ADX_GATEWAY_ADDRESS, ADX_GATEWAY_TLS, ADX_COMMUNICATION_MARGIN_SECONDS, and ADX_REQUEST_MAX_ATTEMPTS. The communication margin must be finite and between 0 and 3600 seconds; request_max_attempts must be a positive integer. The gateway address defaults to the frontend address for reverse-tunnel and user-port routes. PTY uses the Ingress data-plane route when a gateway address is configured, and falls back to the server address for combined deployments.

Build and test

From this directory:

PYTHON=python3 bash build.sh /tmp/adx-environment-dist
PYTHONPATH=. python3 -m unittest discover -s tests/unit

The SDK version is defined once in pyproject.toml, matching the AKernel SDK packaging model. Change [project].version when publishing a new SDK release; the build does not infer or override it from repository tags or environment variables.

From the monorepo root, the SDK wrapper does the same:

PYTHON=python3 bash platform/sdk/environment/build.sh /tmp/adx-environment-dist

Live K8S/frontend checks also need ADX_SERVER_ADDRESS, ADX_GATEWAY_ADDRESS, and a valid token:

PYTHONPATH=. python3 tests/e2e_execd_direct.py
PYTHONPATH=. python3 examples/reverse_tunnel.py

Runnable examples

Examples are client demonstrations. Basic commands/files are supported; tunnel and user-port examples require matching published routes and are not part of the new basic K8s acceptance:

• examples/basic_usage.py
• examples/command_stdin.py
• examples/persistent_shell.py
• examples/tunnel_large_response.py
• examples/port_forwarding.py
• examples/reverse_tunnel.py
• examples/named_environment.py
• examples/bench_cp.py
• examples/recoverable_command.py

Infra-specific demos should be documented separately instead of being shipped as runnable SDK examples.

Architecture

• Control plane — environment v1 create, delete, lifecycle, and invoke routes.
• Direct data plane — frontend/gateway /direct/{environment}/... routes for command invoke and binary file I/O.
• Reverse tunnel — gateway /tunnel/{environment} WebSocket back to a local upstream (adx_environment/tunnel_client.py).

See TODO.md for remaining SDK work.

Recover a background command

Background commands run without an execution deadline by default. Pass timeout to set one explicitly; foreground commands default to 60 seconds.

The SDK process does not persist command handles. Generate a command ID and persist the (environment_id, command_id) pair in the caller's own database before submission, then bind fresh handles after a restart:

from adx_environment import Environment

environment = Environment.from_id(saved_environment_id)
command = environment.commands.run(
    "python train.py",
    background=True,
    command_id=saved_command_id,  # generate and persist before submission
)

# In a later SDK process:
environment = Environment.from_id(saved_environment_id)
command = environment.commands.get(saved_command_id)
result = command.wait()

wait() and wait_async() use one hidden multiplexed WebSocket per connection context as a notification channel. The final result is always read back from EXECD's authoritative command registry. When the wait deadline expires, they return CommandResult(status=RUNNING, error_code="WAIT_TIMEOUT"); the remote command remains active and the handle can wait again. A client restart also does not terminate the remote command. CommandHandle.kill() and environment.commands.kill() return False when the command is absent or already finished, and still raise on other execution or transport failures. Long-running command polling retries connection failures and errors marked retryable by the server. Terminal errors keep their original error and request ID. Temporary downstream subscription failures (including a checkpoint freeze) reconnect the watch for the same command, without starting it again. The SDK uses a 30-second continuous-outage budget; an Ingress WebSocket handshake alone does not reset it. Receipt of an actual PENDING/RUNNING or terminal command state resets the budget. Authorization and permanent subscription errors fail directly.

Placement

Environment(..., labels={"app": "worker"}, schedule_affinities=[...]) exposes the platform's grouped node and same-tenant instance placement rules. Conditions use the public scheduleAffinities shape with kind, affinity, labelOps, optional weight and preferredPriority. node_id remains a hard requirement when combined with alternatives. See placement rules for the matching, ranking and label contracts. Caller-provided option lists are copied before the SDK adds a node constraint.