sequence-ai

Cloud inference for robot policies.

pip install sequence-ai

import sequence_ai

with sequence_ai.connect(model="pi05-droid") as policy:
    out = policy.act(
        "pick up the cup",
        observe=robot.read_observation,   # cameras + joints
        act=robot.apply_action,           # one joint target
        validate=robot.is_safe,
        hold=robot.hold,
        max_seconds=30,
    )

policy.interrupt()      # from any thread, at any moment

Two brains, one string between them

A VLM looks at a snapshot, decides what to do next, and says it in a sentence. This package is the how: it takes that sentence with the current cameras and joints, asks a vision-language-action model for the next second of joint targets, and plays them out while fetching the next second.

The two halves run at completely different speeds — the brain thinks between subtasks, the arm needs a chunk every second — and instruction is their entire interface.

act() is shaped to be called as a tool. It overwrites the observation's instruction on every call, so your eyes never have to know what the brain last decided and a stale sentence cannot leak into a chunk after the brain has moved on. max_seconds is required and has no default: this drives hardware, and a tool call a language model can start but nothing bounds is one an arm can be left running by a dropped conversation.

Wiring the two together — retries, failure detection, choosing the next sub-goal — is yours. This package provides inference and a control loop, not orchestration.

Changing your mind

policy.interrupt()

Thread-safe, and the only method meant to be called from a thread other than the one driving.

It stops within one control period, not one chunk. The loop checks between every action, so at 15 Hz the arm stops in about 67 ms. Waiting for the current chunk to finish playing would be up to a full second of an arm still reaching for something the brain has already given up on.

It also cuts through a stalled fetch. If the buffer has run dry and the loop is blocked on a chunk that is taking five seconds, an interrupt does not wait that out — stop latency is a property of this loop and never of the service.

Interruption unwinds through hold() like any other early stop, and is not raised: a stop you asked for is an outcome, not an error. RunOutcome.reason says which ending happened.

What it does for you

Four things, and each of them is something a loop written straight against the HTTP endpoints has to get right before it behaves properly on a robot:

None of the four is guesswork. Every one is a defect that existed in this client, was measured, and was fixed — the numbers below are those measurements, not estimates.

It keeps the connection open

Measured against the production gateway, six samples each:

184.9 ms per call — 60% of the total — is TCP and TLS handshake. That is what with sequence_ai.connect(...) removes. Nothing proprietary: any HTTP client that reuses a connection gets the same result. This package just makes it the default rather than something you have to remember.

Why it matters: a chunk is a deadline. pi05-droid returns 1.00 s of motion per call, and a warm end-to-end /v1/act through this gateway measured 793 ms — the next chunk has to arrive before the current one finishes playing, so 185 ms of avoidable handshake is a fifth of the entire budget.

Actions come in chunks

One call returns a block of future actions, not a single command — between 0.5 s and 2.1 s of motion depending on the model. That is why cloud inference works at all: the control loop does not need a network round trip per control step.

with sequence_ai.connect(model="pi05-droid") as policy:
    pred = policy.predict(observation)
    print(pred.action_chunk)
    # ActionChunk(15 steps x 8 dims, joint_delta, droid_franka, 15 Hz, covers 1.00s, replan after 8)

Read joint_delta in that line before you use the rows. It is the field that decides what the numbers mean, and pi0.5-DROID's are increments — feeding them to a controller that expects absolute joint targets is invisible: same shape, same dtype, values in the normal range, no error, and the arm never completes the task. replan after 8 is the second one: executing all 15 rows when the model asks to be replanned after 8 fails the same quiet way.

You do not have to handle either one. Declare what your controller takes and commands() applies both, plus the gripper exception and the value clip:

policy = sequence_ai.connect(model="pi05-droid", controller="joint_position")
for target in policy.commands(observation):     # already absolute, already 8 rows
    arm.move_to_joint_positions(target)

What to do with the rows covers both controllers and the by-hand route. If you are wiring this into an existing harness rather than starting from run(), read that section first — it is the only part of this document where getting it wrong produces no error at all.

The gap between chunks is the hard part

A loop that waits for the buffer to empty before asking for the next chunk stops the arm once per chunk, every chunk, for a full round trip. It is not an occasional hiccup — it is structural, and on the faster models it dominates:

So run() starts the next inference while the current chunk is still playing, and reports what happened rather than hoping:

out = policy.run(observe=..., act=..., max_actions=250, on_underrun=robot.hold_position)

print(out)        # RunOutcome(250 actions over 17 chunks in 16.8s, completed)
out.underruns     # times the buffer ran dry before the next chunk arrived
out.underrun_s    # total seconds the arm spent with no command
out.max_seam_jump # largest per-dimension step across a chunk boundary

The trade, stated plainly: a prefetched chunk is computed from an observation taken before the previous chunk finished, so the overlap window is open-loop. Freshness and continuity are in direct opposition here and no setting gets both. prefetch=False restores strictly closed-loop behaviour, stall included — right for bench work, wrong for a moving arm.

max_seam_jump is a measurement, not a correction, and it stays one unless you ask otherwise. smooth_seam=N ramps the first N steps of each new chunk out of the last executed action, and it is the only setting in this library that changes a number on its way to the motors — so it is off by default and guarded twice: it applies only when the chunk declares an absolute action space (action_chunk.action_space), and never to the first chunk of a run. Blending deltas is not smoothing; it rescales the increments and moves the arm somewhere the model never asked for, so a delta chunk is passed through untouched even when you ask. max_seam_jump is measured before any blending, so turning it on cannot hide what it smooths.

Many robots at once

One Policy per control loop. A Policy holds one action buffer, and two loops popping from it do not take turns — they interleave. Driving the same handle from a second thread raises, because both quieter options are worse: interleaving sends each robot a shuffled half of the other's plan while both loops report success, and a blocking lock would make the second loop run at half rate, underrunning on every chunk.

def drive(robot, model):
    with sequence_ai.connect(model=model) as policy:      # one each
        return policy.run(observe=robot.read, act=robot.apply,
                          validate=robot.is_safe, max_actions=250)

with ThreadPoolExecutor() as pool:
    left, right = pool.map(drive, [arm_l, arm_r], ["pi05-droid"] * 2)

This costs nothing to follow. The handshake is paid once per Policy, not once per call. Measured with eight loops running concurrently against one gateway: eight connections, ten requests each, zero sequence breaks, zero underruns. Requests arriving while the gateway is busy queue on the server rather than displacing work already in flight.

What the loop says while it runs

run() reports lifecycle, not telemetry:

  starting…      first chunk on its way, or a cold worker loading — nothing is moving yet
  running        a command has reached the robot
  done

Waiting out a cold worker (three to six minutes) and losing a few hundred milliseconds at a chunk boundary are the library's problems, not yours — startup_timeout_s defaults to 1800 s, so the first call on a cold worker waits rather than failing. Every number is still in RunOutcome afterwards if you want it.

policy.run(..., on_status=log.info)     # programmatic
policy.run(..., progress=False)         # silent

The default progress="auto" writes a single line to stderr only when stderr is a terminal — nothing in a script, a pipe, or a log file.

Cold starts

A worker that is not loaded takes three to six minutes to become ready — tens of gigabytes of weights fetched, extracted and JIT-compiled. A control loop's budget for one chunk is 533 ms. Those two numbers are why connecting is a separate call from controlling.

Measured end to end, request to first action:

The worker stays warm for 10 minutes

This is the number that decides how you structure a session. After it becomes ready, a worker is held for 10 minutes past its last request. Every call inside that window is warm — the table's right-hand column. The clock restarts on each request, so a loop that calls even once a minute never goes cold.

Cross that 10-minute gap with no calls and the worker is released, and the next request pays the full cold start again.

call ──► 180-215 s cold start ──► warm ◄──────── 10 min idle window ────────► released
                                    ▲                                            │
                                    └────── any call restarts the 10 min ────────┘

Why ten and not thirty. A held GPU bills whether or not you call it, and the shape of that bill is brutal for a session with thinking time in it: over one real day of use, inference was 3% of the GPU seconds paid for, cold starts 19%, and idle holding 77%. Thirty minutes made the worst case a session that calls once every twenty-five minutes — never cold, almost never working, and billed for all of it. Ten covers a loop and a short pause between runs without paying for a coffee break. {measured 2026-09-16 across 2,147 calls: 322 s of inference against 7,200 s of idle holding}

The practical consequence, if you are running a batch of experiments: the gap between them is what costs you, not the number of them. Ten tasks run back to back pay one cold start. The same ten with a 20-minute analysis gap between each pay ten — that is up to half an hour of waiting that looks like the model being slow and is actually the fleet being released and rebuilt.

If a gap is unavoidable and you want the worker held, keep it alive with a cheap call inside the window; wait_until_ready() is the right one, because it touches no GPU and is not billed.

Connect first, then drive

policy.wait_until_ready()          # once, before the robot needs to move
while running:
    policy.next_action(observe())  # every one of these is warm by construction

wait_until_ready() needs no observation — a robot should be able to bring its model up before it is in position, which is exactly when it has no frame worth sending. It is authenticated but not billed: it touches no GPU. It is also what starts the worker, so polling it is the thing that brings the model up, not merely a way to watch.

ready() is the non-blocking form, returning (ready, eta_seconds).

Skipping it does not fail — it silently costs you the loop. Measured on one run against a cold endpoint, the client starting anyway:

first control request   13,533 ms      <- the cold start, now inside the loop
p50                        415 ms      <- the steady state was always fine
mean                     1,096 ms      -> 0.49 arms sustainable   
mean without that one      441 ms      -> 1.21 arms sustainable   

One request that should not have been in the loop is the entire difference between sustainable and not.

If you skip it anyway

The gateway does not stall on a cold worker — it answers 503 immediately with the number:

try:
    policy.predict(observation)
except sequence_ai.Unavailable as exc:
    if exc.warming:
        print(f"loading; ready in ~{exc.retry_after_s}s")   # 174

run() waits that out for you by default (startup_timeout_s=1800), and only before the first action. It reports starting… while it does, so a wait is never mistakable for a hang:

policy.run(..., startup_timeout_s=0)     # opt out: fail immediately on a cold worker

That line is the whole design. Before the first action nothing is moving, so waiting is free. Once the arm is in motion, silently pausing it for two minutes and resuming from a two-minute-old plan is worse than stopping — so mid-run warming is raised, and hold fires.

How big your frames are is how fast you go

Latency scales with the bytes you send, at roughly 16 ms per kB. This is the single largest thing under your control, and it is larger than the model:

observation    p50      mean     sustainable arms
14.6 kB      586 ms    623 ms         0.86        
 3.4 kB      384 ms    433 ms         1.23        

Measured through the gateway, alternating A/B against one warm worker over one connection so that drift in the service cancels. inference_ms did not move between the two (119 ms against 111 ms) — every millisecond of the difference was transport. A direct measurement against the worker agreed: 181 ms for the same 11.2 kB.

Send JPEG, not arrays. A frame as a list of integers is roughly 16x the bytes of the same frame as base64 JPEG, and it is the most common way to land on the slow side of that table.

Use quality 95. Do not go below 90. With the sampling noise pinned, on a real DROID frame:

Quality 85 and below is where the error jumps by an order of magnitude. This section previously read "quality 85 is safe" and quoted the 0.51% figure next to it — but 0.51% is q95's number, and q85 is nearly four times worse. sequence_ai.encode() defaults to 95 so the decision is not yours to get wrong.

Hand it the native frame and let it size it

Do not downscale before calling encode(). Give it whatever your camera produces; it reads input_square and input_aspect_ratio off the catalogue and resizes once, straight to the block the model's own transform produces — 224×126 for pi0.5-DROID. Downscaling first makes it two resamples, and the intermediate size has no claim to being right: training_frame_size is what the dataset stored, not what the model reads.

At that size it encodes PNG, not JPEG, and quality is ignored. The table above is about a real trade-off higher up; at the model's own input there is nothing left to buy:

Fifty-one times smaller and bit-identical — it runs the resize openpi would have run, with openpi's own filter, so the result is the same array. That last part is load-bearing: 0.7.0 and 0.7.1 used a different filter and deviated by 0.78 while claiming to be exact. The codec is the other error term, and the larger one, which is why compression stops once the frame is small.

A model that does not publish input_square is capped at 640 on the long edge and JPEG-encoded as before, so nothing breaks; it simply does not get the saving.

Fidelity below that is not a gentle slope. Degrading the frame further does not degrade the policy proportionally; it stops working. Quantising each channel to four levels — layout, edges, shadows and hues all preserved, a milder change than any sane JPEG setting — took pi0.5-DROID from placing the cube in the bowl in 5.3 s to never lifting it at all, with its closest approach 20x worse than the successful run's final distance. Measured in simulation, 2026-09-11.

Four numbers you should never have to remember

Which camera views a model wants, how wide its state vector is, what aspect ratio it trained on, and what its returned floats mean. All four are per-model, none is guessable, and every one of them fails silently — an unknown view name is not rejected, it arrives as an empty camera; a short state vector is not rejected, it is zero-padded. The robot moves either way; it just moves on something other than what you meant.

So read them off the catalogue instead of remembering them:

model = {m.short_id: m for m in sequence_ai.models(endpoint="act")}["pi05-droid"]

obs = sequence_ai.observation(
    model,
    images={"exterior_1": cam.read(), "wrist_left": wrist.read()},   # ndarray or PIL
    joints=arm.joint_positions(), gripper=[arm.gripper_position()],
    instruction="pick up the cup",
)

That centre-crops each frame to the declared ratio — cropping, not padding, because the model's own transform already pads and doing it twice fences the scene with bars on all four sides — encodes at quality 95, and raises locally, before anything is billed, if the views or the state width do not match what the model declares. policy.predict() runs the same check and warns when you build an observation yourself; connect(validate="strict") makes it raise, "off" skips it.

state_dim is the input width and is not action_dim. Across the served catalogue it is 8, 17, 8 and 0 — the last meaning that model never reads joints at all, so you may omit them. Read it off the model, not off this sentence.

state_layout says how that width splits — {"joint_positions": 7, "gripper": 1} for pi0.5-DROID — and the halves are not interchangeable. Eight joints with no gripper sums to 8 and is accepted; the eighth is then dropped and the gripper zero-filled, which holds the hand open for the whole episode. observation() checks each half, not the sum.

What to do with the rows that come back

Tell it what your controller takes, once, and stop deciding.

policy = sequence_ai.connect(model="pi05-droid", controller="joint_position")
for target in policy.commands(observation):
    arm.move_to_joint_positions(target)

That is the whole loop body. commands() applies the model's action space, its conversion reference, its gripper exception and its replan horizon, and hands back rows that are already what your controller expects — the list it returns is what should be executed before observing again, not the whole chunk with a footnote.

controller has two values and no default, because guessing is exactly the failure this removes:

The reference for the increments is taken from the observation you just sent — that is "state_at_request" — so there is no second value to read at the wrong moment.

env = RobotEnv(action_space="joint_velocity", gripper_action_space="position")   # DROID stack
chunk = policy.predict(obs).action_chunk
for i in range(chunk.open_loop_horizon or len(chunk)):
    env.step(sequence_ai.prepare(chunk, model, i))

q_ref = arm.joint_positions()                  # ONCE, before the chunk is requested
for target in sequence_ai.to_joint_positions(chunk, q_ref, model):
    arm.move_to_joint_positions(target)

Three ways to drive it

From most control to least. They are the same request underneath; the difference is who owns the loop.

policy.predict(obs)            # the whole chunk, you do everything
policy.next_action(obs)        # we hold the buffer, you own the cadence
policy.run(observe=, act=)     # we own the loop
policy.act("...", observe=, act=)   # we own the loop and the instruction

act() is run() with the instruction pinned and a time bound required — the shape a VLM calls as a tool. Everything run() accepts, act() accepts.

Safety

An action returned by any model is model output, not a safe robot command.

This library does not check joint limits, reachability, collisions, or whether a step is safe at the robot's current velocity. Bounds checking, a watchdog and an e-stop belong between this library and your motors.

policy.run(
    observe=robot.read_observation,
    act=robot.apply_action,
    validate=robot.is_safe,   # return False to stop the loop
    hold=robot.hold,          # called if it stops early, or if act() raises
    max_actions=250,
)

max_actions is required and keyword-only. There is no run_forever() — an unbounded loop that moves a robot should not be startable by accident.

validate=None is allowed for bench and simulation work, and warns once so it cannot happen silently on real hardware.

Errors are typed

Because a controller reacts differently to each:

Configuration

export SEQUENCES_API_KEY=seq_live_...      # or pass api_key= to connect()
export SEQUENCES_BASE_URL=...              # for staging; defaults to production

Get a key at app.generalsequences.com.

Install footprint

One dependency: httpx. Python 3.9+.

A robot controller is often on a Jetson with a pinned, fragile Python environment, and ROS 2 Humble ships Python 3.10. Every transitive dependency is another chance for the install to fail on the machine that actually matters.

If your controller is not Python

The endpoints underneath are public and documented at generalsequences.com/docs. There is no private control plane and nothing this package reaches that you cannot:

curl https://api.generalsequences.com/v1/act \
  -H "Authorization: Bearer $SEQUENCES_API_KEY" \
  -d '{"model":"accounts/sequences/models/pi05-droid","observation":{...}}'

Keeping that door open is deliberate — a vendor SDK that is the only supported way in locks you to one language, and robot controllers are very often C++. But it is a door, not the front entrance. The four items in the table at the top are work your client then has to do itself, and the reference documents each precisely enough to reimplement. Reaching for Python first is simply cheaper.