Skip to main content

openadapt-types

Packages CI PyPI Python License: MIT

Pydantic models for computer-use agents: what is on the screen, what to do to it, and what happened when you did. Use them when you build an agent, a screen recorder, or a benchmark harness and want one validated format for observations and actions, plus JSON Schema for code that isn't in Python.

It needs Python 3.10 or later, and its only dependency is Pydantic 2. openadapt-flow depends on this package and re-exports it as openadapt_flow.contracts, so the same classes are reachable under both names.

Quick start

pip install openadapt-types
from openadapt_types import (
    Action, ActionTarget, ActionType, BoundingBox, ComputerState, UINode,
)

# What is on the screen
state = ComputerState(
    viewport=(1920, 1080),
    nodes=[
        UINode(node_id="n0", role="window", name="My App", children_ids=["n1"]),
        UINode(node_id="n1", role="button", name="Submit", parent_id="n0",
               bbox=BoundingBox(x=500, y=400, width=100, height=40)),
    ],
)
print(state.to_text_tree())
# [n0] window: My App
#   [n1] button: Submit

# What to do to it
action = Action(type=ActionType.CLICK, target=ActionTarget(node_id="n1"))
print(action.model_dump_json(exclude_none=True))
# {"type":"click","target":{"node_id":"n1","is_normalized":false}}

to_text_tree() returns the element tree as text you can put straight into a prompt. Every model checks its fields when you create it, round-trips through JSON, and gives you a JSON Schema from model_json_schema().

OpenAdapt's benchmark harness (openadapt-evals) and training code (openadapt-ml) use these types. If you want to show a task and have it run again on its own, install OpenAdapt (pip install openadapt) instead.

What's in it

Type What it holds
ComputerState One observation: screenshot, the full UI element tree, windows, URL, and platform
UINode One element: role, name, value, bounding box, parent and children, and platform anchors such as a Windows AutomationId, an XPath, or a CSS selector
Action, ActionTarget What to do (click, type, press a key, scroll, drag, open an app, and more) and where
ActionResult Whether the action ran, an error category, and which elements changed
Episode, Step A whole trajectory of observations, actions, results, and rewards, saved to and loaded from JSON
FailureRecord A failure sorted into a category such as grounding, navigation, or execution, for dataset pipelines
PlainOutcomeV1 An OpenAdapt run's result in plain words, whether anything may have changed, and whether a retry is safe

The same screen model covers web, Windows, macOS, and Linux. It keeps the screenshot and the element tree together, because each holds information the other lacks. raw, attributes, and metadata fields let you carry your own data through without forking the schema.

Targeting

ActionTarget accepts three kinds of answer to "which element". Prefer them in this order:

from openadapt_types import ActionTarget

ActionTarget(node_id="n1")                          # an element in the tree
ActionTarget(description="the blue submit button")  # resolved later by a grounding model
ActionTarget(x=550, y=420)                          # pixel coordinates, last resort
ActionTarget(x=0.29, y=0.39, is_normalized=True)    # coordinates as a fraction of the screen

Have your agent produce a node_id or a description and work out the pixels when the action runs. Coordinates break when a window moves.

Read an action from a model's reply

from openadapt_types import parse_action

action = parse_action("Thought: the form is complete.\nAction: CLICK(x=0.29, y=0.39)")
print(action.type.value, action.target.x, action.target.y)  # click 0.29 0.39
print(action.reasoning)                                     # the form is complete.

print(parse_action("I'm not sure what to do").type.value)  # fail

parse_action() reads the NAME(arg=value) form shown here and JSON, including JSON inside a code fence. When x and y are both between 0 and 1, it reads them as fractions of the screen; otherwise it reads them as pixels. A reply it can't read becomes a fail action with the reason in reasoning, and the parser logs a warning. A harness can't mistake an unreadable reply for a finished task.

Report how a run ended

plain_outcome() turns the status values from an OpenAdapt run into one result a person can act on:

from openadapt_types import plain_outcome

result = plain_outcome(
    transaction_outcome="RECONCILIATION_REQUIRED",
    execution_outcome="HALTED",
)
print(result.label)          # Check the record
print(result.next_action)    # A save may have gone through. Check the record before anyone runs this again.
print(result.safe_to_retry)  # False

An unknown or contradictory status reads "Check the record", never "nothing was written". The table that maps status values to results also ships as JSON for other languages, and the openadapt-vocab-lint command checks user-facing text against it. See docs/VOCABULARY.md.

JSON Schema for other languages

import json
from openadapt_types import ComputerState

print(json.dumps(ComputerState.model_json_schema(), indent=2))

The messages that OpenAdapt components exchange ship under openadapt_types/schemas/: forty-four files, including execute-v1-openapi.json, the OpenAPI document for OpenAdapt Execute, the API that starts runs and reports their results. Two files hold test vectors, and the rest are JSON Schema files. docs/CONTRACTS.md lists each contract, its Python model, and what it may carry.

Convert from older formats

from openadapt_types import from_benchmark_action

action = from_benchmark_action({"type": "click", "x": 0.5, "y": 0.3})
print(action.type.value, action.target.is_normalized)          # click True
print(from_benchmark_action({"type": "teleport"}).type.value)  # fail

from_benchmark_action() reads an openadapt-evals action dict. The other converters are in openadapt_types._compat: from_benchmark_observation() for openadapt-evals observations, from_ml_* for openadapt-ml, and from_omnimcp_* for omnimcp. Each converter takes a plain dict, such as obs.__dict__. An action type it doesn't recognize becomes a fail action, never done.

Limits

  • The API can still change between minor versions before 1.0. Pin the minor version you test against.
  • ComputerState.screenshot_png holds raw bytes, which don't convert to JSON: model_dump_json() and Episode.to_json() raise an error. To save a state or an episode as JSON, put the screenshot in screenshot_path or screenshot_base64 instead.
  • In the NAME(arg=value) form, parse_action() reads only x and y as a target. CLICK(node_id="n1") returns a click with no target. Use the JSON form for node_id and description targets.
  • Some rules compare one field with another, such as "a result that says nothing changed can't also say something may have changed". The Python models enforce them. The JSON Schema, from the files or from model_json_schema(), doesn't always, so code in another language has to check them itself.

License

MIT

Metadata

Release files for openadapt-types 0.24.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for openadapt-types 0.24.0
File Size Uploaded Provenance Actions
openadapt_types-0.24.0.tar.gz 318.2 kB With provenance Details for openadapt_types-0.24.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for openadapt-types 0.24.0
File Interpreter ABI Platform Actions
openadapt_types-0.24.0-py3-none-any.whl Python 3 none any Details for openadapt_types-0.24.0-py3-none-any.whl

Total release size: 542.9 kB

Release files / openadapt_types-0.24.0.tar.gz

Download URL openadapt_types-0.24.0.tar.gz
Size 318.2 kB
Tags Source
SHA-256 checksum
How to use checksums
16dcd1da1c2c78c08d650a19eb9cc439474f996812a87198b316d51bfac443db
BLAKE2b-256 checksum
How to use checksums
c5fb9ba7017c2dd42132425a2b15ebcbb79fc7920d8a98961da9bb93d3a9f79b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.

Transparency log

Release files / openadapt_types-0.24.0-py3-none-any.whl

Download URL openadapt_types-0.24.0-py3-none-any.whl
Size 224.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8025e8a4dc9235fd81d4b24ae7fa7e544120abdb8d4ee7df160e4f50d9230d75
BLAKE2b-256 checksum
How to use checksums
c8382c7c3a7ebac23cc2e79039c4eca66aae5c3fd1c11a1076cb5cdc189771d6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.

Transparency log

Security for openadapt-types 0.24.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for openadapt_types-0.24.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.24.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page