The repository or configuration that created this release differ from a previous release. View details.
openadapt-types
Pydantic models for computer-use agents: what is on the screen, what to do to it, and what happened when you did. Use them when you build an agent, a screen recorder, or a benchmark harness and want one validated format for observations and actions, plus JSON Schema for code that isn't in Python.
The release on PyPI (0.22.0) needs Python 3.10 or later, and its only
dependency is Pydantic 2. On main, the models live in Flow as
openadapt_flow.contracts, and this package keeps import openadapt_types
working on top of it. Releases built from main depend on openadapt-flow
and need Python 3.10 to 3.12.
Quick start
pip install openadapt-types
from openadapt_types import (
Action, ActionTarget, ActionType, BoundingBox, ComputerState, UINode,
)
# What is on the screen
state = ComputerState(
viewport=(1920, 1080),
nodes=[
UINode(node_id="n0", role="window", name="My App", children_ids=["n1"]),
UINode(node_id="n1", role="button", name="Submit", parent_id="n0",
bbox=BoundingBox(x=500, y=400, width=100, height=40)),
],
)
print(state.to_text_tree())
# [n0] window: My App
# [n1] button: Submit
# What to do to it
action = Action(type=ActionType.CLICK, target=ActionTarget(node_id="n1"))
print(action.model_dump_json(exclude_none=True))
# {"type":"click","target":{"node_id":"n1","is_normalized":false}}
to_text_tree() returns the element tree as text you can put straight into a
prompt. Every model checks its fields when you create it, round-trips through
JSON, and gives you a JSON Schema from model_json_schema().
OpenAdapt's benchmark harness
(openadapt-evals) and
training code (openadapt-ml)
use these types. If you want to show a task and have it run again on its
own, install OpenAdapt
(pip install openadapt) instead.
What's in it
| Type | What it holds |
|---|---|
ComputerState |
One observation: screenshot, the full UI element tree, windows, URL, and platform |
UINode |
One element: role, name, value, bounding box, parent and children, and platform anchors such as a Windows AutomationId, an XPath, or a CSS selector |
Action, ActionTarget |
What to do (click, type, press a key, scroll, drag, open an app, and more) and where |
ActionResult |
Whether the action ran, an error category, and which elements changed |
Episode, Step |
A whole trajectory of observations, actions, results, and rewards, saved to and loaded from JSON |
FailureRecord |
A failure sorted into a category such as grounding, navigation, or execution, for dataset pipelines |
PlainOutcomeV1 |
An OpenAdapt run's result in plain words, whether anything may have changed, and whether a retry is safe |
The same screen model covers web, Windows, macOS, and Linux. It keeps the
screenshot and the element tree together, because each holds information the
other lacks. raw, attributes, and metadata fields let you carry your own
data through without forking the schema.
Targeting
ActionTarget accepts three kinds of answer to "which element". Prefer them
in this order:
from openadapt_types import ActionTarget
ActionTarget(node_id="n1") # an element in the tree
ActionTarget(description="the blue submit button") # resolved later by a grounding model
ActionTarget(x=550, y=420) # pixel coordinates, last resort
ActionTarget(x=0.29, y=0.39, is_normalized=True) # coordinates as a fraction of the screen
Have your agent produce a node_id or a description and work out the pixels
when the action runs. Coordinates break when a window moves.
Read an action from a model's reply
from openadapt_types import parse_action
action = parse_action("Thought: the form is complete.\nAction: CLICK(x=0.29, y=0.39)")
print(action.type.value, action.target.x, action.target.y) # click 0.29 0.39
print(action.reasoning) # the form is complete.
print(parse_action("I'm not sure what to do").type.value) # fail
parse_action() reads the NAME(arg=value) form shown here and JSON,
including JSON inside a code fence. When x and y are both between 0 and
1, it reads them as fractions of the screen; otherwise it reads them as
pixels. A reply it can't read becomes a fail action with the reason in
reasoning, and the parser logs a warning. A harness can't mistake an
unreadable reply for a finished task.
Report how a run ended
plain_outcome() turns the status values from an OpenAdapt run into one
result a person can act on:
from openadapt_types import plain_outcome
result = plain_outcome(
transaction_outcome="RECONCILIATION_REQUIRED",
execution_outcome="HALTED",
)
print(result.label) # Check the record
print(result.next_action) # A save may have gone through. Check the record before anyone runs this again.
print(result.safe_to_retry) # False
An unknown or contradictory status reads "Check the record", never "nothing
was written". The table that maps status values to results also ships as
JSON for other languages, and the openadapt-vocab-lint command checks
user-facing text against it. See
docs/VOCABULARY.md.
JSON Schema for other languages
import json
from openadapt_types import ComputerState
print(json.dumps(ComputerState.model_json_schema(), indent=2))
The messages that OpenAdapt components exchange ship under
openadapt_types/schemas/: forty-four files, including execute-v1-openapi.json,
the OpenAPI document for OpenAdapt Execute, the API that starts runs and
reports their results. Two files hold test vectors, and the rest are JSON
Schema files.
docs/CONTRACTS.md
lists each contract, its Python model, and what it may carry.
Convert from older formats
from openadapt_types import from_benchmark_action
action = from_benchmark_action({"type": "click", "x": 0.5, "y": 0.3})
print(action.type.value, action.target.is_normalized) # click True
print(from_benchmark_action({"type": "teleport"}).type.value) # fail
from_benchmark_action() reads an openadapt-evals action dict. The other
converters are in openadapt_types._compat: from_benchmark_observation()
for openadapt-evals observations, from_ml_* for openadapt-ml, and
from_omnimcp_* for omnimcp. Each converter takes a plain dict, such as
obs.__dict__. An action type it doesn't recognize becomes a fail action,
never done.
Limits
- The API can still change between minor versions before 1.0. Pin the minor version you test against.
ComputerState.screenshot_pngholds raw bytes, which don't convert to JSON:model_dump_json()andEpisode.to_json()raise an error. To save a state or an episode as JSON, put the screenshot inscreenshot_pathorscreenshot_base64instead.- In the
NAME(arg=value)form,parse_action()reads onlyxandyas a target.CLICK(node_id="n1")returns a click with no target. Use the JSON form fornode_idanddescriptiontargets. - Some rules compare one field with another, such as "a result that says
nothing changed can't also say something may have changed". The Python
models enforce them. The JSON Schema, from the files or from
model_json_schema(), doesn't always, so code in another language has to check them itself.
License
Metadata
Release files for openadapt-types 0.23.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Change of provenance
The repository or configuration that created these release files differ from that of a previous release.
We recommend inspecting the repository and configuration before proceeding. View details.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| openadapt_types-0.23.0.tar.gz | 152.5 kB | With provenance | Details for openadapt_types-0.23.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| openadapt_types-0.23.0-py3-none-any.whl | Python 3 | none | any | Details for openadapt_types-0.23.0-py3-none-any.whl |
Total release size: 159.4 kB
Release files / openadapt_types-0.23.0.tar.gz
Change of provenance
This file belongs to a release with changed provenance: the repository or configuration that created the release files differ from that of a previous release.
We recommend inspecting the repository and configuration before proceeding. View details.
| Download URL | openadapt_types-0.23.0.tar.gz |
|---|---|
| Size | 152.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f97df3259522500debe140346d32ae1ccdb9426b83ae05c4d9e22fdd155f9aa7
|
|
BLAKE2b-256 checksum How to use checksums |
37806fab94fda2d27cf2f2ca15b4a5752e02bf6a5199e5a20da8945c322de07f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.
Transparency logRelease files / openadapt_types-0.23.0-py3-none-any.whl
Change of provenance
This file belongs to a release with changed provenance: the repository or configuration that created the release files differ from that of a previous release.
We recommend inspecting the repository and configuration before proceeding. View details.
| Download URL | openadapt_types-0.23.0-py3-none-any.whl |
|---|---|
| Size | 6.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
67b398c7adf161ae4ba2acd7c393767a06b238ce5fd490a1e4943e591f7b2f21
|
|
BLAKE2b-256 checksum How to use checksums |
e288e8be7a7118a9a2f51e7d333c0400b4bbec317b485b2810f07e6648486bca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.
Transparency logSecurity for openadapt-types 0.23.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Change of provenance
The repository and configuration that created this release differ from a previous release.
While this could reflect a routine change by the project maintainers, it may also indicate that the release files have been compromised.
Repository
- Removed 0.22.0 github.com/OpenAdaptAI/openadapt-types
- Added 0.23.0 github.com/OpenAdaptAI/OpenAdapt
Attestation for openadapt_types-0.23.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.