openadapt-types

Canonical Pydantic schemas for computer-use agents.

pip install openadapt-types

What's in the box

Schema Purpose
ComputerState Screen state: screenshot + UI element graph + window context
UINode Single UI element with role, bbox, hierarchy, platform anchors
Action Agent action with typed action space + flexible targeting
ActionTarget Where to act: node_id > description > (x, y) coordinates
ActionResult Execution outcome with error taxonomy + state delta
Episode / Step Complete task trajectory (observation → action → result)
FailureRecord Classified failure for dataset pipelines

Quick start

from openadapt_types import (
    Action, ActionTarget, ActionType,
    ComputerState, UINode, BoundingBox,
)

# Describe what's on screen
state = ComputerState(
    viewport=(1920, 1080),
    nodes=[
        UINode(node_id="n0", role="window", name="My App", children_ids=["n1"]),
        UINode(node_id="n1", role="button", name="Submit", parent_id="n0",
               bbox=BoundingBox(x=500, y=400, width=100, height=40)),
    ],
)

# Agent decides what to do
action = Action(
    type=ActionType.CLICK,
    target=ActionTarget(node_id="n1"),
    reasoning="Click Submit to proceed",
)

# Render element tree for LLM prompts
print(state.to_text_tree())
# [n0] window: My App
#   [n1] button: Submit

Action targeting

ActionTarget supports three grounding strategies (in priority order):

# 1. Element-based (preferred — most robust)
ActionTarget(node_id="n1")

# 2. Description-based (resolved by grounding module)
ActionTarget(description="the blue submit button")

# 3. Coordinate-based (fallback)
ActionTarget(x=550, y=420)
ActionTarget(x=0.29, y=0.39, is_normalized=True)

Agents SHOULD produce node_id or description. The runtime resolves to coordinates.

Compatibility with existing schemas

Converters for three existing OpenAdapt schema formats:

from openadapt_types._compat import (
    from_benchmark_observation,   # openadapt-evals BenchmarkObservation
    from_benchmark_action,        # openadapt-evals BenchmarkAction
    from_ml_observation,          # openadapt-ml Observation
    from_ml_action,               # openadapt-ml Action
    from_omnimcp_screen_state,    # omnimcp ScreenState
    from_omnimcp_action_decision, # omnimcp ActionDecision
)

# Convert existing data
state = from_benchmark_observation(obs.__dict__)
action = from_benchmark_action(act.__dict__)

JSON Schema

Export for language-agnostic tooling:

import json
from openadapt_types import ComputerState, Action, Episode

# Get JSON Schema
schema = ComputerState.model_json_schema()
print(json.dumps(schema, indent=2))

Design principles

  • Pydantic v2 — runtime validation, JSON Schema export, fast serialization
  • Pixels + structure — always capture both visual and semantic UI state
  • Node graph — full element tree, not just focused element
  • Platform-agnostic — same schema for Windows, macOS, Linux, web
  • Extension-friendly — raw, attributes, metadata fields everywhere
  • Backward compatible — _compat converters for gradual migration

Dependencies

Just pydantic>=2.0. No ML libraries, no heavy deps.

License

MIT

Project links

Data verified by PyPI on Jul 13, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.10.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for abrichr from gravatar.com abrichr

Credits

Author: OpenAdapt

GitHub Statistics

Data verified by PyPI on Jul 13, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Provides Extra

dev

Tags

agents computer-use gui-automation pydantic schemas

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

openadapt_types-0.3.0.tar.gz (55.4 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

openadapt_types-0.3.0-py3-none-any.whl (22.9 kB view details)

Uploaded Python 3

File details

Details for the file openadapt_types-0.3.0.tar.gz.

File metadata

  • Download URL: openadapt_types-0.3.0.tar.gz
  • Upload date:
  • Size: 55.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for openadapt_types-0.3.0.tar.gz
Algorithm Hash digest
SHA256 7a6760a957a9266b91a09771ed84378d7673e41c0be79cd1f6d026c05ec4e51b
MD5 86ffda973c4fa420cb01cbf22142124e
BLAKE2b-256 2e0481afdf8ffa820cbf99c4d1d0b733ad13eb5072bc7bb5f4ae768a0f8e85e4

See more details on using hashes here.

Provenance

The following attestation bundles were made for openadapt_types-0.3.0.tar.gz:

Publisher: publish.yml on OpenAdaptAI/openadapt-types

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file openadapt_types-0.3.0-py3-none-any.whl.

File metadata

  • Download URL: openadapt_types-0.3.0-py3-none-any.whl
  • Upload date:
  • Size: 22.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for openadapt_types-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 72d7c2a301121c2cdbcb57095b2b5193e0780741a01d7571b3bed021b214bfe7
MD5 edcf9515896b6979d020da2a3d0d44ee
BLAKE2b-256 5163eb7d4b025d4796e7c71da7f155936f0174ecf44400408740b707cb6862c7

See more details on using hashes here.

Provenance

The following attestation bundles were made for openadapt_types-0.3.0-py3-none-any.whl:

Publisher: publish.yml on OpenAdaptAI/openadapt-types

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page