ui-locator

Locate precise click coordinates of UI elements in screenshots via natural-language queries, for use with computer-use agents.

"Where is the 'Invia' button?" → (742, 511) (absolute pixels) + bounding box.

It wraps GUI grounding models — primarily ByteDance UI-TARS-1.5-7B (Apache-2.0) running locally via MLX on Apple Silicon, or any OpenAI-compatible cloud endpoint (a local mlx-vlm server, vLLM, a HuggingFace Inference Endpoint, or the Volcengine/Doubao Ark API for the proprietary UI-TARS-2).

Quick start

The package is published as ui-locator-cli; the command it installs is ui-locator.

# Run it without installing anything
uvx --from "ui-locator-cli[cloud,uitars]" ui-locator --help

# Or install the command on your PATH
uv tool install "ui-locator-cli[cloud,uitars]"   # cloud-only, any OS
uv tool install "ui-locator-cli[mlx,uitars]"     # local inference, Apple Silicon

# As a library in a project
uv add "ui-locator-cli[cloud,uitars]"

From a clone, for development:

uv pip install -e ".[mlx,uitars]"    # Apple Silicon
uv pip install -e ".[cloud,uitars]"  # cloud-only, any OS

For a cloud endpoint (Qwen, UI-TARS, ...), set the project up once:

ui-locator init                                   # writes .ui-locator/config.toml
echo 'UI_LOCATOR_OPENAI_API_KEY=<key>' >> .ui-locator/.env
ui-locator config show                            # check what resolved

from ui_locator import Locator

loc = Locator()  # auto-selects MLX on Apple Silicon
res = loc.locate("screen.png", "where is the 'Invia' button?")

print(res.click_point)   # Coordinates(x=742, y=511)
print(res.bbox)          # BoundingBox(...) or None
print(res.best.confidence)
print(res.best.thought)  # model reasoning (agent mode)

Intent-style queries work too — the model resolves which element to click:

loc.locate("screen.png", "chiudi il terminale")      # finds the close button
loc.locate("screen.png", "minimizza il browser")     # finds the minimise button
loc.locate("screen.png", "clicca sul tab Google")    # finds the browser tab

Command line

ui-locator screen.png "where is the 'Invia' button?"
ui-locator screen.png "close the terminal" --think          # ambiguous/intent query
ui-locator screen.png "Submit" --annotate                   # saves screen_test.png
ui-locator screen.png "Submit" out.png --annotate           # custom output path
ui-locator screen.png "Submit" --json                       # machine-readable

Option	Meaning
--id <dotted.id>	Element id — reuse a known position instead of calling the model (below)
--session <id>	Tag learned entries so one run can be dropped later
--no-cache	Re-ask the model and overwrite the stored entry
--verify	Re-check a crop around the stored point before trusting it
--fill id=query ..., --prefix	Populate several ids in one model load
--map-dir <path>	Element-map directory (default: the project's, see below)
--use <profile>	Model profile from [models.<name>] in the project config
--provider {auto,mlx,qwen,qwen-local,openai,anthropic}	Force a backend
--model <hf-id>	Local MLX model id (default mlx-community/UI-TARS-1.5-7B-4bit)
--think	Enable UI-TARS reasoning (agent mode)
--base-url, --api-key, --openai-model, --openai-prompt {uitars,qwen}	Cloud endpoint settings
--annotate PATH	Save an annotated copy of the image
--json	Emit JSON

Configuration (.ui-locator/)

A project is any directory containing .ui-locator/, found by walking up from the current directory the way git finds .git. That one root holds both the configuration and the element map, so the coordinates you have already established travel with the automation that uses them:

progetto/
├── .ui-locator/
│   ├── config.toml          # committed: providers, endpoints, models
│   ├── .env                 # gitignored: API keys only
│   └── map/
│       └── sap.screen.json  # committed: known coordinates
└── automazioni/sap/         # any depth: the same root is found

ui-locator init            # create .ui-locator/ here, with a config template
ui-locator config show     # the resolved settings, and where each came from
ui-locator config path     # the project root, if any

Model profiles

Adding a backend is a TOML block, not a code change:

default_model = "qwen-cloud"      # used when --use is not given

[models.qwen-cloud]
provider = "openai"
openai_base_url = "https://api.fireworks.ai/inference/v1"
openai_model = "accounts/fireworks/models/qwen3p7-plus"
openai_prompt = "qwen"

[models.uitars-local]
provider = "mlx"
model = "mlx-community/UI-TARS-1.5-7B-4bit"

ui-locator screen.png "Invia" --use uitars-local    # or UI_LOCATOR_USE=...

Any Settings field works as a top-level key (shared baseline) or inside a profile (specialisation). Unknown keys are reported on stderr — including those in profiles you have not selected, so a typo surfaces before the day you switch to it.

Precedence

Highest first:

1. CLI flags
2. environment variables (UI_LOCATOR_*, and the plain OPENAI_* the cloud provider also honours)
3. .env — ./.env, then .ui-locator/.env
4. the selected [models.<name>] block
5. top-level keys of .ui-locator/config.toml
6. the same of $XDG_CONFIG_HOME/ui-locator/config.toml
7. field defaults

ui-locator config show prints the winner of each field and the layer it came from (model:qwen-cloud, env:OPENAI_MODEL, config:/path, default), naming the exact environment variable — which is the only quick answer to "why is it calling the wrong endpoint". API keys are shown as ***.

Keep secrets out of config.toml so it can be committed: ui-locator init writes a .gitignore for .env alone — config.toml and map/ are meant to be committed. Positions the model guessed during a run land in the same files as the ones you declared, so drop them with ui-locator map forget --session <id> before committing if you only want to keep what you have verified.

Element map (--id)

Grounding a screenshot costs seconds. Most agent loops ask the same question about the same screen over and over, so --id lets the caller name an element once and reuse its position:

# first time: the model runs, and the position is stored under that id
ui-locator screen.png --id sap.screen.submit "il bottone Invia"    # ~3 s

# afterwards: answered from the map — no model, no image decode
ui-locator screen.png --id sap.screen.submit "il bottone Invia"    # ~0.13 s

The id is an assertion by the caller, not a guess from the image: passing --id sap.screen.submit states that this element sits at a stable position on that screen, whatever data the screen happens to be showing. That is exactly the case a screenshot fingerprint gets wrong — a data-heavy screen looks different on every visit while its buttons never move — and it is why a hit needs no image analysis at all: only the PNG header is read, to get (width, height).

On a hit the query is ignored, so rephrasing it ("Invia" / "clicca su Invia") still hits. It is only used on a miss, and can be omitted entirely once an entry exists.

What takes part in the lookup

The map lives in the project (.ui-locator/map/) when there is one, otherwise in $XDG_DATA_HOME/ui-locator/map — a durable location, not a cache directory.

	In the key?	Why
element id	yes	the caller's assertion
screenshot size	yes	a resized window genuinely moves elements
backend / model	no	recorded as provenance only: which model found the button does not change where it is, so switching provider does not cold-start the map
the screenshot itself	no	the whole point of --id

The map is a file you can edit

One small JSON file per id prefix (last segment = element, the rest = group):

// .ui-locator/map/sap.screen.json
{
  "submit": [
    { "click_point": [742, 511], "bbox": [700, 495, 790, 530],
      "size": [2560, 1440], "source": "model", "hits": 12 }
  ]
}

So when you already know a coordinate, declare it and the model is never involved. Declared entries win over learned ones and are never overwritten; one with "size": null applies at any resolution.

ui-locator map set --id sap.screen.submit --point 742,511
ui-locator map set --id sap.screen.submit --point 742,511 --size 2560x1440

Managing it

ui-locator map show                          # everything
ui-locator map show --id sap.screen          # one screen (dotted prefix)
ui-locator map show --json                   # machine-readable
ui-locator map miss --id sap.screen.submit   # the click failed: drop what was learned
ui-locator map forget --id sap               # app updated: wipe the whole prefix
ui-locator map forget --session run-42       # drop just what one run learned
ui-locator map path

Populate a whole screen in one model load:

ui-locator screen.png --prefix sap.screen \
  --fill submit=Invia --fill cancel=Annulla --fill search="campo di ricerca"

Using it from an agent

The --json output says where the answer came from, so the caller can decide how much to trust it:

{ "click_point": {"x": 742, "y": 511}, "provider": "map",
  "map": {"element_id": "sap.screen.submit", "hit": true,
          "source": "declared", "hits": 12} }

Two rules make the loop reliable:

1. When a click does not have the expected effect, say so — ui-locator map miss --id <id> — or re-run with --no-cache, which re-asks the model and overwrites the entry. A stored position is only ever corrected by feedback: nothing else can tell that it went stale.
2. --session <run-id> tags what a run learns, so a whole run's guesses can be dropped without touching hand-declared entries.

The residual risk of --id is not that the button moved — you asserted it does not — but that the caller misidentified which screen it is on (a modal is up, or it is on a different screen than it thinks). --verify covers that: it compares a small stored crop around the point against the screenshot and falls back to the model on mismatch. It costs one image decode and is insensitive to data changing elsewhere on the screen.

Python

from ui_locator import Locator

loc = Locator()
res = loc.locate("screen.png", "il bottone Invia",
                 element_id="sap.screen.submit", session="run-42")
res.map.hit        # False on the first call, True afterwards

Settings also come from the environment: UI_LOCATOR_MAP_DIR, UI_LOCATOR_SESSION, UI_LOCATOR_USE_MAP=0, UI_LOCATOR_VERIFY=1.

Use it from an agent

A ready-made skill for a coding / computer-use agent ships with the package:

ui-locator skill install                    # ~/.claude/skills/ui-locator
ui-locator skill install --scope project    # the layout the project already uses
ui-locator skill install --scope agents     # ./.agents/skills/ui-locator
ui-locator skill install --scope claude     # ./.claude/skills/ui-locator
ui-locator skill install --dir path/skills  # anywhere
ui-locator skill install --link             # symlink, so edits apply immediately
ui-locator skill status                     # where it is, and whether it is current
ui-locator skill update                     # refresh after upgrading ui-locator
ui-locator skill remove

SKILL.md is the same file for any agent — only the directory is vendor-specific, so it is not hardcoded. --scope project installs into whichever of .agents/skills or .claude/skills the project already uses, and into the vendor-neutral .agents/ when neither exists yet.

The agent picks it up in a new session. The skill covers the --id workflow, when a stable id is a valid assertion and when it is not, and how to report a bad position back — which is what keeps the map honest.

install refuses to overwrite an existing install (use update, or --force), and neither install nor remove will touch a directory that is not this skill.

Providers

Provider	Engine	When to use
mlx	UI-TARS-1.5-7B via mlx-vlm	Default on Apple Silicon. On-device, private, free. Returns a click point.
qwen	Qwen2.5-VL, cloud or local	Bounding-box grounding (native <box> tokens). Uses the configured endpoint when there is one, the local MLX checkpoint otherwise, and always the Qwen prompt style.
qwen-local	Qwen2.5-VL via mlx-vlm	Force the on-device checkpoint even when an endpoint is configured.
openai	Any OpenAI-compatible endpoint	Cloud; the only way to use UI-TARS-2 (Doubao/Ark).
omniparser	Microsoft OmniParser v2	Planned — note its detector is AGPL-3.0.
cua	Anthropic Claude Computer Use	Planned fallback — easy but least precise.

Provider selection

Locator() auto-selects: MLX (UI-TARS) on Apple Silicon, otherwise an OpenAI-compatible endpoint if configured. Override with UI_LOCATOR_PROVIDER (or --provider), or name a profile with --use.

qwen means Qwen grounding wherever it is available: the configured endpoint if there is one, the local MLX checkpoint otherwise. Either way it forces openai_prompt = "qwen", because the default uitars style parses <point> and finds nothing in a <box> answer — a mismatch that produces no error, just no coordinates. When the two disagree anyway (say --provider openai with a Qwen model), ui-locator warns on stderr rather than letting it look like a grounding failure. Use qwen-local to pin the on-device checkpoint (--qwen-model to change it).

Cloud endpoints

OpenRouter UI-TARS-1.5:

export UI_LOCATOR_PROVIDER=openai
export UI_LOCATOR_OPENAI_BASE_URL=https://openrouter.ai/api/v1
export UI_LOCATOR_OPENAI_API_KEY=<your-openrouter-key>
export UI_LOCATOR_OPENAI_MODEL=bytedance/ui-tars-1.5-7b
export UI_LOCATOR_OPENAI_PROMPT=uitars

Fireworks Qwen bbox grounding:

export UI_LOCATOR_PROVIDER=openai
export UI_LOCATOR_OPENAI_BASE_URL=https://api.fireworks.ai/inference/v1
export UI_LOCATOR_OPENAI_API_KEY=<your-fireworks-key>
export UI_LOCATOR_OPENAI_MODEL=accounts/fireworks/models/qwen3p7-plus
export UI_LOCATOR_OPENAI_PROMPT=qwen

UI-TARS-2 via Volcengine/Doubao Ark:

export UI_LOCATOR_PROVIDER=openai
export UI_LOCATOR_OPENAI_BASE_URL=https://ark.cn-beijing.volces.com/api/v3
export UI_LOCATOR_OPENAI_API_KEY=<your-ark-key>
export UI_LOCATOR_OPENAI_MODEL=<doubao-ui-tars-endpoint-id>
export UI_LOCATOR_OPENAI_PROMPT=uitars

You can equally point OPENAI_BASE_URL at a local server (mlx_vlm.server --model mlx-community/UI-TARS-1.5-7B-4bit) or a vLLM instance.

Coordinates

UI-TARS-1.5/2 emit click points as absolute pixels in the "smart-resized" image space (Qwen2.5-VL preprocessing). ui-locator maps these back to the original screenshot resolution and always returns absolute pixels. The optional ui-tars package is used for parsing when available; a self-contained regex + smart_resize fallback is used otherwise.

• Element.click_point — where to click, absolute pixels (always present).
• Element.bbox — bounding box, absolute pixels (when available).
• Element.confidence — a heuristic [0, 1] signal (UI-TARS does not emit a score).

Important notes

• Recommended order: local UI-TARS-1.5 via MLX when available; OpenRouter UI-TARS-1.5 for hosted use; Qwen bbox models when you want region grounding; UI-TARS-2 via Volcengine Ark if you can access it.
• UI-TARS-2 is not open-weight. It is accessible only via ByteDance's Volcengine/Doubao Ark API. The local option is UI-TARS-1.5-7B (Apache-2.0), which shares UI-TARS-2's architecture and grounding format.
• MLX is Apple-Silicon only. On Intel Macs / Linux use the cloud provider.
• Memory: UI-TARS-1.5-7B at 4-bit uses ~4–5 GB (default; good for 8–16 GB Macs). Use --model mlx-community/UI-TARS-1.5-7B-6bit (~6–8 GB) or -8bit (~9–11 GB) for higher quality on machines with more memory.
• The ui-tars PyPI package is ByteDance's action parser (Apache-2.0), not the model itself.

Examples & benchmark

The examples/ folder contains runnable scripts:

uv run python examples/locate_button.py screen.png "where is the 'Invia' button?"
uv run python examples/locate_button.py screen.png "the Submit button" --provider qwen
uv run python examples/annotate_demo.py screen.png "the 'Invia' button" out.png  # draw bbox + click
uv run python examples/serve_mlx.py     screen.png "the 'Invia' button"          # local mlx-vlm server + query

The bench/ folder is a precision benchmark. It runs a provider over cases (screenshot + query + ground-truth bounding box) and reports hit-rate, click-to-centre distance and IoU — useful to validate accuracy on your own UIs.

uv run python bench/make_fixtures.py          # generate synthetic screenshots + GT
uv run python bench/benchmark.py              # benchmark the default provider
UI_LOCATOR_PROVIDER=qwen uv run python bench/benchmark.py   # compare with Qwen2.5-VL
uv run python bench/benchmark.py --models bench/fixtures/model_matrix.json   # multi-model benchmark
uv run python bench/compare_models.py ~/Desktop/screen.png "locate refresh button in active window"  # annotated per model

For multi-model runs, edit bench/fixtures/model_matrix.json. bench/benchmark.py reports scores per model; bench/compare_models.py saves one annotated image per model.

Add your own benchmark cases by editing bench/fixtures/benchmark.json ({"image": "...", "query": "...", "bbox": [x1,y1,x2,y2]}).

Development

uv sync --extra dev --extra mlx --extra uitars
uv run ruff check
uv run mypy
uv run pytest

# Live model test (Apple Silicon, downloads checkpoint on first run):
UI_LOCATOR_RUN_MODEL=1 uv run pytest -m integration

uv.lock is not committed, so CI resolves dependencies the way a user installing from PyPI does. The supported typer range is tested at both ends (0.16 and latest) because 0.27 stopped being click-based.

Releasing

The distribution is named ui-locator-cli on PyPI while [project.scripts] keeps the command itself ui-locator, so extras are addressed by the distribution name (ui-locator-cli[mlx]) and uvx needs --from.

.github/workflows/publish.yml uploads to PyPI when a GitHub release is published, gated on lint, types, the test matrix, and two release-only checks: the tag must match the version in pyproject.toml (PyPI never lets a version be reused), and SKILL.md must be present in the wheel.

# 1. bump version = "x.y.z" in pyproject.toml, commit
# 2. rehearse: builds and round-trips the artifacts, uploads nothing
gh workflow run publish.yml -f target=testpypi
# 3. release (the tag must be vx.y.z)
gh release create v0.1.0 --generate-notes

Authentication uses the PYPI_API_TOKEN secret. Trusted Publishing is preferable — GitHub mints short-lived OIDC credentials per run, so there is no long-lived secret to leak or rotate. Configure a publisher for this repo and publish.yml, then delete the password: line from the PyPI step.

License

Apache-2.0. The optional OmniParser detector model is AGPL-3.0 (evaluate separately if used).