locate-anything 0.1.0

Simple, one-call Python interface for the nvidia/LocateAnything-3B UI element detection model.

locate-anything

A tiny Python package that wraps the full nvidia/LocateAnything-3B workflow — dependency setup, model loading, preprocessing, inference, and output parsing — behind a single class, so you don't have to re-copy notebook cells every time.

Install

Because torch needs a CUDA-specific build, install it first from the PyTorch index, then install this package:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
pip install locate-anything

(Or, from a local checkout: pip install . after the torch step above.)

LocateAnything-3B is a gated model, so you'll also need to authenticate once:

huggingface-cli login

or pass a token directly when creating the client (see below).

Usage

Python API

from locate_anything import LocateAnything

# Loads tokenizer, processor, and model once (put this outside any loop)
la = LocateAnything()  # optionally: LocateAnything(hf_token="hf_...")

result = la.detect("screenshot.png", save_path="detected.png")

print(result["total"])         # number of detections
print(result["counts"])        # {"button": 12, "icon": 30, ...}
print(result["detections"])    # [{"label": "button", "bbox_pixels": [x1,y1,x2,y2]}, ...]
result["annotated_image"].show()  # PIL.Image with boxes drawn

Custom categories:

result = la.detect(
    "screenshot.png",
    categories=["play button", "volume slider", "progress bar"],
)

Batch of images:

results = la.detect_batch(["a.png", "b.png", "c.png"])

Command line

locate-anything screenshot.png -o detected.png --json-output detections.json

What it handles for you

• Dependency management — pinned versions declared in pyproject.toml (torch must be installed separately due to the CUDA index URL requirement).
• Model initialization — tokenizer, processor, and model loaded once per LocateAnything instance, in bfloat16 on the best available device.
• Preprocessing — builds the chat-template prompt and processes image/video inputs for you.
• Inference — calls model.generate(...) with sane defaults (generation_mode="hybrid", use_cache=True).
• Output processing — parses the model's <box> tags back into pixel coordinates, tallies counts per label, and optionally draws + saves an annotated image.

Package layout

locate_anything/
├── __init__.py        # public API: LocateAnything, DEFAULT_CATEGORIES
├── config.py           # default model name, categories, regex pattern
├── core.py              # LocateAnything class (load + detect)
├── postprocessing.py    # parse_detections(), draw_detections()
└── cli.py                # `locate-anything` command line entry point