Som (Set-of-Mark) is a visual grounding component for the Computer-Use Agent (Cua) framework powering Cua, for detecting and analyzing UI elements in screenshots. Optimized for macOS Silicon with Metal Performance Shaders (MPS), it combines YOLO-based icon detection with EasyOCR text recognition to provide comprehensive UI element analysis.
Features
- Optimized for Apple Silicon with MPS acceleration
- Icon detection using YOLO with multi-scale processing
- Text recognition using EasyOCR (GPU-accelerated)
- Automatic hardware detection (MPS → CUDA → CPU)
- Smart detection parameters tuned for UI elements
- Detailed visualization with numbered annotations
- Performance benchmarking tools
System Requirements
- Recommended: macOS with Apple Silicon
- Uses Metal Performance Shaders (MPS)
- Multi-scale detection enabled
- ~0.4s average detection time
- Supported: Any Python 3.11+ environment
- Falls back to CPU if no GPU available
- Single-scale detection on CPU
- ~1.3s average detection time
Installation
# Using PDM (recommended)
pdm install
# Using pip
pip install -e .
Quick Start
from som import OmniParser
from PIL import Image
# Initialize parser
parser = OmniParser()
# Process an image
image = Image.open("screenshot.png")
result = parser.parse(
image,
box_threshold=0.3, # Confidence threshold
iou_threshold=0.1, # Overlap threshold
use_ocr=True # Enable text detection
)
# Access results
for elem in result.elements:
if elem.type == "icon":
print(f"Icon: confidence={elem.confidence:.3f}, bbox={elem.bbox.coordinates}")
else: # text
print(f"Text: '{elem.content}', confidence={elem.confidence:.3f}")
License
cua-som is licensed under AGPL-3.0-or-later. See LICENSE for the
complete terms. Its dependencies retain their own licenses. Inspect the exact
resolved Ultralytics dependency version and its notices before redistribution.
By default, OmniParser can download model files from Microsoft's separate
OmniParser-v2.0 model repository. Those downloaded files are not relicensed by
the cua-som package. Inspect the license and provenance published with the
exact model revision before use or redistribution.
Metadata
Release files for cua-som 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| cua_som-0.2.0.tar.gz | 30.8 kB | Without provenance | Details for cua_som-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| cua_som-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for cua_som-0.2.0-py3-none-any.whl |
Total release size: 63.0 kB
Release files / cua_som-0.2.0.tar.gz
| Download URL | cua_som-0.2.0.tar.gz |
|---|---|
| Size | 30.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b075d377f410686e04de13814b1f227b479b59cfc9ea39df920a8820d334ea02
|
|
BLAKE2b-256 checksum How to use checksums |
c8873a1b9b9a0b664075aa0a607292f4160db4f429d6ba8d0bbbc2427f517f26
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.9
|
Release files / cua_som-0.2.0-py3-none-any.whl
| Download URL | cua_som-0.2.0-py3-none-any.whl |
|---|---|
| Size | 32.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
456174931bce2888a939225742b8edfad27387c5ae5bc33aefaaa0547864c329
|
|
BLAKE2b-256 checksum How to use checksums |
996687c1066061b259619b490ac95ad1f47654c032c16dc82b646a9dac9bab8d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.9
|
Security for cua-som 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.