Skip to main content

Shows my svg

Python macOS Discord PyPI

Som (Set-of-Mark) is a visual grounding component for the Computer-Use Agent (Cua) framework powering Cua, for detecting and analyzing UI elements in screenshots. Optimized for macOS Silicon with Metal Performance Shaders (MPS), it combines YOLO-based icon detection with EasyOCR text recognition to provide comprehensive UI element analysis.

Features

  • Optimized for Apple Silicon with MPS acceleration
  • Icon detection using YOLO with multi-scale processing
  • Text recognition using EasyOCR (GPU-accelerated)
  • Automatic hardware detection (MPS → CUDA → CPU)
  • Smart detection parameters tuned for UI elements
  • Detailed visualization with numbered annotations
  • Performance benchmarking tools

System Requirements

  • Recommended: macOS with Apple Silicon
    • Uses Metal Performance Shaders (MPS)
    • Multi-scale detection enabled
    • ~0.4s average detection time
  • Supported: Any Python 3.11+ environment
    • Falls back to CPU if no GPU available
    • Single-scale detection on CPU
    • ~1.3s average detection time

Installation

# Using PDM (recommended)
pdm install

# Using pip
pip install -e .

Quick Start

from som import OmniParser
from PIL import Image

# Initialize parser
parser = OmniParser()

# Process an image
image = Image.open("screenshot.png")
result = parser.parse(
    image,
    box_threshold=0.3,    # Confidence threshold
    iou_threshold=0.1,    # Overlap threshold
    use_ocr=True         # Enable text detection
)

# Access results
for elem in result.elements:
    if elem.type == "icon":
        print(f"Icon: confidence={elem.confidence:.3f}, bbox={elem.bbox.coordinates}")
    else:  # text
        print(f"Text: '{elem.content}', confidence={elem.confidence:.3f}")

License

cua-som is licensed under AGPL-3.0-or-later. See LICENSE for the complete terms. Its dependencies retain their own licenses. Inspect the exact resolved Ultralytics dependency version and its notices before redistribution.

By default, OmniParser can download model files from Microsoft's separate OmniParser-v2.0 model repository. Those downloaded files are not relicensed by the cua-som package. Inspect the license and provenance published with the exact model revision before use or redistribution.

Metadata

Release files for cua-som 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cua-som 0.2.0
File Size Uploaded Provenance Actions
cua_som-0.2.0.tar.gz 30.8 kB Without provenance Details for cua_som-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cua-som 0.2.0
File Interpreter ABI Platform Actions
cua_som-0.2.0-py3-none-any.whl Python 3 none any Details for cua_som-0.2.0-py3-none-any.whl

Total release size: 63.0 kB

Release files / cua_som-0.2.0.tar.gz

Download URL cua_som-0.2.0.tar.gz
Size 30.8 kB
Tags Source
SHA-256 checksum
How to use checksums
b075d377f410686e04de13814b1f227b479b59cfc9ea39df920a8820d334ea02
BLAKE2b-256 checksum
How to use checksums
c8873a1b9b9a0b664075aa0a607292f4160db4f429d6ba8d0bbbc2427f517f26
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Release files / cua_som-0.2.0-py3-none-any.whl

Download URL cua_som-0.2.0-py3-none-any.whl
Size 32.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
456174931bce2888a939225742b8edfad27387c5ae5bc33aefaaa0547864c329
BLAKE2b-256 checksum
How to use checksums
996687c1066061b259619b490ac95ad1f47654c032c16dc82b646a9dac9bab8d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Security for cua-som 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page