✨ Features

• 🔀 Hybrid keywords – Click, Input Text, Drag And Drop, ... work with images, UI elements or regions; the engine is picked from the element argument.
• 🖼️ Image recognition – pyautogui + OpenCV, per-call similarity, multiple candidate images, offsets and search regions.
• 🌳 UI Automation – pywinauto (uia and win32 backends) for WPF, WinForms, UWP and Win32 apps.
• 🔤 OCR – RapidOCR on onnxruntime, models bundled, no external binary; accent and case insensitive text search.
• ⏳ Busy cursor aware – actions wait automatically while the app shows the busy cursor.
• 🧩 SikuliLibrary compatible – implements all its actions, plus UIA keywords (Attach To Window, Get Element Text, Set Element Value, Select Combobox Item, ...).

📦 Install

pip install robotframework-desktop          # core
pip install robotframework-desktop[ocr]     # + OCR (rapidocr + onnxruntime)

🚀 Quick start

*** Settings ***
Library    DesktopLibrary    image_paths=${CURDIR}/images;${CURDIR}/shared    similarity=0.6

*** Test Cases ***
Write In Notepad
    Open Application     notepad.exe
    Attach To Window     name_re:.*Notepad
    Input Text           class:Edit    Hello
    Click                ${{ {"path": ["save.png", "save_dark.png"], "similarity": 0.85} }}
    Wait Until Screen Contain    done.png    10s
    [Teardown]    Close Application    notepad.exe

🎯 Locators

🗂️ Locators in a YAML file

Keep all element locators in one variable file (full example):

APP:
  BTN_SAVE: "automation_id:btnSave"
  TXT_NAME:
    automation_id: txtName
    control_type: Edit
  BTN_SAVE_WIN32:
    name: Save
    class_name_re: ".*BUTTON.*"
    backend: win32
  IMG_TARGET_OFFSET:
    path: target.png
    offset:
      x: 30
      y: 20
  IMG_IN_REGION:
    path: save.png
    region:
      x: 0
      y: 0
      width: 800
      height: 600

*** Settings ***
Library      DesktopLibrary
Variables    elements.yml          # requires PyYAML

*** Test Cases ***
Save
    Input Text    ${APP.TXT_NAME}    Robot
    Click         ${APP.BTN_SAVE}
    Click         ${APP.IMG_TARGET_OFFSET}

🔤 OCR

No language setup needed — the default RapidOCR models read Latin text (accented letters may come out unaccented, so matching ignores accents and case).

${text}=    Get Text    automation_id:lblStatus
${text}=    Read Text From Region    [10, 400, 300, 30]
Click Text    Clear
Set ROI    [0, 0, 600, 400]
Region Click Text    Clear

🔍 Scaled images (DPI / font scaling)

Remote sessions (Citrix, RDP) may render menus, window titles and message boxes with a different font scaling (e.g. 125%) than when the reference images were captured. Enable multi-scale search globally or per locator:

Library    DesktopLibrary    image_scales=1, 1.1-1.6

Set Image Scales    1.0, 0.8-0.9, 1.1-1.6
Click    ${{ {"path": "menu_file.png", "scales": "1, 125%"} }}

Scale 1 is tried first; when it finds nothing, the other scales are tried and the best score wins. Ranges (1.1-1.6, step 0.02, or 1.1-1.6:0.05) are recommended because fonts are rounded to whole pixels, so the real factor varies per text. More scales mean slower searches when the image is not on screen, so combine it with Set ROI.

📋 Clipboard in remote sessions

Paste Text (and Input Text with non-ASCII text) waits clipboard_delay (default 0.5s) before and after Ctrl+V, so remote sessions receive the new clipboard content before the previous one is restored. Tune it with clipboard_delay= on import or Set Clipboard Delay.

🔁 Looping over elements

@{items}=    Find Elements    control_type:ListItem          # [] when nothing is found
FOR    ${item}    IN    @{items}
    ${text}=    Get Element Text    ${item}
    IF    $text == "Item 3"    Click    ${item}
END
@{icons}=    Find Elements    icon.png    sort_by=column    min_count=3
@{rows}=     Find Child Elements    automation_id:grid    control_type:DataItem    depth=1

⏳ Busy cursor handling

Action keywords wait while the cursor is busy (wait / appstarting) until it stays normal for 0.5 s — before searching, before acting and after the action (watched for 0.2 s). On timeout (default 30 s) the keyword fails, or only warns with busy_cursor_fail=False / ignore_busy_error=True. Use show_warnings=False to log warnings as INFO.

Library    DesktopLibrary    busy_cursor_timeout=1 min    busy_cursor_after_delay=0.3s

Click    automation_id:btnLoad    ignore_busy_error=True
Set Busy Cursor Wait     False
Set Busy Cursor Types    wait,appstarting,custom
Wait Until Cursor Is Not Busy    2 min
${cursor}=    Get Current Cursor Type

📚 Documentation

Full keyword reference: docs/DesktopLibrary.html (just docs to regenerate).

🛠️ Development

Requires uv and just (PowerShell recipes).

Release

CI (.github/workflows/ci.yml) runs lint, unit and acceptance tests on every push and pull request. To publish a release, bump version in pyproject.toml, then:

git tag v0.2.0
git push --tags

.github/workflows/release.yml runs the tests, builds both wheels and creates a GitHub Release with them attached:

• robotframework-desktop - OCR optional ([ocr] extra)
• robotframework-desktop-full - OCR deps required

📄 License

Apache-2.0