guirecognizer 0.1.1

Recognize on-screen patterns and perform GUI interactions

guirecognizer is a Python library for recognizing on-screen patterns and driving GUI interactions.

The main goal of guirecognizer is to recognize on-screen patterns, such as retrieving a pixel color, computing an image hash over a screenshot area, or finding a specific image. It can also perform basic GUI interactions, such as clicking on screen elements.

The companion application guirecognizerapp helps you create and preview actions and preprocessing operations through a visual interface, before using them in a Python bot with guirecognizer.

See documentation with tutorials and examples: https://guirecognizer.readthedocs.io.

Quick overview

guirecognizer relies on a configuration file that defines the actions.

Let’s install the companion application guirecognizerapp to generate a configuration file. This will also install guirecognizer.

(venv) $ pip install guirecognizerapp

Launch the application:

(venv) $ python -m guirecognizerapp

From the application, take a screenshot using Capture -> Take Screenshot or use the keyboard shortcut Ctrl+Alt+T.

The first step is to define the borders. The borders represent the absolute coordinates of the screen region that serve as a reference for all actions. All action selections are defined relative to these borders. This greatly improves the reusability of the configuration file across different screen resolutions or setups.

Click on the Make Selection button to define the borders.

Select an area of the screenshot using the mouse or the controls at the bottom of the interface.

Select the borders on the screenshot.

For the sake of this tutorial, let’s retrieve the color of a single pixel. Create a new Get Pixel Color action: Manage Actions -> Add Action Get Pixel Color. Name your action getColor and select a pixel.

Select a point within the borders on the screenshot.

You can preview the action by clicking on the eye icon.

Preview the action getColor by clicking on the eye icon.

Preview of the action getColor.

The preview shows the pixel color retrieved by the getColor action.

Save the configuration file as guirecognizerConfig.json in your project folder: File -> Save or Ctrl+S. Now the configuration file can be used with guirecognizer.

In your Python script:

from guirecognizer import Recognizer

recognizer = Recognizer('guirecognizerConfig.json')
color = recognizer.executePixelColor('getColor')
print(color)

This produces the following output:

(243, 207, 85)

Development

Install the dependencies

(venv) $ pip install -e ".[dev]"

Create wheel

(venv) $ python -m build

OCR

Two optical character recognition libraries are supported: EasyOCR and tesseract.

Install EasyOCR (already in the dev dependencies).

(venv) $ python install easyocr

Install pytesseract (already in the dev dependencies).

(venv) $ python install easyocr

Install tesseract: follow installation instruction in https://github.com/tesseract-ocr/tesseract.

Generate full coverage report

(venv) $ coverage run
(venv) $ coverage html

Then open htmlcov/index.html

Documentation

Generate doc

(venv) $ make html

Then open docsBuild/index.html

Use the script scripts/optimizeImages.sh in a bash console to reduce the size of images before pushing them in git. Image files are converted to webp files.

For instance:

$ ./scripts/optimizeImages.sh docs/_static/app

Coding style

Some visual studio code settings of visualStudioCodeSettings.json should be used to ensure some homogeneity in the coding style.

In Visual Studio Code, install extensions

• isort to sort Python automatically

• Pylance for type checking

Improvements

Improve logger usage: in code and tests.