pyxelator 0.4.0

Image-based automation for Selenium, Playwright & Appium - locate elements by screenshots

Pyxelator

Pixel-based Element Locator for Web & Mobile Automation

Visual element automation for Selenium, Playwright & Appium. No more XPath hunting - just use screenshots!

---

Why Pyxelator?

Traditional web automation requires finding elements by:

• XPath (breaks easily)
• CSS Selectors (tedious to write)
• IDs/Classes (may not exist)

With Pyxelator, you just need screenshots!

• Take screenshot of button/field
• Use it to find & interact
• Works with ANY element

---

Installation

pip install pyxelator

Optional Framework Dependencies:

# For Selenium
pip install pyxelator[selenium]

# For Playwright
pip install pyxelator[playwright]

# For Appium (Beta)
pip install pyxelator[appium]

# For all frameworks
pip install pyxelator[dev]

---

Quick Start

Super Simple API

from pyxelator import find, click, fill
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://example.com')

# That's it! Just use images!
if find(driver, 'login_button.png'):
 click(driver, 'login_button.png')
 fill(driver, 'email_field.png', '[e-mail]')

Or use OOP style

from pyxelator import Pyxelator
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://example.com')

px = Pyxelator(driver)

if px.find('login_button.png'):
 px.click('login_button.png')
 px.fill('email_field.png', '[e-mail]')

---

API Reference

Module Functions (Recommended)

find(driver, image, confidence=0.7)

Check if element exists.

if find(driver, 'button.png'):
 print("Button found!")

Parameters:

• driver - Selenium WebDriver or Playwright Page
• image - Path to template image
• confidence - Match confidence 0.0-1.0 (default: 0.7)

Returns: True if found, False otherwise

---

locate(driver, image, confidence=0.7)

Get element coordinates.

coords = locate(driver, 'button.png')
if coords:
 print(f"Button at {coords}") # (x, y)

Returns: (x, y) tuple or None

---

click(driver, image, confidence=0.7, retries=3, delay=0.5, debug=False)

Click element by image.

click(driver, 'submit_button.png')
click(driver, 'button.png', retries=5, delay=1.0, debug=True)

Parameters:

• driver - Selenium WebDriver, Playwright Page, or Appium driver
• image - Path to template image
• confidence - Match confidence 0.0-1.0 (default: 0.7)
• retries - Number of retry attempts (default: 3, Selenium/Playwright only)
• delay - Delay between retries in seconds (default: 0.5, Selenium/Playwright only)
• debug - Print debug information (default: False)

Returns: True if clicked, False if not found

---

fill(driver, image, text, confidence=0.7, debug=False)

Fill text into element.

fill(driver, 'email_field.png', '[e-mail]')
fill(driver, 'password.png', 'secret123', debug=True)

Parameters:

• driver - Selenium WebDriver, Playwright Page, or Appium driver
• image - Path to template image
• text - Text to fill into the element
• confidence - Match confidence 0.0-1.0 (default: 0.7)
• debug - Print debug information (default: False)

Returns: True if filled, False if not found

---

Class API

Pyxelator(driver)

from pyxelator import Pyxelator

px = Pyxelator(driver)
px.find('button.png')
px.click('button.png')
px.fill('input.png', 'text')

Methods:

• find(image, confidence=0.7) bool
• locate(image, confidence=0.7) tuple or None
• click(image, confidence=0.7) bool
• fill(image, text, confidence=0.7) bool

---

Usage Examples

Selenium

from selenium import webdriver
from pyxelator import find, click, fill
import time

driver = webdriver.Chrome()
driver.get('https://example.com')
time.sleep(1)

# Login flow
if find(driver, 'login_button.png'):
 fill(driver, 'username.png', 'myuser')
 fill(driver, 'password.png', 'mypass')
 click(driver, 'submit.png')
 print("Logged in!")

driver.quit()

Playwright

from playwright.sync_api import sync_playwright
from pyxelator import find, click, fill
import time

with sync_playwright() as p:
 browser = p.chromium.launch(headless=False)
 page = browser.new_page()
 page.goto('https://example.com')
 time.sleep(1)

 # Same API!
 if find(page, 'login_button.png'):
 fill(page, 'username.png', 'myuser')
 fill(page, 'password.png', 'mypass')
 click(page, 'submit.png')
 print("Logged in!")

 browser.close()

Appium (Beta - Mobile Automation)

from appium import webdriver
from appium.options.android.uiautomator2.base import UiAutomator2Options
from pyxelator import find, click, fill
import time

# Setup Appium
options = UiAutomator2Options()
options.udid = "your_device_id"
options.platform_name = "Android"
options.app_package = "com.example.app"
options.app_activity = "com.example.app.MainActivity"

driver = webdriver.Remote('http://127.0.0.1:4723', options=options)
time.sleep(2)

# Same API works for mobile!
if find(driver, 'login_button.png'):
 fill(driver, 'email_field.png', '[e-mail]')
 fill(driver, 'password_field.png', 'password123')
 click(driver, 'submit_button.png')
 print("Logged in!")

driver.quit()

Note: Appium support is currently in Beta. Template matching works best with unique UI elements. For optimal results, use high-confidence templates and ensure elements are fully visible on screen.

Pytest Integration

import pytest
from selenium import webdriver
from pyxelator import find, click, fill
import time

@pytest.fixture
def driver():
 driver = webdriver.Chrome()
 driver.maximize_window()
 yield driver
 driver.quit()

def test_login(driver):
 driver.get('https://example.com')
 time.sleep(1)

 assert find(driver, 'login_button.png') == True
 assert click(driver, 'login_button.png') == True
 assert fill(driver, 'email.png', '[e-mail]') == True

---

Creating Template Images

Best Practices:

1. Screenshot ONE element (button, field, icon)
2. Keep it small (50x50 to 300x200 pixels)
3. Choose unique elements (avoid generic ones)
4. Crop tightly around the element
5. Use descriptive names (login_button.png, email_field.png)

How to Create:

# Helper script to create templates
from selenium import webdriver
import time

driver = webdriver.Chrome()
driver.get('https://yoursite.com')
time.sleep(2)

# Take full screenshot
driver.save_screenshot('fullpage.png')

# Now open fullpage.png in image editor
# Crop the specific element you want
# Save as: login_button.png, email_field.png, etc.

---

Confidence Levels

Adjust confidence parameter for matching accuracy:

# Strict matching (0.8-1.0)
find(driver, 'button.png', confidence=0.9)

# Balanced (0.7) - RECOMMENDED
find(driver, 'button.png', confidence=0.7)

# Relaxed (0.5-0.6)
find(driver, 'button.png', confidence=0.5)

Recommendation: Start with 0.7, adjust if needed.

---

Framework Compatibility

Selenium WebDriver - Fully supported Playwright - Fully supported Appium - Beta (mobile & desktop apps)

Pyxelator automatically detects which framework you're using!

Appium Beta Status

Appium support is functional but still in beta. Known considerations:

• Works with native mobile apps (Android/iOS)
• Uses W3C Actions API for tap gestures
• Template matching may require lower confidence (0.5-0.6) for some elements
• Best results with high-resolution screenshots and unique UI elements

---

Error Handling & Debugging

Debug Mode (v0.4.0+)

Enable detailed logging to troubleshoot issues:

# Click with debug mode
click(driver, 'button.png', debug=True)

# Fill with debug mode
fill(driver, 'input.png', 'text', debug=True)

Debug output shows:

• File existence validation
• Element search attempts
• Coordinates found
• Clickability/fillability validation
• Success/failure details

Common Errors & Solutions

File Not Found

[Pyxelator ERROR] Template image file not found: 'button.png'

Solution: Check file path, use absolute path, or verify file exists

Element Not Found

[Pyxelator ERROR] Element not found after 3 attempts: 'button.png'

Solutions:

1. Lower confidence: click(driver, 'button.png', confidence=0.6)
2. Recapture template at same window size
3. Use debug mode: click(driver, 'button.png', debug=True)

Element Not Clickable

[Pyxelator ERROR] Element is not clickable
[Pyxelator] Found: <DIV> "Some text..."

Solution: Recapture a smaller screenshot focused on the actual button

Element Not Fillable

[Pyxelator ERROR] Element is not fillable
[Pyxelator] Found: <BUTTON> "Submit"

Solution: Ensure template captures an input/textarea field, not a button

Retry Mechanism

Configure retry attempts for flaky elements:

# Retry up to 5 times with 1 second delay
click(driver, 'button.png', retries=5, delay=1.0)

---

Troubleshooting

Element Not Found?

1. Use debug mode - click(driver, 'button.png', debug=True)
2. Check template size - Must be smaller than viewport
3. Lower confidence - Try confidence=0.6
4. Verify element visibility - Element must be on screen
5. Check template quality - Use clear screenshots
6. Window size consistency - Capture templates at same window size as tests

Click Not Working?

1. Enable debug mode to see what's happening
2. Verify element is clickable (Pyxelator now validates this automatically)
3. Add retry attempts - click(driver, 'button.png', retries=5)
4. Check if element is covered by other elements

Fill Not Working?

1. Enable debug mode for detailed error info
2. Ensure element is input field (Pyxelator validates this)
3. Try clicking before filling to focus the element
4. Recapture template focused on input field, not label

---

Advanced Usage

Multiple Actions

from pyxelator import Pyxelator

px = Pyxelator(driver)

# Chain actions
if px.find('login.png'):
 px.fill('email.png', '[e-mail]')
 px.fill('pass.png', '12345')
 px.click('submit.png')

Custom Confidence Per Element

# Strict for critical buttons
click(driver, 'delete_button.png', confidence=0.9)

# Relaxed for dynamic elements
fill(driver, 'search_box.png', 'query', confidence=0.6)

Error Handling

if not find(driver, 'button.png'):
 print("Button not found, taking alternative action...")
 # Fallback logic here

---

Testing

Run tests:

# Selenium tests
pytest test_pyxelator_selenium.py -v

# Playwright tests
pytest test_pyxelator_playwright.py -v

# All tests
pytest -v

---

Contributors

Author & Maintainer:

• Aria Uno Suseno (@idejongkok)

Contributors:

• Eri Permadi
• Yali Yanto Silitonga

Testers:

• Eri Permadi
• Yali Yanto Silitonga

---

Contributing

Contributions welcome! Please:

1. Contact me first on Instagram @idejongkok
2. Fork the repository
3. Create feature branch
4. Add tests
5. Submit pull request

---

License

MIT License - see LICENSE file

---

Changelog

v0.4.0 (Latest)

• NEW: Comprehensive error handling for all adapters (Selenium, Playwright, Appium)
• NEW: File validation - checks if template image exists before processing
• NEW: Clickability detection - validates element is clickable before clicking (Selenium & Playwright)
• NEW: Fillability detection - validates element is fillable before filling text (Selenium & Playwright)
• NEW: Debug mode - detailed logging with debug=True parameter for troubleshooting
• NEW: Retry mechanism - configurable retry attempts for Selenium & Playwright (default: 3 retries)
• NEW: React compatibility - proper event handling for React form inputs
• IMPROVED: Clear, actionable error messages with troubleshooting tips
• IMPROVED: Smart element detection - finds clickable/fillable parent elements
• DOCS: Complete error handling guide (ERROR_HANDLING_GUIDE.md)
• DOCS: Updated implementation status for all adapters

v0.3.1

• Beta: Appium support for mobile automation
• W3C Actions API integration for mobile gestures
• Enhanced template matching algorithm
• Bug fixes and stability improvements

v0.2.x

• Playwright support improvements
• Enhanced error handling

v0.1.0 (Initial Release)

• Selenium support
• Playwright support
• Auto framework detection
• Simple function API
• OOP class API
• Template matching with OpenCV
• Click, fill, find, locate actions

---

Support

• Issues: GitHub Issues
• Docs: Full Documentation
• Discussions: GitHub Discussions

---

more question find me on:

• Instagram: https://instagram.com/idejongkok
• YouTube: https://youtube.com/idejongkok
• Website: https://idejongkok.com

Happy Automating! from idejongkok with love