Qontinui

A Python library for model-based GUI automation with intelligent state management and visual recognition.

Research Foundation

Based on Model-based GUI Automation published in Springer's Software and Systems Modeling journal (October 2025).

The research provides:

• Mathematical proof of complexity reduction (exponential → polynomial)
• First testable approach to GUI automation (unit tests, integration tests)
• Formal framework for robust visual APIs for RL agents
• Enables reliable dataset generation for AI training

Overview

Qontinui enables building robust GUI automation through:

• Model-based state management using MultiState | Docs
• Visual recognition with OpenCV template matching
• JSON configuration for defining automation workflows
• Cross-platform support (Windows, macOS, Linux)

Qontinui is a Python port of Brobot, a Java library for GUI automation (2018-2024).

Installation

From PyPI (Recommended)

pip install qontinui

From Source (Development)

# Clone the repository
git clone https://github.com/jspinak/qontinui.git
cd qontinui

# Install with Poetry
poetry install

# Or with pip
pip install -e .

Dependencies

Qontinui requires:

• MultiState - Multi-state state management
• OpenCV - Image template matching
• PyAutoGUI/pynput - Input control

Quick Start

JSON Configuration

Create an automation configuration in JSON:

{
  "version": "1.0",
  "states": [
    {
      "name": "LoginScreen",
      "stateImages": [
        {
          "imageId": "login_button",
          "threshold": 0.9
        }
      ]
    }
  ],
  "processes": [
    {
      "name": "Login",
      "actions": [
        {
          "type": "CLICK",
          "target": {"type": "image", "imageId": "login_button"}
        },
        {
          "type": "TYPE",
          "text": "username@example.com"
        }
      ]
    }
  ]
}

Python API

from qontinui.json_executor import JSONRunner

# Initialize runner
runner = JSONRunner()

# Load configuration
runner.load_configuration("automation_config.json")

# Execute automation
success = runner.run(monitor_index=0)

Desktop Application

Use Qontinui Runner for a GUI interface to create and run automations.

Architecture

qontinui/
├── src/qontinui/
│   ├── json_executor/       # JSON configuration execution
│   ├── model/               # State, Transition, Image models
│   ├── hal/                 # Hardware Abstraction Layer
│   └── multistate_adapter/  # MultiState integration

Features

Current

• ✅ JSON-based automation configuration
• ✅ Template-based image matching
• ✅ Multi-state state management
• ✅ Process and state machine execution modes
• ✅ Cross-platform input control (PyAutoGUI, pynput)
• ✅ Hardware abstraction layer for multiple backends

Planned

• 🔄 AI-enhanced visual recognition (SAM, CLIP)
• 🔄 Domain-specific language (DSL)
• 🔄 Advanced Brobot migration tools
• 🔄 Cloud execution via qontinui-web

Testing

Run the test suite:

# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=src/qontinui

# Run specific test file
poetry run pytest tests/json_executor/test_json_runner.py

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Please note that this project is released with a Code of Conduct. By participating in this project you agree to abide by its terms.

Qontinui is a faithful port of Brobot. When contributing, please preserve Brobot's architecture and behavior.

Documentation

• GitHub: github.com/qontinui/qontinui
• Issues: GitHub Issues
• MultiState Docs: qontinui.github.io/multistate
• Research Paper: Springer SoSyM

License

MIT License - See LICENSE file for details.

Acknowledgments

• Brobot - The original Java library
• MultiState - Multi-state state management
• PyAutoGUI - GUI automation
• OpenCV - Computer vision

Related Projects

• qontinui-runner - Desktop application (Rust/TypeScript)
• qontinui-api - REST API bridge (enables custom frontends)
• qontinui-web - Web-based visual builder (launching Feb 2026)
• multistate - State management library | Docs
• Brobot - Original Java implementation (2018-2024)