🌟 Hikari (光) — Autonomous Windows AI Agent

---

⚡ Overview

Hikari (光) is a next-generation Autonomous Computer-Use AI Agent engineered specifically for the Microsoft Windows operating system.

Unlike traditional browser-restricted agents or costly vision-only agents that burn thousands of tokens capturing full-screen screenshots on every turn, Hikari reads and decomposes the Native Windows UI Accessibility Tree directly via OS APIs. This enables lightning-fast, ultra-precise, token-efficient desktop control across any Windows application — from native Win32 apps and UWP modern apps to web browsers, terminal consoles, and complex creative suites.

Hikari thinks like an expert human operator: observing open windows, analyzing interactive elements, planning next actions, moving the cursor via natural Bezier curves, typing text, executing PowerShell scripts, and self-correcting when errors occur.

---

🚀 Key Features & Superpowers

• ⚡ Native Windows Accessibility Tree Engine: Directly extracts interactive, informative, and scrollable UI controls (buttons, inputs, menus, panes) with exact bounding boxes and coordinates without requiring heavy image processing.
• 🧠 Chain-of-Thought State Machine: Uses structured XML reasoning tags (<evaluate>, <memory>, <plan>, <thought>, <action_name>, <action_input>) to ensure rock-solid decision-making and loop recovery.
• 🎯 Human-like Precision Control: Simulates natural human mouse trajectories using Bezier curves (SystemCursor) to avoid robotic click detection and maintain smooth UI interaction.
• 🌐 Multi-Model LLM Agnostic: Fully supports Groq Cloud (ultra-low latency), Google Gemini 2.0 Flash, OpenAI GPT-4o, GPT-4o-mini, and Local Offline models via Ollama.
• 💻 Hybrid GUI + Shell Automation: Combines graphical mouse/keyboard interactions with direct PowerShell and CMD command execution for instant speedups.
• 🪟 Smart Window Manager: Detects active windows, monitors application launch statuses, switches foreground windows, and resizes layouts dynamically.
• 📋 System Clipboard & Web Scraping: Built-in clipboard read/write capabilities and automated webpage scraping with markdown conversion.

---

🏗️ System Architecture

                                    +-----------------------+
                                    |     User Request      |
                                    +-----------+-----------+
                                                |
                                                v
                                    +-----------------------+
                                    |   Hikari Entrypoint   |
                                    |  (CLI / Python API)   |
                                    +-----------+-----------+
                                                |
                                                v
                               +---------------------------------+
                               |   hikari.desktop.service        |
                               |  - Window Enumeration (GetApps) |
                               |  - Active Window Detection      |
                               |  - Screenshot (Optional Vision) |
                               +----------------+----------------+
                                                |
                                                v
                               +---------------------------------+
                               |     hikari.tree.service         |
                               |  - Windows Accessibility Tree   |
                               |  - Interactive Nodes (Clickable)|
                               |  - Informative Nodes (Text)     |
                               |  - Scrollable Panes (Offsets)   |
                               +----------------+----------------+
                                                |
                                                v
                               +---------------------------------+
                               |     hikari.agent.prompt         |
                               |  - Dynamic XML Context Builder  |
                               |  - Desktop State Injection      |
                               +----------------+----------------+
                                                |
                                                v
                               +---------------------------------+
                               |      LLM Reasoning Engine       |
                               |  (Groq / Gemini / GPT-4o / etc) |
                               |  Output: <thought> & <action>   |
                               +----------------+----------------+
                                                |
                                                v
                               +---------------------------------+
                               |    hikari.agent.tools.service   |
                               |  - Click / Type / Shell / Drag  |
                               |  - Launch / Switch / Shortcut   |
                               +----------------+----------------+
                                                |
                                                v
                                    +-----------------------+
                                    | Windows OS Execution  |
                                    +-----------------------+

---

🧰 Built-in Tool Catalog

Hikari features an extensible, type-safe tool registry powered by Pydantic models:

Tool Name	Arguments / Parameters	Description
Launch Tool	name: str	Launches any application found in the Start Menu or PATH (e.g., notepad, chrome, spotify, calc).
Click Tool	loc: (x, y), button: left|right|middle, clicks: 0|1|2|3	Simulates human mouse clicks with hover (0), single (1), double (2), or triple (3) clicks at exact coordinates.
Type Tool	loc: (x, y), text: str, clear: true|false, caret_position: start|idle|end, press_enter: true|false	Clicks target element, handles caret positioning, optionally clears existing text, types naturally, and sends Enter.
Shell Tool	command: str	Executes native PowerShell commands and returns terminal output, errors, and exit codes.
Switch Tool	name: str	Brings target running application window directly to the foreground.
Resize Tool	name: str, loc: (x, y), size: (w, h)	Repositions and resizes any application window.
Scroll Tool	loc: (x, y), type: horizontal|vertical, direction: up|down|left|right, wheel_times: int	Performs vertical or horizontal scroll wheel steps on active or targeted controls.
Drag Tool	from_loc: (x1, y1), to_loc: (x2, y2)	Drags UI elements or files across coordinates with mouse down, trajectory move, and mouse up.
Move Tool	to_loc: (x, y)	Moves the mouse pointer smoothly without clicking.
Shortcut Tool	shortcut: list[str]	Presses keyboard shortcuts (e.g., ['ctrl', 'c'], ['alt', 'f4'], ['win', 'r']).
Key Tool	key: str	Presses single special keys (e.g., enter, escape, tab, backspace, f5).
Clipboard Tool	mode: copy|paste, text: Optional[str]	Reads or writes text to the Windows system clipboard.
Scrape Tool	url: str	Fetches webpage HTML and converts it into clean, token-efficient Markdown text.
Wait Tool	duration: int	Pauses execution for a specified duration in seconds to allow UI loading.
Done Tool	answer: str	Signals task completion and returns the final markdown summary to the user.

---

💻 Installation & Setup

Prerequisites

• Operating System: Windows 10 or Windows 11 (64-bit)
• Python Version: Python 3.13 or higher
• Package Manager: uv (recommended) or pip

1. Clone the Repository

git clone https://github.com/Codewithevilxd/hikari.git
cd hikari

2. Create Virtual Environment & Install Dependencies

Using uv (ultra-fast):

uv venv
uv pip install -e ".[dev]"

Or using standard pip:

python -m venv .venv
.\.venv\Scripts\activate
pip install -e ".[dev]"

3. Configure Environment Variables

Create a .env file in the root directory (or copy .env-example):

# Groq Cloud API Key (Ultra-fast LLMs)
GROQ_API_KEY="your-groq-api-key-here"

# Google Gemini API Key (Recommended free tier)
GOOGLE_API_KEY="your-gemini-api-key-here"

# OpenAI API Key (Optional)
OPENAI_API_KEY="your-openai-api-key-here"

---

🎮 Usage Guide

1. Interactive CLI Interface

Launch the interactive terminal:

.\.venv\Scripts\python main.py

You will be greeted with the Hikari interactive menu:

  _    _ _____ _  __     _____  _____ 
 | |  | |_   _| |/ / /\ |  __ \|_   _|
 | |__| | | | | ' / /  \| |__) | | |  
 |  __  | | | |  < / /\\ \  _  /  | |  
 | |  | |_| |_| . \ / ____ \ | \ \ _| |_ 
 |_|  |_|_____|_|\_/_/    \_\|  \_\_____|
           光 - Autonomous Windows AI Agent
     Developed by Nishant Gaurav (Codewithevilxd)

Choose your AI Model Engine:
  1. Google Gemini 2.0 Flash (Recommended - Superfast & Free Tier)
  2. OpenAI GPT-4o (High Precision Desktop Control)
  3. OpenAI GPT-4o-mini (Budget Fast Option)
  4. Groq Cloud AI (Ultra-low Latency - GPT-OSS / Qwen / Compound)
  5. Local Ollama (Private / Offline)

Type any task in plain English or Hindi:

• "Open Notepad and write a 4-line poem about rain, then save it to Desktop"
• "Launch Chrome, search for latest space discoveries, and copy the first headline"
• "Open Spotify and play my favourite playlist"
• "Check available disk space in PowerShell and report back"

---

2. Programmatic Python SDK Integration

You can easily integrate Hikari into your own Python applications and automated pipelines:

import os
from dotenv import load_dotenv
from langchain_groq import ChatGroq
from hikari.agent import Agent

load_dotenv()

# 1. Initialize your preferred LLM
llm = ChatGroq(
    model_name="openai/gpt-oss-120b",
    temperature=0.1
)

# 2. Instantiate Hikari Agent
agent = Agent(
    llm=llm,
    browser="chrome",       # Preferred browser
    use_vision=False,       # Set True for vision-augmented workflows
    auto_minimize=True,     # Auto minimize console during execution
    max_steps=15,           # Max autonomous decision cycles
    consecutive_failures=3  # Retry limit for stuck states
)

# 3. Execute any desktop task
task = "Open Calculator and compute 1024 * 768"
result = agent.invoke(task)

print("Task Result:", result.content)

---

🧪 Running Automated Tests

Hikari comes with a comprehensive test suite containing 159 unit tests covering all modules, registry services, desktop bindings, and tree parsing:

.\.venv\Scripts\python -m pytest tests/ -v

Expected Output:

============================= 159 passed in 1.94s =============================

---

📂 Project Directory Structure

hikari/
├── hikari/
│   ├── __init__.py                  # Package root exporter
│   ├── agent/
│   │   ├── __init__.py
│   │   ├── service.py               # Main Agent orchestrator & LangGraph state machine
│   │   ├── state.py                 # Pydantic agent memory states
│   │   ├── utils.py                 # Regex XML parser & image serialisation
│   │   ├── views.py                 # Core Agent data schemas
│   │   ├── prompt/
│   │   │   ├── __init__.py
│   │   │   ├── service.py           # Dynamic prompt formatter
│   │   │   ├── system.md            # Hikari Core System Persona
│   │   │   ├── action.md            # Action decision prompt
│   │   │   ├── answer.md            # Final answer prompt
│   │   │   └── observation.md       # Desktop observation prompt
│   │   ├── registry/
│   │   │   ├── __init__.py
│   │   │   ├── service.py           # Dynamic tool execution registry
│   │   │   └── views.py             # ToolResult & metadata models
│   │   └── tools/
│   │       ├── __init__.py
│   │       ├── service.py           # Native tool functions (Click, Type, Shell, etc.)
│   │       └── views.py             # Pydantic tool argument schemas
│   ├── desktop/
│   │   ├── __init__.py
│   │   ├── config.py                # Avoided & excluded system applications
│   │   ├── service.py               # Windows process & window management
│   │   └── views.py                 # App & DesktopState view models
│   └── tree/
│       ├── __init__.py
│       ├── config.py                # Accessibility control type maps
│       ├── service.py               # Native Accessibility Tree parser
│       ├── utils.py                 # Bounding box & coordinate helpers
│       └── views.py                 # Interactive & scrollable node schemas
├── tests/
│   ├── conftest.py                  # Pytest fixtures & mock configuration
│   └── unit/                        # 14 unit test suites (159 tests)
├── .env-example                     # Template for API keys
├── LICENSE                          # MIT License
├── main.py                          # Interactive Rich CLI application
├── main_optimized.py                # Optimized fast runner
└── pyproject.toml                   # Project metadata & package specification

---

📜 License

This project is licensed under the MIT License — see the LICENSE file for details.

---

👤 Author & Maintainer

Nishant Gaurav (Codewithevilxd)

• 🐙 GitHub: @Codewithevilxd
• 📧 Email: codewithevilxd@gmail.com
• 🌐 Project Repository: https://github.com/Codewithevilxd/hikari

---

Hikari (光) — Illuminating Windows Desktop Automation.