gillm

GUI Control Plugin with NLP & Intent Contracts

Sterowanie GUI przez DSL, NLP i wiele adapterów wejścia (CLI, REST, MCP, URI). Domena (focus, inject, capture, orchestrator) żyje w src/gillm/; warstwa kontroli w packages/*2gillm.

AI Cost Tracking

• 🤖 LLM usage: $2.6608 (19 commits)
• 👤 Human dev: ~$1494 (14.9h @ $100/h, 30min dedup)

Generated on 2026-07-03 using openrouter/qwen/qwen3-coder-next

Graphical interface for LLM automation in the semcod/coru ecosystem. Pair with tillm for text / shell control (aider, claude, codex, …).

Contents

• Architecture
• Control layer packages
• Installation
• Quick start
• DSL verbs
• CLI
• REST API
• Environment variables
• Testing
• Project layout
• Documentation
• License

Architecture

SUMD (description) → DOQL/source (code) → taskfile (automation) → testql (verification)

Sources: SUMD.md · app.doql.less · Makefile

flowchart TB
  subgraph adapters [Input adapters — packages]
    NL[nlp2gillm]
    URI[uri2gillm]
    CLI[cli2gillm]
    MCP[mcp2gillm]
    REST[rest2gillm]
  end

  subgraph control [Control layer]
    TXT[DSL text line]
    DICT[JSON dict]
    PBIN[protobuf bytes]
    SCH[JSON Schema validate]
    DSL[dsl2gillm.dispatch]
    Q[QueryHandler]
    C[CommandHandler]
    ES[(EventStore)]
  end

  subgraph domain [Domain — src/gillm]
    ORCH[orchestrator.drive]
    FOCUS[focus / injection]
    CAPTURE[capture]
    NLP[nlp_bridge]
  end

  NL --> TXT
  URI --> TXT
  CLI --> TXT
  MCP --> TXT
  MCP --> DICT
  MCP --> PBIN
  REST --> TXT
  REST --> DICT
  REST --> PBIN

  TXT --> SCH
  DICT --> SCH
  PBIN --> SCH
  SCH --> DSL
  DSL --> Q
  DSL --> C
  C --> ORCH
  C --> ES
  Q --> ORCH
  Q --> FOCUS
  Q --> CAPTURE
  Q --> NLP

Jedyny punkt mutacji w warstwie kontroli: dsl2gillm.dispatch().

Control layer packages

Overview: packages/README.md.

Installation

Runtime (core GUI domain only):

pip install -e .
# or: make install

Development (control layer + test tooling):

bash packages/install-dev.sh
# or: pip install -e ".[dev]"
# or: make dev-install

Optional control extras without full dev stack:

pip install -e ".[control]"

Requires Python ≥ 3.10.

Quick start

# Schema + dry-run workflow (no real GUI)
dsl2gillm validate-schema
gillm run fixtures/workflow.json --dry-run
gillm run fixtures/workflow-dry.json --dry-run

# NLP → DSL
nlp2gillm to-dsl "check health"

# REST smoke test (separate terminal)
rest2gillm serve --port 8220
curl http://127.0.0.1:8220/health
curl -X POST http://127.0.0.1:8220/v1/dsl -d 'HEALTH'

DSL verbs

Example lines:

HEALTH
ORIENT
ACTIONS
PARSE "focus vscode and type hello"
VALIDATE STEPS [{"action":"wait","config":{"seconds":0.01}}]
RESOLVE "capture screen"
CAPTURE SCALE 0.2
EXECUTE FILE workflow.json
SIMULATE FILE workflow.json
FOCUS HINTS vscode,cursor
INJECT "hello" IDE default

Generate Pydantic models from JSON schemas:

python -m dsl2gillm.codegen
# or: dsl2gillm codegen

CLI

gillm run fixtures/workflow-dry.json --dry-run
gillm nlp "focus vscode and type hello" --dry-run
gillm capture --scale 0.2
dsl2gillm -c 'HEALTH'

REST API

GET / returns an endpoint map. Default base URL: http://127.0.0.1:8220/.

rest2gillm serve --port 8220
curl http://127.0.0.1:8220/
curl http://127.0.0.1:8220/health
curl -X POST http://127.0.0.1:8220/v1/dsl -d 'HEALTH'

Environment variables

Copy .env.example to .env for local development.

GUI injection (Wayland/X11) also reads KORU_* variables — see SUMD.md for the full list (KORU_OS_INJECTOR, KORU_INJECTOR_BACKEND, WAYLAND_DISPLAY, DISPLAY, …).

Testing

Core domain (src/gillm):

python -m pytest tests/ -q
# or: make test

Main modules:

• tests/test_injector.py — gillm.injection.injector.Injector (keyboard backends: xdotool/ydotool/wtype)
• tests/test_os_injector.py — calibrated profiles under ~/.koru/ide-os-injector.json
• tests/test_gui_driver.py, tests/test_drive_backend.py, tests/test_recovery.py

Control layer (packages/*2gillm):

python -m pytest packages/dsl2gillm/tests packages/uri2gillm/tests packages/nlp2gillm/tests \
       packages/cli2gillm/tests packages/mcp2gillm/tests packages/rest2gillm/tests -q

Makefile shortcuts: make test-fast, make test-unit, make test-integration, make test-slow.

Koru keeps integration tests only (daemon wire protocol, IDE shims, CLI). Do not re-add duplicate injector unit tests under koru/tests/.

Project layout

gillm/
├── src/gillm/          # GUI domain: focus, injection, capture, orchestrator, recovery
├── packages/           # Control adapters: dsl2gillm, uri2gillm, nlp2gillm, cli2gillm, mcp2gillm, rest2gillm
├── tests/              # Domain unit tests
├── fixtures/           # Workflow JSON for dry-run smoke tests
├── app.doql.less       # DOQL application declaration
├── SUMD.md             # Structured Unified Markdown Descriptor (full project spec)
└── pyproject.toml

Documentation

Per-package docs: dsl2gillm · uri2gillm · nlp2gillm · cli2gillm · mcp2gillm · rest2gillm

License

Licensed under Apache-2.0.