TestQL — Interface Query Language for Testing

AI Cost Tracking

• 🤖 LLM usage: $29.9715 (130 commits)
• 👤 Human dev: ~$7463 (74.6h @ $100/h, 30min dedup)

Generated on 2026-07-20 using openrouter/qwen/qwen3-coder-next

TestQL is a declarative DSL (Domain Specific Language) for testing GUI, REST API, and hardware encoder interfaces. It provides a simple, readable syntax for writing automated tests without programming overhead.

What's new in 1.2.x

• Native Desktop E2E: DESKTOP_* commands with vdisplay mirror→Xvfb capture, img2nl + imgl OCR on Linux (no Wayland portal)
• Artifact Discovery & Topology: Complete codebase inspection with manifest generation
• Web Inspection: Live URL scanning with Playwright browser automation
• Multi-DSL Platform: TestTOON, NL, SQL, Proto, GraphQL adapters with Unified IR
• Autoloop Integration: MCP-driven development with autonomous test loops
• Refactor Plans: Structured refactoring plans with NLP summaries
• Production deployment in c2004: 50-assertion realtime-health scenario running every 60s as part of Prometheus + planfile self-healing pipeline

Recent Improvements (1.2.48)

• c2004 Healing Pipeline: testql-watchdog container, Prometheus metrics, LLM-ready ticket generation on failure
• MCP Server: testql/mcp/server.py — resources for topology, tools for discover/run/explain
• Internal IR: testql/ir/ module with steps, assertions, captures, fixtures, plan
• TestTOON Adapter: full round-tripping with validation layer
• Configuration Management: refactored config system
• Code Analysis Engine: refactored analysis pipeline
• Markdown Output: --output markdown for human-readable reports
• Artifact Discovery: testql discover with topology graph generation
• Web Inspection: --scan-network with headless browser capture, console errors, network logs
• Multi-Format Output: JSON, YAML, TOON for all results and topology
• Browser Testing: Playwright integration for JS-rendered DOM capture
• API Testing: 7 OpenAPI endpoints with FastAPI integration
• CLI/Shell Execution: SHELL, EXEC, RUN commands with assertions
• GUI Execution: Playwright/Selenium support with GUI_START, GUI_CLICK, GUI_INPUT
• Unit Test Execution: UNIT_PYTEST, UNIT_ASSERT, UNIT_IMPORT integration
• Test Coverage: 65% with pyqual.yaml quality pipeline

Installation

# Install from source
pip install -e .

# Install with development dependencies
pip install -e ".[dev]"

Requirements

• Python 3.10+
• HTTPX for API testing
• Chrome/Chromium for browser GUI testing. TestQL installs the Python Playwright API and can also reuse Playwright from the calling project's node_modules automatically.
• Native desktop E2E (optional): vdisplay, img2nl, imgl — see Desktop GUI E2E

Optional: native desktop + vision

pip install -e ".[desktop,vision]" \
  -e ~/github/wronai/vdisplay[pillow] \
  -e ~/github/wronai/img2nl[analyze,similarity,opencv,scan] \
  -e ~/github/semcod/imgl

# System: xvfb x11-apps xdotool xrandr tesseract-ocr

Quick Start

# Run a test scenario
testql scenarios/tests/test-api.testql.toon.yaml --url http://localhost:8101

# Dry-run (parse + validate only)
testql scenarios/views/connect-id-barcode.testql.toon.yaml --dry-run

# JSON output for CI integration
testql scenarios/tests/test-api.testql.toon.yaml --output json

# Run with verbose logging
testql scenarios/tests/test-api.testql.toon.yaml --verbose

Public verification API

Orchestrators should use the versioned Python contract instead of importing CLI internals. TestQL owns path/glob resolution and batch result normalization; the caller receives canonical request/result hashes for audit and replay.

from testql.verification import VerificationRequest, run_verification

request = VerificationRequest(
    file_specs=("scenarios/**/*.testql.toon.yaml",),
    project_dir=".",
    dry_run=True,
)
result = run_verification(request)
print(result.to_dict())  # testql.verification-result.v1

Packaged JSON Schemas are available through verification_contract_schema().

Artifact Discovery, Topology, and Web Inspection

TestQL can now inspect codebases, manifests, and opt-in live URLs, then write structured topology and result artifacts for humans, CI, and LLM workflows.

# Discover local artifact types
python3 -m testql.cli discover ./testql --format json

# Build a topology graph from discovered artifacts
python3 -m testql.cli topology ./testql --format toon

# Inspect a live web page and write all data/metadata to .testql
python3 -m testql.cli inspect https://tom.sapletta.com/ \
  --scan-network \
  --out-dir .testql

The .testql/ artifact bundle contains:

metadata.json
topology.{json,yaml,toon.yaml}
result.{json,yaml,toon.yaml}
refactor-plan.{json,yaml,toon.yaml}
inspection.{json,yaml,toon.yaml}
summary.md

Live URL inspection currently extracts:

• HTTP metadata: status code, final URL, content type.
• Page schema: title, links, assets, forms.
• Topology graph: page, link, asset, form, interface, evidence nodes.
• Structured results: web checks for status, title, links, assets, forms.
• Reports: TOON/YAML/JSON plus NLP summary.

Example:

examples/web-inspection-dot-testql/run.sh https://tom.sapletta.com/

Current capabilities:

• Asset classification: script, stylesheet, image, icon, preload, link.
• Bounded link validation: HEAD checks for all internal links (up to 100).
• Bounded asset validation: HEAD checks for all extracted assets.
• Broken resource detection: assets or links returning error status are flagged as findings.
• Bounded sitemap crawl: fetches up to 10 internal subpages, extracts titles and link counts, adds subpage nodes to the topology.
• Sitemap checks: crawl coverage, broken subpage detection, duplicate title warnings.
• Playwright browser inspection (--browser): renders the page in a headless browser, captures console errors, network calls (REST/GraphQL/WebSocket), and JS-rendered DOM.
• Browser checks: render detection, console error count, network call capture, title extraction, link/asset/form enumeration.

Current limitations:

• Per-resource validation uses HEAD requests only; full page content is not fetched for linked pages.
• Screenshots, performance metrics, accessibility checks, and auth flows are planned next.

API Endpoint Detection

TestQL includes advanced endpoint detection for multiple frameworks:

# List all detected endpoints in a project
testql endpoints ./my-project
testql endpoints ./my-project --format json
testql endpoints ./my-project --framework fastapi

# Analyze project structure and generate tests
testql analyze ./my-project
testql generate ./my-project

Supported Frameworks

• FastAPI — Detects routers, decorators, include_router patterns
• Flask — Blueprints, MethodView, route decorators
• Django — URL patterns from urls.py
• Express.js — JavaScript/TypeScript route definitions
• OpenAPI/Swagger — Spec file parsing (JSON/YAML)
• GraphQL — Schema and resolver detection
• WebSocket — WS endpoint detection

Endpoint Detection Features

• Framework Detection: Automatically identifies the web framework
• Handler Names: Extracts function/method names for documentation
• Parameters: Detects path/query/body parameters
• Docstrings: Uses function docstrings for test descriptions
• Test Inference: Analyzes existing tests to discover endpoints

OpenAPI Generation

Generate OpenAPI 3.0 specifications from your code:

# Generate OpenAPI spec (YAML format)
testql openapi ./my-project
testql openapi ./my-project --format json

# Generate with contract tests
testql openapi ./my-project --contract-tests

# Custom output file
testql openapi ./my-project -o ./docs/api-spec.yaml --title "My API"

Contract Testing

TestQL generates contract tests from OpenAPI specs:

# Auto-generated from openapi.yaml
API[25]{method, endpoint, expected_status}:
  GET, /api/v1/users, 200
  POST, /api/v1/users, 201
  ...

ASSERT[3]{field, operator, expected}:
  content_type, ==, application/json
  schema_valid, ==, true
  status, <, 500

Project Echo (AI Context)

Generate AI-friendly project metadata by combining TESTQL scenarios with DOQL system models:

# Generate unified project context
testql echo --toon-path testql-scenarios/ --doql-path app.doql.less

# JSON output for LLM consumption
testql echo --toon-path ./tests --doql-path ./app.doql.less --format json -o context.json

# Text output (human-readable)
testql echo --doql-path ./app.doql.less --format text

Echo Layers

Example Output

📦 Project: weboql (0.1.2)

🧠 Type:
  • API (fastapi)

🛠️ Workflows:
  • install: pip install -e .
  • test: pytest
  • run: HARDWARE_MODE=mock weboql-server

🌐 API scenarios:
  • API Health Check (api) - 4 endpoint(s)

💡 LLM suggestions:
  • Run tests: task test
  • Start server: task run

Language Reference

Variables

SET api_url "http://localhost:8101"
SET timeout 5000
GET api_url              # Prints variable value

Variables support ${var} and $var interpolation:

SET base_url "http://localhost:8101"
API GET "${base_url}/api/devices"

Logging

LOG "Starting test suite"
LOG "Current value: ${var}"

API Commands

# HTTP methods
API GET "/api/v3/data/devices"
API POST "/api/v3/scenarios" {"id": "ts1", "name": "Test"}
API PUT "/api/v3/devices/123" {"status": "active"}
API DELETE "/api/v3/scenarios/old"

Assertions

# Status code assertions
ASSERT_STATUS 200
ASSERT_OK                  # Shorthand for 2xx status

# Content assertions
ASSERT_CONTAINS "device"
ASSERT_CONTAINS "status": "ok"

# JSON path assertions
ASSERT_JSON data.length > 0
ASSERT_JSON devices[0].id == "dev-001"
ASSERT_JSON status != "error"

Operators: ==, !=, >, >=, <, <=

GUI Navigation (Playwright)

Browser-based GUI tests via Playwright. For native OS desktop (monitors, windows, OCR, mirror capture on Wayland), see Desktop GUI E2E.

GUI_START first uses the Python Playwright installed with TestQL. Existing environments that predate this dependency automatically fall back to node_modules/playwright in the calling project and its parent directories. An installed Chrome/Chromium executable is detected automatically. Custom locations can be supplied with TESTQL_NODE_PLAYWRIGHT_PATH, TESTQL_PLAYWRIGHT_SEARCH_PATHS, or TESTQL_BROWSER_EXECUTABLE.

# Navigation
NAVIGATE "/connect-workshop"
NAVIGATE "${base_url}/devices"

# Interaction
WAIT 500
CLICK "[data-action='search']"
INPUT "#search-input" "drager"
CLICK "button[type='submit']"

# Assertions
ASSERT_VISIBLE "[data-testid='results']"
ASSERT_TEXT "#status" "Connected"

Native Desktop (Linux)

Real OS desktop E2E via vdisplay mirror→Xvfb (no Wayland portal). Full guide: Desktop GUI E2E.

DESKTOP_MONITORS
DESKTOP_CAPTURE "shot.png" primary
DESKTOP_ASSERT_WINDOW "Toolbox"
DESKTOP_ASSERT_ELEMENTS 1 "shot.png"
DESKTOP_ANALYZE "shot.png" "layout.json"
DESKTOP_FOCUS "Toolbox"

DISPLAY=:0 testql run examples/desktop/gui-e2e-inspect.oql
DISPLAY=:0 bash examples/desktop/run-all.sh

Hardware Encoder Commands

ENCODER_ON                    # Activate encoder mode
ENCODER_FOCUS column1         # Focus specific column
ENCODER_SCROLL 3              # Scroll 3 steps
ENCODER_CLICK                 # Confirm (single click)
ENCODER_DBLCLICK              # Cancel (double click)
ENCODER_PAGE_NEXT             # Next page
ENCODER_PAGE_PREV             # Previous page
ENCODER_STATUS                # Print current encoder state
ENCODER_OFF                   # Deactivate encoder mode

Script Composition

# Include common setup
INCLUDE "common-setup.testql.toon.yaml"

# Include relative paths
INCLUDE "../helpers/auth.testql.toon.yaml"

Control Flow (Planned)

IF status == "ready"
  LOG "System ready"
ELSE ERROR "System not ready"

LABEL start
REPEAT 3 {
  API GET "/api/ping"
}
GOTO start

Project Structure

testql/
├── testql/
│   ├── cli.py           # Command-line interface
│   ├── base.py          # Base test runner
│   ├── commands/        # Command implementations
│   ├── runners/         # Test execution runners
│   └── reporters/       # Output formatters
├── scenarios/           # Sample test scenarios
│   ├── tests/          # Automated test suites
│   ├── views/          # GUI view tests
│   ├── diagnostics/    # System diagnostic scripts
│   ├── examples/       # Usage examples
│   └── recordings/     # Recorded test sessions
├── docs/
│   ├── testql-spec.md  # Full language specification
│   └── recipes/        # Common testing patterns
└── tests/              # Unit tests

Scenario Organization

TestQL scenarios use two formats:

• *.testql.toon.yaml — TestTOON tabular format for test sequences (preserves order)
• *.testql.less — LESS format for shared test configuration (variables, mixins)

TestTOON Format

Test files use the tabular TestTOON format where each section declares its schema:

# SCENARIO: API Smoke Test
# TYPE: api
# VERSION: 1.0

API[3]{method, endpoint, status}:
  GET,  /api/v3/health,   200
  GET,  /api/v3/devices,  200
  POST, /api/v3/start,    201

ASSERT[2]{field, op, expected}:
  status,  ==, ok
  count,   >,  0

LESS Configuration

Shared test configuration uses LESS for variables and mixins:

// config.testql.less
@api_url: http://localhost:8101;
@timeout_ms: 30000;

.api-test(@method, @endpoint) {
  method: @method;
  endpoint: @endpoint;
  expect_status: 200;
}

CLI Options

testql <file> [options]

Options:
  --url <url>         Base URL for API tests (default: http://localhost:8101)
  --dry-run           Parse and validate only, don't execute
  --output <format>   Output format: text (default), json, junit
  --verbose           Enable verbose logging
  --timeout <ms>      Default timeout for operations

Generate from Topology

Generate executable scenarios from discovered topology paths:

# Generate TestTOON from the first topology trace
testql generate-topology ./project

# Generate IR JSON from a specific trace
testql generate-topology ./project --trace-id trace.001 --format ir-json

# Write to file with live network scanning
testql generate-topology ./project --scan-network -o scenario.testql.toon.yaml

Testing

# Run all tests
pytest

# Run with coverage
pytest --cov=testql

# Run specific test file
pytest tests/test_runner.py -v

Example Scenario

# scenarios/tests/test-api.testql.toon.yaml
# SCENARIO: Device API Test
# TYPE: api
# VERSION: 1.0

CONFIG[1]{key, value}:
  base_url,  http://localhost:8101

API[2]{method, endpoint, status}:
  GET,   /api/v3/data/devices,   200
  POST,  /api/v3/scenarios,      201

ASSERT[2]{field, op, expected}:
  data.length,  >,   0
  data.id,      !=,  null

Real-world deployment — c2004 healing pipeline

Since May 2026 TestQL runs as the active-probing component of the c2004 fleet management monorepo's self-healing pipeline. It's the canonical, production-grade reference for how to wire testql run into Prometheus + Alertmanager + an LLM-agnostic ticket layer.

What c2004 uses TestQL for

Pipeline shape

testql-watchdog (loop)
    │ runs `testql run realtime-health.testql.toon.yaml --output json`
    ▼
50 assertions execute → exposes testql_scenario_pass_total / fail_total
    │ on failure: POST http://healing-webhook:8810/probe-failure
    ▼
healing-webhook → planfile ticket create --label llm-ready
    │
    ▼
LLM agent (Windsurf/Cursor/Claude Code/aider) reads
  `planfile ticket show PLF-XXX` and proposes a fix

Operator flow

Note: The observability stack (including testql-watchdog) does not start automatically with make dev / make run. Those commands only start the application processes (Node.js + Python). You must launch the monitoring stack separately.

# 1. Terminal A — start the application (no monitoring)
make dev                        # starts frontend + backend + encoder

# 2. Terminal B — start observability + self-healing (separate step)
task monitor:up                 # brings up testql-watchdog + 10 other containers

# 3. Verify watchdog is running
task monitor:probe              # ad-hoc run of the scenario (50 assertions)
docker logs c2004-testql-watchdog | tail
# → cycle done exit=0 pass=50 fail=0

# 4. View Prometheus metrics
open http://localhost:9101/metrics   # testql_scenario_pass_total / fail_total

Key commands:

• task monitor:up — start full observability stack
• task monitor:core — start core stack (lighter, without Loki/Blackbox)
• task monitor:down — stop all observability containers
• task monitor:logs:watchdog — tail testql-watchdog logs

Why TestQL was chosen for this role

1. Black-box + business-logic — pinging /health is not enough; TestQL exercises real endpoints (/api/v3/devices, /api/v3/menu/configurations, etc.) with full assertion blocks.
2. TOON format — the scenario file is human-editable and ~40% smaller than the equivalent JSON when sent to an LLM for diagnosis.
3. Single binary — pip install testql and the watchdog container is ~80 MB.
4. JSON output mode — --output json makes Prometheus metric extraction a one-liner in watchdog.py.

See also

• c2004 master doc: docs/planfile-llm-guide.md
• c2004 README section: "🩺 Real-time Monitoring + Planfile Self-Healing Pipeline"
• The actual scenario: testql-testing/scenarios/realtime-health.testql.toon.yaml
• The watchdog: monitoring/testql-watchdog/

Examples

Documentation

• TestQL Specification — Complete language reference
• Desktop GUI E2E — Native Linux desktop testing (vdisplay, img2nl, imgl)
• Desktop examples — Scenario index and run-all script
• Recipes — Common testing patterns and examples

License

Licensed under Apache-2.0.