Testence

Agent-first browser testing with evidence-backed verdicts.
Agents author and investigate. A deterministic runner proves what happened.

English docs · Документация на русском · Demo contract · Benchmark protocol

The machine-readable support matrix is the source of truth for the declared Python, operating-system and dependency floors. Hosted receipts for the exact clean candidate are still required before a platform is called release-verified.

Testence turns a product claim into a reviewable PlanSpec, deterministic browser test, structured evidence pack and typed verdict. Routine replay contains no LLM call and no provider lock-in. The agent does the work that benefits from reasoning; the runner does the work that must be fast and reproducible.

feature or risk
      │
      ▼
agent plans ──► writes ordinary pytest ──► deterministic replay
                                                   │
                                                   ▼
human/policy ◄── reviewable repair ◄── evidence pack + verdict

Why Testence

The key distinction is not “AI writes tests.” Many tools can do that. Testence makes the complete agent workflow auditable: plan, claim, evidence, verdict and repair are explicit contracts that another agent, a reviewer or policy can verify.

Quick start

git clone git@github.com:kiselas/testence.git
cd testence
uv sync --locked --extra dev --extra parallel
uv run playwright install chromium

uv run testence plan validate examples/specs/target-page.md --json
uv run testence plan prepare examples/specs/target-page.md --project . --json
uv run pytest examples -q --testence-headless

plan prepare is the fail-fast gate before browser discovery. It checks each scenario's engine capabilities, required oracle adapters and project prerequisites once, then returns ready or explicit blockers with phase timings. Configure reusable, read-only checks in testence.json:

{
  "project_id": "shop",
  "base_url": "https://qa.example.test",
  "readiness": {
    "schema": "testence/readiness/1",
    "oracle_adapters": ["api", "custom"],
    "checks": [
      {"id": "credentials", "type": "env", "variables": ["TESTENCE_USER", "TESTENCE_PASSWORD"]},
      {
        "id": "seed",
        "type": "http",
        "path": "/api/qa/seed",
        "json_pointer": "/ready",
        "equals": true,
        "fix": {"argv": ["python", "scripts/seed_qa.py"], "timeout_ms": 120000}
      }
    ],
    "scenarios": {"checkout": ["credentials", "seed"]}
  }
}

Supported checks are env, project-relative file, and read-only http. HTTP checks may assert a status and a JSON Pointer value. Secret values are never included in the report. A blocked result exits with status 3; invalid plans or configuration exit with 2. After reviewing a blocked report, plan prepare --apply-fixes runs only the explicitly configured argument arrays without a shell and checks the failed prerequisites again. Command output is discarded from the receipt so credentials cannot leak through tool logs.

For a wheel-only onboarding check in a new or existing project:

testence doctor --json
testence init . --json
testence run --project . --run-id r-onboarding
testence inspect runs/r-onboarding --json

Or run the complete portable green/failure demo and render both local reports with one command:

testence demo run --project testence-demo --json

The scaffold lives under .testence/ and is not added to the existing pytest suite; the default testence run selects it explicitly. Every generated file is bound by a scaffold manifest, and conflicting user files are left untouched.

Every failed test produces a bounded evidence pack containing the relevant UI state, network and console signals, intent-bearing steps, independent oracle observations and a verdict template. Render a standalone report with:

uv run testence report runs/<run-id>

To integrate a real application, define a target profile in testence.json, keep credentials in environment variables or .env.local, and run ordinary pytest:

TESTENCE_PROFILE=staging uv run pytest tests_e2e -q

Follow Testing a UI feature for the full workflow.

Fast agent authoring

Keep the authenticated browser alive and attach short runs over CDP:

# terminal 1
uv run python -m testence.dev_browser --profile staging

# terminal 2: rerun on save without restarting Python or pytest
uv run testence watch --warm \
  -w tests_e2e -w src -- \
  python -m pytest tests_e2e/test_widget.py -k created_widget \
  --testence-profile staging-attached -q

Warm mode retains the Playwright/CDP engine connection while creating a new pytest session, fixtures, auth context, run id and evidence writer on every iteration. Fresh processes remain the required path for CI and release validation.

On the maintained production-built React profile, warm engine reuse reduced bootstrap p50 from 2,808 ms to 447 ms (−84.1%) and whole-run p50 from 3,447 ms to 1,133 ms (−67.1%). See the result snapshot, machine-readable budget and ADR-0018. Numbers are local engineering evidence, not a cross-machine performance promise.

Evidence, not runtime magic

A verdict is a versioned document bound to the plan, claims and captured run:

{
  "schema": "testence/verdict/2",
  "project_id": "widgets",
  "case_id": "create-widget",
  "variant_id": "default",
  "attempt_id": "attempt-controller-1",
  "run_id": "r-20260906-120000-abc123",
  "proof_id": "proof-0123456789abcdef0123",
  "plan_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "test_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
  "policy_digest": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
  "pack_digest": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
  "plan_id": "widgets.create",
  "verdict": "real_bug",
  "test_id": "test_created_widget_is_visible",
  "confidence": 0.96,
  "summary": "POST succeeded, but the new row never appeared in the UI",
  "claim_results": [
    {
      "claim_id": "widgets.create.persisted",
      "status": "failed",
      "reason": "The persisted row is absent from the rendered collection",
      "evidence": ["oracle.json#/0"]
    }
  ]
}

When evidence is insufficient, the agent must abstain and state what is missing. Locator healing follows the same rule: it creates a proposed patch with provenance and confidence; it never silently changes the selector during execution.

Agent-native by design

The package ships portable skills for four bounded jobs:

• testence-plan — turn a feature or risk into claims and a PlanSpec;
• testence-author — create deterministic tests and independent oracles;
• testence-triage — classify failures from bounded evidence;
• testence-repair — propose a reviewable change and prove it safely.

The same versioned skill pack is designed for Codex/ChatGPT, Claude Code, OpenCode and other Agent Skills clients. Client adapters stay thin; the contracts remain portable. Start from AGENTS.md, the shared entry point for any coding agent. Optional testence[visual] adds digest-pinned viewport comparisons and manifested expected/actual/diff evidence. The client simulation exercises two adapter layouts from an installed wheel, including visual defects and harmless controls; it does not claim independent model/client acceptance. See Agent Skills and the agent workflow.

Architecture

flowchart LR
    A[Agent control plane<br/>plan · author · triage · repair]
    V[Verification plane<br/>pytest · DSL · Playwright/CDP]
    T[Trust plane<br/>ledger · evidence · verdict · policy]
    A -->|versioned contracts| V
    V -->|append-only events| T
    T -->|reviewable proposal| A

• src/testence/contracts — PlanSpec and verdict schemas with claim traceability;
• src/testence/engine — replaceable execution protocol and Playwright/CDP backend;
• src/testence/dsl — intent-bearing actions and exact-by-default assertions;
• src/testence/evidence — append-only testence/2 ledger and bounded artifacts;
• src/testence/triage — evidence packs, verdicts and reviewable healing proposals;
• src/testence/agent — packaged, versioned, client-neutral skills;
• bench and corpus — real-React latency gates and seeded correctness defects.

Important decisions are recorded as bilingual ADRs with measurable tripwires: English · Русский.

Proof before claims

Testence keeps benchmark inputs, budgets and raw result snapshots in the repository. The current proof surface includes:

• a production-built React latency gate for navigation, controlled inputs and mutation synchronization;
• a shared-browser and warm-runner benchmark for the authoring loop;
• a seeded mutation corpus for false-green, false-red and healing quality;
• a competitive replay protocol with frozen scenarios and environment disclosure.

Start with the launch protocol and benchmark corpus. A number without its command, environment and failure criteria is deliberately not treated as a product claim.

Project status

Testence is an alpha release candidate. PlanSpec propagation, verdict validation, portable skills, deterministic execution and evidence packs are implemented. Public agent bootstrap and managed verdict submission are implemented through the CLI; independent client and public release acceptance are pending. APIs may change.

Do not use the current preview against sensitive production data. Systematic evidence redaction and the remaining security review are release gates. API credentials are origin-bound and the opt-in session cache has TTL and identity checks, but the complete canary/path/retention matrix from the release specification is not yet accepted. See SECURITY.md, CONTRIBUTING.md and the roadmap.

License

Apache-2.0. Runtime dependencies are restricted to permissive licenses. The runner contains no LLM SDK and makes no call to a model provider during ordinary replay.