aitlc

A CLI for debugging Behave + Playwright test suites and keeping them in sync with Xray. Structured JSON output first, so a person and an agent can read the same result.

It never asks you to edit the suite it debugs. No hook blocks, no environment.py changes, nothing to keep in sync — instrumentation attaches through behave's own runner API, with a fallback for versions that lack it.

uv tool install aitlc      # or: pipx install aitlc
cd /path/to/your/project
aitlc init                 # detects your layout, writes aitlc.toml
aitlc run PROJ-1234

The command is always aitlc. It is published under two names — aitlc and dax-aitlc — which install the same tool; use whichever your org prefers.

What it does

Keep one investigation in one directory. Point a workspace at what you are working on and every artifact from every command lands there — traces, cached CI reports, session state, browser profiles, logs.

aitlc --workspace PROJ-29019 debug start PROJ-29019

Switch the name and the previous investigation stays beside it; delete the directory and it all goes together. Unset, everything stays under reports/.

Reproduce a CI failure without editing the suite. Park on any step, walk forward one at a time, and never comment a line out to get there.

aitlc preflight PROJ-1234              # would this run here like it does in CI?
aitlc -w PROJ-1234 debug start PROJ-1234 --at 12
aitlc -w PROJ-1234 debug next PROJ-1234

Steps run inside a real, paused behave process holding your suite's real Context, so before_scenario, before_step and after_step all fire, Examples rows and data tables are behave's own, and run-scoped data is minted once rather than regenerated per step. The debug engine does not re-implement behave's loop — it single-steps the real one.

Run and target tests. Bare test IDs resolve recursively, so you never need a full path, and never need to tag other features to narrow a run. One Examples row of a Scenario Outline can be run on its own.

aitlc run PROJ-1234                 # structured JSON result
aitlc run PROJ-1234 --debug         # a real failure parks into a live session, not a dead end
aitlc parallel run -j 4             # concurrent, without editing tags
aitlc parallel focus PROJ-1234      # pin a selection, then just `aitlc parallel run`

--debug runs the same engine debug start uses: on the first real failure the process parks into a live, socket-served session instead of exiting, so aitlc debug retry PROJ-1234 continues it immediately — no restart, no guessed step index, even for the very first failure.

Debug a failure end to end. A session holds one isolated browser and your position in the scenario, so fixing a step costs a re-run of that step rather than the whole scenario — and a scenario with four defects costs one setup instead of four.

aitlc s3 triage-run --suite <plan>       # what CI actually failed on
aitlc debug start PROJ-1234 --at 12      # isolated browser, driven to the step
aitlc debug retry PROJ-1234              # edit -> re-run that step -> repeat
aitlc debug next PROJ-1234               # forward, from the state you have
aitlc debug certify PROJ-1234 --times 2  # fresh instance, real feature, twice
aitlc debug stop PROJ-1234               # browser down; no cleanup hooks fired
aitlc debug stop --all                   # every leftover session; Pulse Chrome stays up
aitlc debug stop PROJ-1234 --cleanup     # + fires the suite's real after_scenario/after_feature first

stop --cleanup is opt-in, never the default: the debug browser is often the persistent one run --debug reuses across invocations, and a suite's own cleanup can include a logout — reach for it only when you deliberately want a clean handoff.

retry and next drive one paused behave process rather than spawning a new one each time. That is a correctness feature first: a process per step regenerates run-scoped data — generated names, ids, emails — so a step waiting on something an earlier step created polls forever for a name that never existed, which looks exactly like the application hanging. One paused process means one set of that data, and retry re-runs the real behave step (its table, docstring and Examples binding intact) after reloading your edit.

Edit and step, don't restart. When a step fails, edit it — the Gherkin line and its params, or the Python behind it — and just run retry. Before every retry/next the gate re-parses the feature (mtime-gated, so an unedited step pays nothing) and reloads your step modules, so the edit runs against the browser, login and setup you already paid for. Editing Gherkin is now as cheap as editing Python; you only need a fresh debug start for a change to a setup step before your park point.

aitlc debug run-line PROJ-1234 42       # run the step at that file line, no retyping/escaping
aitlc debug screenshot PROJ-1234        # this session's page, no --cdp-url to look up
aitlc debug inspect PROJ-1234           # same, for the accessibility tree

Every failed step already carries a real Python traceback, no setup required. next/retry/continue's reply includes failed_at (file/line/function) and the full traceback — read from the raw exception behave stores on every failed step, not from error_message (which, for a plain assert failure — the most common case — carries no file/line at all unless behave itself runs --verbose). Try this before reaching for a breakpoint: it answers "where and why" for free; a breakpoint is for when you need an actual variable's value, which a traceback can't show.

Drop a real breakpoint(), not just a JS eval. A paused session's eval runs JavaScript on the live page — useful, but blind to the Python side of a failing step. A code-level breakpoint() anywhere in project code under debug/run --debug parks on its own socket instead of hanging (stdin is closed in the gate subprocess, so a bare pdb prompt has nowhere to go):

aitlc debug status PROJ-1234            # "paused_at": "breakpoint" when one is hit
aitlc debug py PROJ-1234 "some_local_var"    # evaluate Python in the paused frame's own scope
aitlc debug resume PROJ-1234            # continue exactly where it stopped — nothing restarted

next/retry on the same session simply wait until resume is sent — the main gate socket stays busy serving the very step that hit the breakpoint.

Find out what CI did, without guessing which report to open.

aitlc s3 verify-test PROJ-1 PROJ-2   # pass/fail + the failing step and error
aitlc s3 history PROJ-1 --days 14    # chronic, intermittent, or an outage day

A test key is usually a scenario tag inside a differently-named file, not an execution key, so a filename-only lookup reports "did not run" for a test that ran. history groups failures by signature: one signature every time means reproduce it, varying signatures mean establish a base rate before bisecting.

certify is deliberately separate and never uses the debug browser: a CDP-attached browser reuses an existing context, so it is never proof. Two consecutive passes are the default because one pass does not disprove a race.

Read back what already happened. Every run is recorded, and fetched reports are cached, so a follow-up question is a file read rather than another run.

aitlc journal list --last 5
aitlc journal diff <earlier> <later>     # did the fix work, or was that luck?

Payloads are redacted before they touch disk, size-capped and pruned. debug next/retry/continue/run-text/run-line journal themselves too, each entry shaped exactly like a step out of a plain run's own steps list (keyword, step text, status, duration, error) — so journal list is one place to look regardless of whether a test ran end to end or one step at a time.

Check locator hygiene. aitlc locators lint flags selectors that pass while reading the wrong element — positional row indices, grid cells with no role='cell' guard (a header carries data-field too), unanchored //* xpaths — each with the rewrite attached, not just the diagnosis.

Debug live. Keep one browser across many iterations instead of paying setup and login on every change. aitlc steps run --range 14-19 resumes a scenario partway through in an already-open browser — replacing the habit of commenting out the steps that already passed.

aitlc cdp launch                    # detached; survives the shell that started it
aitlc cdp launch --new              # isolated: own port + own profile
aitlc steps run PROJ-1234 --range 14-19 --cdp-url http://127.0.0.1:9333

Read a page as text, not pixels. The accessibility tree answers "is X on screen" as assertable text, and carries nesting, control state and field values a screenshot cannot express. Measured on one real page: 55 KB screenshot → 1,961 characters for the full tree → 20 characters for a targeted query.

aitlc cdp inspect --cdp-url http://127.0.0.1:9333 --a11y --a11y-query "Save"

Find dead step definitions. behave has no equivalent of Cucumber's unused-step report. Matching goes through behave's own registry, so the answer agrees with what the runner would dispatch, and steps invoked via context.execute_steps(...) count as used.

aitlc steps unused

Track real flakiness. Signature matching only ever covers flakes somebody already described. aitlc history records every run outcome, so a new flake is visible the second time it happens. A test that has only ever failed is reported as broken rather than flaky — retrying it spends time to reach the same answer.

Sync with Xray. Read, compare and write a Test's Gherkin; pull every feature from a Test Execution or Plan; find where a step is really used.

Escape hatches. aitlc behave, aitlc pw and aitlc paver run those tools directly with your project's .env and interpreter already set up, so adopting aitlc never means losing a flag it does not wrap — aitlc paver run parallel --local is your suite's own paver task, unchanged. --print-command shows the exact invocation without running it.

Get the same environment into your own shell. aitlc's own commands load .env and resolve the interpreter for you — but sometimes you need it in the shell itself (a raw poetry run behave, a one-off script, step_repl.py):

source "$(aitlc env)"    # activates the venv, exports every .env key

One command, and nothing sensitive touches stdout: it writes the real export ... lines to an owner-only file and prints just the path, so the command is safe even through a logged/captured shell.

Reuse a live browser instead of paying setup every run. Launch one debug Chrome and every behave-based command attaches to it instead of starting fresh:

aitlc cdp launch                    # detached CDP browser, survives the shell
aitlc run PROJ-1234 --cdp           # opt in to attach to it (fresh browser by default)
aitlc paver run parallel --local    # reuses the same browser automatically

aitlc sets the suite's CDP env var (default PLAYWRIGHT_CDP_URL, configurable via [project].playwright_cdp_env) so the suite connects to the open browser. Pass --cdp-url (run) or --aitlc-cdp-url (paver/behave) to point at a specific endpoint.

Reusing a browser means reusing whatever state it was left in — a plain aitlc run against an instance something already drove warns rather than failing silently mid-scenario (a leftover login, say, skipping past a step that expected a fresh one). debug list shows every tracked session across more than one live browser; --prune drops the bookkeeping for any whose gate process is no longer actually running, without ever touching a browser still in use. debug start now also stops any PREVIOUS session for the same test_id before launching a new one — it used to just overwrite the record, orphaning whatever was already running under it. debug reap cleans up anything already orphaned before that fix (or from a crash, a killed shell) by scanning real processes, not session records.

--extra-tag skip_login (on run, debug start and debug restart) is the generic fix for that warning: it adds the tag onto feature.tags/ scenario.tags before hooks run, so a project's own tag-driven logic treats it exactly as if it were physically in the file — no editing required. debug restart re-runs a scenario from step 0 on the SAME browser (stop, browser-preserving, then start again); debug jump <line> moves the cursor to a file line with no execution, for when the browser no longer matches where the session thinks it is; debug continue --from <line> does that jump and then keeps going.

Documentation

Requirements

Python 3.10+. The suite under test keeps its own environment; aitlc runs outside it and shells in, so the two never need to share dependencies.

Design notes

Three rules the code holds to, each learned from a bug that cost real time:

• Never report success for something that did not happen. Missing setup, an incomplete feature corpus, or an instrumentation fallback are each reported explicitly, because silence surfaces later as an unrelated-looking failure.
• Ask the tool, do not assume its version. behave's custom-runner option changed both its name and its argument format across releases, so aitlc probes behave --help rather than parsing a version string.
• Nothing project-specific in the code. Layout, hook names and credential variable names all come from aitlc.toml, and aitlc init detects them.

Development

python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
.venv/bin/pytest tests/ -q

License

MIT — see LICENSE.