ForgeQE

Test agent CLI for web, Windows desktop and mobile apps. Say what you want in plain language: a goal to carry out in a browser, a Windows app or an Android or iOS app, test cases to run from Jira, Azure DevOps, Xray or a file, a test to write, or a CI pipeline to set up. ForgeQE's agent works out what that needs and calls ForgeQE's tools. The ForgeQE engine drives Chrome locally, or BrowserStack and TestMu AI/LambdaTest through their Chrome DevTools Protocol (CDP) endpoints, and a run that passes becomes a test in your repo's own language and framework, written by Scribe, ForgeQE's coding agent. Skills (SKILL.md folders, built in or in .forgeqe/skills) teach the agents your repo's conventions and the apps you test.

The engine is the browser service from test-agent-nexus, copied into this package (py/forgeqe_engine). It keeps the nexus behaviour: error-recovery system prompt, fast-mode agent tuning, Chrome launch flags, the search/select loop breaker, the 600 s run timeout, compact workflow YAML with semantic locators. There is no dependency on test-agent-nexus, FastAPI, or a database.

The shell is OpenTUI. forgeqe run does the same without the shell, and forgeqe suite is the fixed command pipelines call. See HOW_TO_USE.md, the interactive architecture page (forgeqe docs --open), and docs/ARCHITECTURE.md.

Install

The CLI is published on npm as @capagents/forgeqe. PyPI package forgeqe-cli is a pip launcher for the same forgeqe command. Either one is the only thing to install, on Windows, macOS or Linux:

npm install -g @capagents/forgeqe
# or
pip install forgeqe-cli

forgeqe setup   # optional: the first live run does this too

Everything else is downloaded on first use into ~/.forgeqe/tools (%USERPROFILE%\.forgeqe\tools on Windows), checksum verified, without admin rights and without touching PATH or shell profiles:

• Bun (the CLI and the OpenTUI shell run on it). A Bun 1.3+ already on PATH or in ~/.bun/bin is used instead. FORGEQE_BUN points at a specific one.
• The agent runtime that ForgeQE's agent and Scribe, the coding agent, run on, at the version ForgeQE is tested with. code.runtime points at a specific binary.
• uv, which then fetches Python for the engine. A uv already installed is used instead; when uv cannot be downloaded, a Python 3.11+ on PATH (or the Windows py launcher) is the fallback.
• The ForgeQE engine and its Python libraries in ~/.forgeqe/engine/.venv. Installed Google Chrome is used, or Chromium is installed when Chrome is missing. FORGEQE_PYTHON points at an existing Python that already has the engine's libraries; when that Python cannot run the engine (missing, too old, or without them), ForgeQE says so and uses its own environment. A broken environment is rebuilt on the next run.

forgeqe setup does all of this up front; otherwise the first command that needs a piece fetches it. Upgrading is forgeqe update: it updates every forgeqe on PATH (bun, npm and pip installs) and waits while the registries catch up with a new release. A release that pins a newer Bun, agent runtime, uv or engine library downloads it on the next run and removes the old one. Deleting ~/.forgeqe removes everything ForgeQE downloaded. Behind a proxy, set HTTPS_PROXY. The npm launcher's Bun download also needs NODE_USE_ENV_PROXY=1 (Node 24+); on older Node, use the pip launcher or install Bun yourself.

From this repo:

cd ForgeQE
bun install
bun src/index.ts init
bun src/index.ts start

One-shot release (same version on both registries; assumes npm and twine are already logged in):

./scripts/publish.sh           # current version
./scripts/publish.sh 0.2.0     # bump + publish
./scripts/publish.sh --dry-run # pack only

Privacy

ForgeQE sends no telemetry and turns it off in everything it starts:

• The engine: PostHog telemetry, cloud sync, and the price-list download are off, and the blank-tab logo its library loads from a CDN is replaced by ForgeQE's start page, which loads nothing.
• The agent runtime: auto-update, session sharing, OpenTelemetry spans, and the model catalogue download are off; it uses its bundled model list.
• The repo's test command: DO_NOT_TRACK=1, plus the opt-outs for Next.js, Nuxt, Astro, Gatsby, Storybook, Turborepo, Angular, .NET, and Cypress.

OTEL_EXPORTER_OTLP_* variables are removed from those processes, and your environment cannot turn any of this back on. The only traffic is to your model endpoint and the sites the agent visits. The engine also downloads its ad-block and cookie-banner extensions from the Chrome Web Store once.

Configure

init writes:

ForgeQE also keeps these next to forgeqe.yaml as it works:

An unchanged copy keeps following ForgeQE's updates; once you edit it, yours wins.

For remote browsers, forgeqe.yaml can now point at provider-specific YAML files under browser.providers such as browserstack and testmu, so the BrowserStack or TestMu AI connection details do not have to live inline under browser.remote. forgeqe init drops sample files at .forgeqe/browserstack.yaml and .forgeqe/testmu.yaml. When both are present, the separate file supplies the provider defaults and the inline browser.remote values override only the active provider. ForgeQE's agent can also switch providers per request by passing browser_provider to browser_run or run_test_cases, and the CLI can pin one with --browser-provider. Devices. A provider file can list named devices under devices:, each the file's shared remote: settings with its own browser, version, and platform on top. In the shell, describe the devices you want ("add Chrome on Windows 11 and Chrome 128 on Windows 10 on TestMu") and ForgeQE's agent saves them there with the browser_devices tool, keeping the file's comments; then "run SHOP-12 on both" runs the case on every device at the same time, each with its own run folder and row in the report (SHOP-12@win11-chrome). From the command line, forgeqe suite <source> --devices win11-chrome,win10-edge does the same, and forgeqe start --devices ... pins devices for a shell session. Devices run side by side, up to eight at once unless --parallel or suite.parallel says more; the grid queues sessions past your plan's limit. Each case posts one combined result back to its test system, failed when any device failed, and gets one test, written from its run on the first device. ForgeQE drives remote browsers over CDP, so remote.browser must be Chrome or Edge (TestMu AI's CDP endpoint serves Chrome only: an Edge session there never opens), and the default endpoints are the providers' CDP (/puppeteer) ones. On a remote grid, the browser stops taking real clicks and keystrokes on a tab once a password form has been submitted (most likely the browser's password manager), so after a sign-in leaves the login page ForgeQE reopens the page it landed on in a fresh tab, with the session intact, and the run log says so.

forgeqe models
forgeqe run "Read the homepage of example.com" --plain

The active profile is llm.active in forgeqe.yaml, else default in llms.json, else the first profile; Tab switches in the shell. With none set up, ForgeQE says so and how to add one.

Model router. With more than one model in llms.json, llm.routing: auto gives each area of work (ForgeQE's agent, browser runs, coding, review, analysis, planning) the cheapest configured model that is good enough for it. ForgeQE checks each model once and rates it, and keeps the result in .forgeqe/model-profile.json. forgeqe models --routes shows the model each area gets, and why. --profile and code.model still win. See docs/MODEL_ROUTING.md.

GitHub Copilot. With Copilot licences, a github_copilot profile runs ForgeQE's agent and Scribe on the Copilot seat through the agent runtime's GitHub Copilot sign-in (forgeqe auth login copilot, GitHub.com or GitHub Enterprise). The browser engine uses GitHub Models with your GitHub sign-in (forgeqe auth login github, or GITHUB_TOKEN), optionally billed to an organization with github_org. See HOW_TO_USE.md, GitHub Copilot seats as the model.

CLI

forgeqe start
forgeqe start -g "Go to booking.com and search hotels in Mumbai" -p azure-gpt4o

forgeqe run "Get a quote on example.com for a family of four" --profile azure-gpt4o --headed
forgeqe run "Run SHOP-12 and SHOP-14, 2 browsers at once"
forgeqe run "Write a Playwright test for the login flow on staging.app.test" --repo ../app
forgeqe run "Test the refund rules on https://acme.atlassian.net/wiki/spaces/QA/pages/123"
forgeqe login shop                      # sign in by hand once; runs reuse the session
forgeqe run --direct "Read the homepage" --url https://example.com --out ./.forgeqe/reports/demo   # one browser goal, no agent

There are no commands to pick a source or a mode. ForgeQE's agent reads the request and decides: a browser goal, test cases (Jira keys, Azure DevOps ids or test plans, Xray, TestRail, Zephyr Scale or qTest cases, files in the working directory, or cases pasted into the message), requirements to turn into cases (a Confluence page, a story), a test, or a pipeline. When a reference could mean more than one thing, it asks. The agent picks the site to open; --url only pins a start page.

Tools per turn. ForgeQE's agent does not carry every tool on every turn. Each turn gets the core (help, setup, the environments) and the capabilities the request is about, picked from its words, the apps it names and its links, plus those used in the last few turns. When it needs another, it calls use_capabilities and carries on with the same request. Fewer tools mean a smaller prompt and no mix-ups between similar ones (browser_run, desktop_explore, mobile_explore). agent.capabilities.always and off in forgeqe.yaml pin or remove capabilities, and FORGEQE_TOOL_SCOPE=off gives every tool. See HOW_TO_USE.md.

Inside the shell, type the request and press Enter (Ctrl+J starts a new line). The only commands are for the session: /stop, /rerun, /headed, /details, /model, /profile, /env, /logs, /features, /generated, /init, /skills, /db, /report, /auth, /clear, /help, /exit (/help says what each does). Clicking the model ▾ in the status line opens the model picker, as /model and Tab do. Everything the other commands do can be asked for in the shell too, since each one is a tool the agent calls: "heal the checkout tests", "check the smoke cases once", "monitor them every 10 minutes", "sign in to shop", "forget the shop session", "which profiles do I have?". While the agents work, the shell shows each step's result, the files the coding agent writes, test results, and problems, with what is happening now on the activity line; /details on (or --verbose) adds the browser agent's reasoning, page memory, engine logs, and every file the coding agent reads. Results appear as cards in the transcript, and when a step needs you (signing in by hand), a Your turn prompt waits until you pick Done, continue or Cancel.

ForgeQE's agent needs the agent runtime (downloaded automatically, see Install) and a live profile. It uses the active llms.json profile, or the model the router picks for its area when llm.routing is auto; --profile and code.model: provider/model still win. Without the agent runtime, the request runs as one browser goal.

forgeqe run exits 0 when the job is done, 1 when it failed, 2 when it was stopped, and 3 when the agent needs more information.

forgeqe --help lists every command, among them forgeqe db, forgeqe data and forgeqe flows (see Databases, test data, and data flows) and forgeqe generated, which says how much of the code ForgeQE wrote and how much is still there (--json for a pipeline).

Page memory

Every live run records each page it visits in .forgeqe/memory next to forgeqe.yaml (~/.forgeqe/memory when there is no config file): every interactive element, with all of its locators (test id, id, role and name, label, placeholder, alt, title, href, text, CSS, XPath), a confidence % for each, and when it was first seen, last seen, and last used. Confidence rises when a locator keeps matching exactly one element and works when used, and falls when it goes missing or fails.

A goal that passed before is replayed from memory with no model calls. Each step finds its element by the highest-confidence locator that still matches, so a renamed button or a changed id heals itself. The agent then checks the result with one model call (replay: verify), or the run finishes without the model (replay: trust). If a step cannot be found, the agent takes over from that point. A run that failed because an expected result did not hold keeps the steps up to that check, so the next run of the case replays them and the agent checks the result again there, instead of exploring the whole case again. On every page memory knows, the agent is told which controls earlier runs used, with the index each has on the page right now, so it acts on them in one step. Generated specs use the best locator from memory.

forgeqe memory                          # hosts, pages, elements, flows
forgeqe memory show booking.com         # a host's pages
forgeqe memory show https://app.test/login --locators   # every locator with confidence
forgeqe memory flows --steps            # recorded goals and their steps
forgeqe memory clear booking.com --yes
forgeqe memory stats                    # how often memory stood in for the model, per area
forgeqe run "..." --replay trust       # or --replay off, --no-memory

A new goal on a site ForgeQE knows replays the start it shares with an earlier goal there: a sign-in with a saved login, a cookie banner, and the steps whose controls or values the new goal names. Mobile and desktop explores do the same from app memory. A goal that passed before replays with no model call when its checks still hold, a failed explore is picked up where it went wrong, and a control a heal found once is used again without the model. Saved API calls replay inside a browser flow when the server answers as before, and API scenarios planned from the same test cases or text are reused. A rerun of any of these runs as recorded, without ForgeQE's agent.

Skills

Skills teach the agents what a prompt can't know: your repo's test conventions, a framework, or how to drive an app. A skill is a folder with a SKILL.md (the Agent Skills format: name, description, then instructions), plus ForgeQE's agents: [forgeqe, scribe, browser] to say who uses it. ForgeQE ships playwright-typescript, dotnet-nunit-playwright and testcontainers for Scribe, dynamics-365 for the browser agent, and writing-skills for ForgeQE's agent. Your own go in .forgeqe/skills next to forgeqe.yaml (commit them) or ~/.forgeqe/skills; a project skill replaces a built-in one with the same name. Skills in ~/.claude/skills and ~/.agents/skills are used only with skills.external: true.

Each agent sees the skills meant for it and loads the ones whose description fits the job. For browser runs, ForgeQE's agent passes the skills a goal needs; in forgeqe suite, monitor and run --direct the engine picks them from the goal once, and --skill NAME (or --skill none) fixes the list. Reports and run.json show which skills a run used.

forgeqe skills                                   # who can use what, and where each skill comes from
forgeqe skills new contoso-grid --agent browser --description "Use when the site is the Contoso ERP (erp.contoso.com)"
forgeqe skills show contoso-grid
forgeqe skills check                             # validate every skill
forgeqe skills disable dynamics-365              # edits skills.disable, keeping comments

Or ask in the shell: "make a skill for our Dynamics date picker".

Tests

Point ForgeQE at a repo and a run that passes becomes a test in that repo. The agent gets the run's steps and the locators and confidence from page memory. It studies the repo's framework, page objects, fixtures, and helpers, reuses them, and writes the test in the repo's language and test framework (C# NUnit, TypeScript, Python, Java, …). The browser agent records what it checked with the exact text on the page, and the test asserts those checks with locators the Playwright replay found. ForgeQE runs the test command and hands any failure back (the error first, the page dump saved to a file), until the test passes or the attempts run out. What Scribe learned about the repo is kept, so the next job there starts from its notes instead of reading the repo again. The test counts only when ForgeQE's own run of it passes.

forgeqe run "Log in and open the invoices page on staging.app.test" --repo ../app
forgeqe code --repo ../app          # write a test for the latest run
forgeqe code .forgeqe/reports/20260928-172700 --repo ../app --attempts 3

In the shell and forgeqe run, every passed run becomes a test unless you say otherwise: in --repo, in code.repo from forgeqe.yaml, or else in the current folder. --no-code turns tests off. The transcript shows the handover to Scribe, the coding agent, the files it reads and writes, and ForgeQE's run of the test, and each request ends with a card listing the run's report and the test with its result (or why no test was written).

Fixing broken tests

When the app changes, tests break. forgeqe heal runs the repo's tests, and for each failing test the agent re-runs its flow in the real browser. If the flow still works, the test is out of date: the agent updates its steps and locators (in the page object, when that's where they live) from what the browser just did and from page memory. If the browser fails at the same step, the app is broken there: the test is left alone and the failure is reported as an app bug. Tests are never deleted, skipped, loosened, or given longer timeouts to make them pass.

ForgeQE re-runs every healed test and then the whole command itself; a test counts as healed only when ForgeQE's own run of it passes.

forgeqe heal --repo ../app                              # code.test_command, or the agent finds the command
forgeqe heal "npx playwright test tests/checkout" --repo ../app
forgeqe heal -- npx playwright test --project=chromium
forgeqe run "The login tests broke after the redesign, fix them" --repo ../app   # the agent heals them the same way

The report is heal.md (and heal.json) in .forgeqe/reports/heal/<time>/, with each test's verdict, the files changed, the browser runs, and the test output before and after. forgeqe heal exits 0 when the tests pass (or nothing needed healing), 1 when some still fail, and 2 when stopped.

Running existing tests

"Run all tests related to login" or "run the tests added this sprint" finds the repo's existing tests and runs them once you agree. Scribe reads the tests (and git history for a date range or sprint) and decides each one from what it does, not from its name. ForgeQE runs the framework's list command to confirm the selection and shows the list. Nothing runs until you pick Run, and then ForgeQE runs exactly those tests and shows a pass or fail for each. For a sprint, ForgeQE fetches the sprints from Azure DevOps and Jira and asks which one you mean.

forgeqe run "Run all test cases related to login" --repo ../app
forgeqe run "Which tests were added this sprint?" --repo ../app

API tests

REST, GraphQL, SOAP, and gRPC. Every browser run records the app's own API calls (secrets redacted), and "turn the last run's API calls into tests" makes contracts from them: status, schema, and key fields per operation. Or start from an OpenAPI or Swagger link, a WSDL, a GraphQL endpoint or schema, or .proto files. ForgeQE runs the contracts straight against the API, with no browser, and Scribe writes them as API tests in your repo's framework.

forgeqe api contracts --spec https://petstore3.swagger.io/api/v3/openapi.json
forgeqe api contracts --run && forgeqe api run

End-to-end scenarios chain calls, passing values from one answer to the next request: from a browser run (the values that flowed are found for you), from a description (each resource's lifecycle, links, and negative cases), or planned from test cases or plain English. Scribe writes them on your repo's existing API helpers.

forgeqe api scenarios --text "create a user, add two items, check out" --spec openapi.yaml
forgeqe api run-scenarios --allow-writes

Environments

env.yaml lists your test environments (dev, qa, uat, …) and the apps in each: web apps by URL, APIs by base URL, Windows apps by .exe, and mobile apps by package or bundle id. Its layout is up to you; ForgeQE reads any shape. The same test cases run in every environment: "run the smoke cases on uat" picks it for one request, and --env NAME (or FORGEQE_ENV) for a command. Secrets are listed by the name of the environment variable that holds them, never by value. forgeqe apps shows the environments and their apps. See HOW_TO_USE.md.

Windows desktop apps

ForgeQE drives Windows apps with Microsoft's UFO² (explore a goal with the model) and pywinauto (replay with no model). A run that passes is saved as a forgeqe-desktop-v1 workflow you can read and edit.

• Replay with timeouts, retries, wait_for (a control gone, or showing some text), and dismiss for dialogs that get in the way. A control that moved is healed with one model call and reported for review.
• Visual checks: expect_image compares a control or the window with a baseline next to the workflow.
• Record by hand: forgeqe desktop record saves the steps you do yourself as a workflow.
• Edit from chat: "make step 4 wait for the Save button" changes the workflow with desktop_edit_workflow, after showing you the diff.
• CI: forgeqe desktop doctor checks the machine (a signed-in, unlocked desktop session; Windows services run in session 0, which has none), and --junit writes results for the pipeline.

forgeqe desktop explore "Create an invoice for Contoso" --app erp
forgeqe desktop record --app erp --name "New invoice"
forgeqe desktop replay desktop-tests/ --junit results.xml

See docs/DESKTOP.md and docs/DESKTOP-CI.md.

Mobile apps (Android, iOS)

One workflow format (forgeqe-mobile-v1) runs on Appium (explore with the model, replay step by step, healing, visual checks) and on Maestro (the workflow becomes a Maestro flow). Your team's own Maestro flows run too, after an allowlist check that refuses scripts, AI commands and file access. Both run on a local emulator or device, or on TestMu AI's devices: forgeqe mobile upload app.apk uploads a build and returns its lt:// id, and Maestro runs there through HyperExecute. Or ask in the shell ("upload courier.apk to TestMu for courier"): the mobile_upload tool asks you first, then, for an app listed under mobile.apps in forgeqe.yaml, writes the lt:// id to its app:. samples/mobile-testmu/ is a ready project that reads LT_USERNAME and LT_ACCESS_KEY from a .env file.

forgeqe mobile explore "Sign in as the courier and accept the first delivery" --app courier
forgeqe mobile replay workflow.yaml --driver maestro
forgeqe mobile maestro .maestro/

See docs/MOBILE.md.

Services and mocks in Docker

With Docker or Podman installed, ForgeQE starts the services a test run needs (a database, a queue, the API under test) as containers.services in forgeqe.yaml declares them, waits until each is ready, and stops them when the session ends. mock_service runs a WireMock mock of an API, built from the calls a browser run recorded or from an OpenAPI description, and can point an API app of the environment at it. ForgeQE builds every container command itself: ports bind to 127.0.0.1, mounts are read-only and inside the project, and privileged containers, the host network and the runtime's socket are refused. docker and podman are blocked in the agents' own shell.

forgeqe services up db         # start and wait until ready
forgeqe services               # what runs
forgeqe services down

Services can also come from a docker-compose file (containers.compose), checked by the same rules. With containers.tests, run_tests runs the repository's tests in a container, after starting the services they need, which the tests reach by name. Scribe's testcontainers skill writes tests that start their own services from code. Or ask: "mock the payments API from the last run and point payments at it". See docs/CONTAINERS.md.

Databases, test data, and data flows

List your databases under databases: in forgeqe.yaml, or under an environment in env.yaml (the active environment's entries win by name). No secret goes in either file: url_env or password_env names the environment variable that holds it. ForgeQE's agent reads the schema, writes SQL in the database's dialect, checks it, and runs it as a read; it writes only to a database marked writable: true, and only after you confirm the statement. In the shell, /db lists the databases, and /db add sets one up with the agent.

Test data comes from a spec that fits the schema: a preview, CSV, JSON or SQL files, or rows inserted into a writable database as a batch that forgeqe data clean removes again. Data flows (flows.yaml) describe how records move between databases; ForgeQE checks each hop (arrived, complete, correct, on time) and follows one record across the systems.

forgeqe db test orders
forgeqe db query orders "SELECT status, COUNT(*) FROM orders GROUP BY status"
forgeqe data gen customers.yaml --db fixtures --insert --yes
forgeqe data batches --db fixtures
forgeqe flows check --junit flows.xml
forgeqe flows trace digital.customers 42

See HOW_TO_USE.md, Test data, and Data flows.

Test cases and suites

Run existing test cases instead of typing goals: ask for them by name ("run SHOP-12", "run test plan 12 suite 34", "run the cases in smoke.yaml") or paste them into the message. Each case becomes one goal: its steps are what the browser agent does and its expected results are what it checks, and the agent's own verdict decides pass or fail. With a repo set, every passed case also gets a test.

forgeqe suite runs cases the same way every time, with no agent choosing anything, which is what pipelines need. It takes explicit references:

forgeqe cases cases.yaml                     # what a source gives, without running anything
forgeqe suite cases.yaml --parallel 3        # run them, 3 browsers at once
forgeqe suite ado:plan=12/suite=34 --repo ../app
forgeqe suite jira:jql="project = SHOP AND labels = smoke"
forgeqe suite xray:plan=SHOP-100
forgeqe suite "Open example.com and check the title says Example Domain"

For a case with numbered steps the browser agent reports a verdict for every step (passed, failed, or not run, with what it actually saw), not only for the case.

Results go back where the cases came from, with the step verdicts: an Azure DevOps test run for a plan (or a comment on the work item), a Jira comment, an Xray Test Execution with the failure screenshot, a TestRail run (the run the cases came from, or a new one) with step results and the screenshot, a Zephyr Scale test cycle, or qTest test logs. --no-write-back or sources.write_back: false turns it off.

Test cases the agent writes from an Azure DevOps or Jira story are saved there, linked to the story: Azure DevOps Test Case work items with their steps (Tests / Tested By, in the story's area and iteration), or Jira issues of type Test. The reply lists each case as a link, and saved Azure DevOps cases are run by id so results post back to them. A case already linked to the story under the same title is reused rather than duplicated. sources.save_cases: false turns this off; the agent then shows the cases formatted in its reply.

Accessibility and performance

Every run checks each page against WCAG (axe-core; WCAG 2.1 AA by default, or 2.2, AAA, Section 508, EN 301 549) and records its Core Web Vitals (LCP, CLS, INP, FCP, TTFB). Every run also compares each page with its approved screenshot (masked regions hidden) and shows baseline, this run, and the difference in the report; forgeqe visual approve accepts an intended change. Set budgets under quality.budgets and a run fails only when one is exceeded. forgeqe quality trends shows what got worse or better per page over time.

Exploratory testing

forgeqe explore ado:1234 (or "explore story 1234" in the shell) reads the story and its scripted test cases, plans charters beyond them (edge cases, invalid input, boundaries, role differences, refresh and back), and runs them in the browser. A defect counts only when a second run reproduces it; everything else is reported as an observation. You get a report, proposed test cases to save to the story, and bugs for confirmed defects only, after you agree.

Bugs

With sources.bugs, each failed case gets a bug in Jira or Azure DevOps with the steps and verdicts, where it ended, the build link, and the last screenshot, linked to the test case. ForgeQE tags the bug with a fingerprint of the case, so the next failure adds a "still failing" comment to the open bug instead of filing another, and a pass adds a note that it passes again.

sources:
  bugs: { system: jira, project: SHOP }          # or system: azure_devops (uses azure_devops.project)

Reports

Every suite writes report.html, junit.xml, and suite.json to .forgeqe/reports/suites/<time>/, with each case's run folder beside them; every single run writes its own report.html too. The report is one HTML file with the screenshots embedded, so you can open it straight from disk, attach it to a build, or mail it; no server is needed. It has an overview (verdict, totals, step checks, tokens, cost, page memory savings, browser errors, accessibility issues, bugs filed, what changed since the last run, results over earlier runs, the cases that need attention, results by source and tag, the slowest cases, and what was posted where), a test results page (each case's step table with expected and actual results, why it failed, its last passing screenshot beside this run's, browser errors per step, page memory with healed locators, the browser agent's steps with a screenshot each and the clicked element outlined, a timing waterfall, accessibility issues, and the trace and workflow to read in place), a failures page, defects (a suite's failures grouped by what caused them, so one fix clears every case it breaks), since last run, trends and flaky tests from the earlier suite runs in the same .forgeqe/reports/suites/ folder, time and cost (model vs browser time, a parallel worker timeline, cost per case from the model's price in llms.json), coverage by requirement and group from the test system, accessibility checks for every page visited, visual (each page against its approved screenshot), performance (Core Web Vitals against the budgets), the environment (model, browser, ForgeQE version, CI build, branch, commit), and Tokenomics, from the usage ledger in ~/.forgeqe/usage (never prompt text): usage (tokens, cache, cost, time), breakdown (by agent, job, task, purpose, model, user and machine), every model call, the models compared, trends and forecast (spend per day with a month forecast, and the budget: limits from forgeqe.yaml: warnings at 80%, 90% and 100%, nothing stopped), a calculator for what runs would cost at another size, cache rate or model, and optimise (where the report could have spent less on models, by how much, and what to change). It exports Markdown and JSON and prints cleanly. See samples/reports/nightly/20260929-020005/report.html for an example. report.screenshots: failures keeps screenshots for failed runs only, off drops them; report.accessibility: false skips the page checks. On GitHub Actions the suite also writes a job summary.

Static BI

forgeqe bi adds an offline BI layer on top of those reports. It exports curated CSV datasets from ForgeQE's own suite history and release-readiness reports, plus optional Azure DevOps WIQL and Jira JQL datasets from bi.datasets in forgeqe.yaml, into an Evidence project that reads them through its CSV source, then runs evidence sources and evidence build to produce a static site in the configured build directory. ForgeQE uses the open-source Evidence line (@evidence-dev/evidence 40.x, pinned in the generated package.json), because the newer Evidence Studio CLI has no local build and keeps data in its cloud. Building it needs Node.js 18 or later; serve the folder to view it (npx serve), since it cannot load its data when opened from disk. When Node.js is missing or the Evidence build fails, forgeqe bi build writes ForgeQE's own self-contained page from the same data instead, which opens straight from disk, and says why; --static always does that, and --evidence fails instead of falling back. Dataset ids become table names (forgeqe.<id>), so they use letters, digits, and _. Pages ForgeQE generates carry a "Managed by forgeqe bi" comment in their front matter; a page without it is yours and is never overwritten. Tables with no rows yet stay out of the Evidence project (Evidence cannot build over an empty table), and their sections read "None yet." No backend service stays up between runs.

bi:
  title: ForgeQE BI
  datasets:
    ado_current_sprint:
      kind: ado_wiql
      project: Shop
      wiql: |
        SELECT [System.Id]
        FROM WorkItems
        WHERE [System.TeamProject] = @project
          AND [System.WorkItemType] IN GROUP 'Microsoft.RequirementCategory'
    jira_open_bugs:
      kind: jira_jql
      jql: project = SHOP AND issuetype = Bug AND statusCategory != Done

forgeqe bi init            # scaffold the Evidence project and install its packages
forgeqe bi build           # refresh the data and build the site (ForgeQE's own page if Evidence cannot build)
forgeqe bi build --static  # ForgeQE's own page only, no Node.js needed

One report for the whole team

Each person runs forgeqe bi export, which writes every kind of run they have (suites, monitor rounds, single browser runs, desktop, mobile, exploratory and API runs, and release-readiness reports) to one JSON file in .forgeqe/reports/shared/, each run tagged with its run type. Whoever builds the report puts the files they receive in .forgeqe/reports/collected/ (or lists shared folders under bi.collect in forgeqe.yaml, or passes --from FILE|DIR, repeatable) and runs forgeqe bi build. Every table gets user and machine columns, the home page and ForgeQE's own page gain "Results by run type" and "Case results by person" tables, and a run that arrives twice (from two bundles, or a bundle and this machine) counts once. A bundle is read as data only: anything that is not one, or is larger than 64 MB, is skipped with a note, and a malformed run inside one is skipped whole.

In the shell, /report builds ForgeQE's own page from the same data, /report export writes your bundle, and /report <what you want> asks the agent, which can now build write_report datasets from these past results (history: case_results, suites, readiness, …) as well as from live Azure DevOps or Jira queries.

forgeqe bi export                          # your runs, in one file to share
forgeqe bi build --from //share/qa-reports # this machine's runs plus every bundle in that folder

Signing in

Sites that need a login go under logins:. The browser agent types credentials from the environment without ever seeing them (it only sees placeholders such as shop_password), fills authenticator codes from a TOTP secret, and keeps the signed-in session in .forgeqe/sessions/ so the next run starts signed in. Replays read the same environment variables, and so do the tests the coding agent writes.

logins:
  shop:
    url: https://staging.shop.test/login
    username_env: SHOP_USER
    password_env: SHOP_PASSWORD
    totp_env: SHOP_TOTP_SECRET      # optional

Credentials in a message or a test case work too: ForgeQE's agent remembers them for the session as a login for the site and its sign-in hosts, so goals and reports never contain the password, the exact value is typed, and follow-up runs start signed in.

For single sign-on, captchas, or security keys, sign in by hand once: forgeqe login shop opens a browser, you sign in, press Enter, and the session is saved. forgeqe logins lists logins and sessions; forgeqe logins clear shop forgets one. Session files are private to your user and have their own .gitignore.

Azure DevOps, Jira, and other systems

ForgeQE's agent connects to each system's MCP server, configured from the same sources: settings in forgeqe.yaml: organization, site URL, and the names of the environment variables that hold the tokens. You don't write any MCP config yourself. Azure DevOps's server is the organization, so a project is optional and only limits ForgeQE's own test, sprint, and bug calls.

sources:
  azure_devops: { org_url: https://dev.azure.com/acme }   # every project; token in ADO_PAT. project: Shop limits ForgeQE's own calls
  jira: { url: https://acme.atlassian.net }                              # JIRA_EMAIL + JIRA_API_TOKEN
mcp:
  github:
    url: https://api.githubcopilot.com/mcp/
    headers: { Authorization: "Bearer {env:GITHUB_TOKEN}" }

Tokens never go into the config: servers get them through {env:NAME} references that the agent runtime fills in, and the agent never sees them. forgeqe mcp list shows what your settings give. When the agent starts, the shell and forgeqe run print which servers connected.

Whatever has no MCP server, or whose server didn't connect, goes through ForgeQE's own tools, which call the REST APIs directly: find_test_cases and run_test_cases for test cases and posting results back, read_confluence for Confluence pages, and query_api to read anything else from Azure DevOps, Jira, Xray, or Confluence. query_api only reads (GET requests and Xray GraphQL queries) and only reaches the configured hosts. sources.azure_devops.mcp: false or sources.jira.mcp: false uses the API only.

Microsoft's Azure DevOps server can't create dashboards, so "create a dashboard in QI-Payments with all defects and this sprint's stories" goes through ForgeQE's create_ado_dashboard: one shared query per widget under Shared Queries/ForgeQE, then a dashboard with a work item list or count tile for each. Your ADO_PAT needs Work Items (read and write) and permission to edit dashboards. Chart widgets aren't made through the API; for those the agent uses the browser, signed in to dev.azure.com with a saved login.

Its wiki tools can only write into a wiki that already exists, so wiki pages go through ForgeQE's write_ado_wiki_page. If the project has no wiki yet, that tool creates the project wiki first, along with any missing parent pages. The PAT needs Wiki (read and write).

Requirements work too: "write and run test cases for the checkout rules page in Confluence" makes the agent read the page, write one case per rule or acceptance criterion (with the main negative paths), show them, and run them. It saves them into a test system only when you ask.

The agent creates or changes items (a bug, a comment, a status change) only when you ask it to; bugs for failed cases come from sources.bugs, not from the agent.

Pipelines

Ask for the pipeline you want and the agent builds it:

forgeqe run "Add a GitHub Actions workflow that runs the smoke cases in cases.yaml on every pull request"
forgeqe run "Set up an Azure DevOps pipeline that runs test plan 12 every night and posts results back"
forgeqe run "The ForgeQE pipeline on main is failing, fix it"

It works in the repo (the working directory, or --repo): it reads the repo's existing CI, writes or updates the workflow so it installs Bun, uv and ForgeQE, calls forgeqe suite <source...> --no-code --plain, and publishes junit.xml and the report, and validates it (actionlint when it is installed). Then it commits to a new branch named forgeqe/ci-<topic>, stores the keys the pipeline needs as secrets, triggers the run with gh or az, watches it, and fixes failures until the run passes (up to code.max_attempts failed runs).

Guard rails:

• Pushing goes through ForgeQE's push_branch tool, which refuses the default branch, main, and master, and never forces. git push itself is blocked for the agent.
• Secrets go through set_ci_secret: the agent names an environment variable and ForgeQE passes its value straight to gh secret set or az pipelines variable. The agent never sees the value.
• It needs gh (GitHub, logged in) or az with the azure-devops extension (Azure DevOps, logged in). Without them it still writes, validates, and commits the pipeline, then tells you what to run.

Monitoring

forgeqe monitor runs checks on a schedule and tells your team when one breaks. A check is any test case forgeqe suite takes: a YAML file, a Jira key, an Azure DevOps plan. Each round replays the flows from page memory, so a flow that passed before needs one model call to confirm the result (or none with replay: trust).

monitor:
  checks: [checks/smoke.yaml, "jira:jql=labels = monitor"]
  every: 15m
  fail_after: 2           # two failing rounds in a row before the first alert, to ride out a flaky run
  remind_every: 2h        # repeat while it keeps failing
  alerts:
    teams: { webhook_env: TEAMS_WEBHOOK_URL }
    email: { host: smtp.office365.com, port: 587, to: [qa-team@example.com] }   # SMTP_USERNAME, SMTP_PASSWORD

forgeqe monitor --test-alerts      # a sample alert, to check the webhook and SMTP settings
forgeqe monitor                    # every 15 minutes until Ctrl+C
forgeqe monitor --once             # one round, for cron or a scheduled pipeline

An alert goes out when a check starts failing, again every remind_every while it fails, and once when it recovers. One round sends one message covering every check that changed. Teams gets an Adaptive Card (both incoming webhooks and Workflows webhooks accept it) and email gets the failing checks' screenshots attached. Each alert links to the round's report: the local file, or <report_url>/<round>/report.html when you publish .forgeqe/reports/monitor/ somewhere. A round that cannot run at all (a test system is down, the engine will not start) alerts as the check "ForgeQE monitor".

Rounds go to .forgeqe/reports/monitor/<time>/, the same as suites, so the report shows trends and flaky checks over the rounds; the last keep rounds (100) are kept. state.json remembers which checks are failing and since when, so with --once in a pipeline, cache that folder between runs. Only one monitor can use a folder at a time. --once exits 0 when every check passed and 1 otherwise. Results are posted back to the test system only with write_back: true. In the shell, "monitor the smoke cases every 10 minutes" runs the rounds in the background while you keep working, and each round shows as a card; they stop when the shell closes.

MCP

forgeqe mcp serves ForgeQE's tools over stdio for other agents such as Claude Code, Cursor, or Codex:

{ "mcpServers": { "forgeqe": { "command": "forgeqe", "args": ["mcp", "--repo", "/path/to/app"] } } }

Tools: browser_run, find_test_cases, save_test_cases, run_test_cases, run_details, run_test, page_memory, read_confluence, query_api, push_branch, set_ci_secret, find_tests, run_tests (runs only with confirmed: true, once the user agreed), review_pull_request, post_pr_review (posts only with confirmed: true), list_sprints, generated_attribution (the commit trailer and pull request footer for code ForgeQE wrote), and one tool per command: write_test, heal_tests, monitor, logins, forgeqe_setup, plus forgeqe_command for the commands no other tool covers (reading commands run at once; the others ask first). Databases have databases, database_schema, run_sql, generate_test_data, test_data_batches, data_flows, check_flows, trace_record, and read_mapping_file. Browser runs and suites log their progress to stderr, and long calls send progress notifications. Desktop (desktop_explore, desktop_replay, desktop_record, desktop_edit_workflow) and mobile (mobile_explore, mobile_replay, mobile_maestro, mobile_upload) tools are listed when the environment has such apps. The client gets the tools this project can use; use_capabilities adds others and sends notifications/tools/list_changed, and forgeqe mcp --all-tools lists every tool.