smokeops 0.1.0

Turn a prompt into a production smoke check you own — local-first synthetic monitoring.

SmokeOps

Turn a plain-language prompt into a smoke check you own — local-first synthetic monitoring with Playwright.

How it works

1. Install the SmokeOps CLI (no need to clone this repo).
2. Point it at a URL you care about.
3. Create one or more checks (a suite file is the preferred path for multiple checks).
4. Approve each proposed check; accepted checks are saved as monitors/<name>.json in your project.
5. Run those JSON files on demand (no LLM call).
6. Optionally export a standalone Playwright script, or schedule runs with GitHub Actions.

Install

Install the CLI into your environment. Keep approved checks and any schedule workflow in your app repo.

# We are working on making it available on PyPI (uv tool install smokeops) Until then, install from git:
uv tool install git+https://github.com/gate3/SmokeOps

On the first local create or run, SmokeOps installs Chromium if it is missing. CI does not auto-install browsers — keep the explicit playwright install step in your workflow (already in the example cron template).

Set LLM environment variables for create only. Scheduled run does not need an API key.

export LLM_PROVIDER=gemini
export DEFAULT_MODEL=gemini-2.0-flash
export GEMINI_API_KEY=…
smokeops version

LLM calls go through LiteLLM. Default provider is Gemini:

Env	Purpose
LLM_PROVIDER	Provider id for LiteLLM (default gemini)
DEFAULT_MODEL	Model id without provider prefix (for example gemini-2.0-flash)
GEMINI_API_KEY	Required when LLM_PROVIDER is gemini
SMOKEOPS_PRUNE_WAIT_MS	Max ms for create DOM-prune networkidle wait (default 10000; soft timeout). Also PRUNE_WAIT_MS.

To use another provider, set for example LLM_PROVIDER=openai, DEFAULT_MODEL=gpt-4o-mini, and OPENAI_API_KEY. No SmokeOps code change is required.

Choose a target site

Any reachable URL works. Run create and run from the project where you want monitors/ to live.

For a local walkthrough, this repo includes a disposable demo site (PeakAir) under examples/peakair-hvac. That fixture needs a checkout of this repository (or a copy of those files):

cd examples/peakair-hvac
npm install
npm run dev

The site listens on http://localhost:3000. PeakAir is only an example — SmokeOps is meant for arbitrary sites.

Create checks (preferred: suite file)

To author several checks at once, pass a suite YAML with --from. The suite file is for authoring only. What run and cron execute are the approved JSON files under monitors/.

Create a file such as suite.yml in your project:

defaults:
  base_url: https://example.com

checks:
  - name: homepage
    url: /
    prompt: |
      Verify the homepage loads and the product name is visible

  - name: signup
    url: /signup
    prompt: |
      Submit the signup form without filling fields and verify a required-field error is shown

Then:

smokeops create --from suite.yml

Each check is proposed and approved in order. You can accept, reject, edit, or skip. If monitors/<name>.json already exists, that check is skipped. Add --headed to watch the browser (press Enter after each check before the next). Add --export to also write a Playwright script after each accept.

If you are using the PeakAir fixture from a checkout of this repo:

smokeops create --from examples/peakair-hvac/suite.yml

Single-check create

For one check, pass a URL and a prompt:

smokeops create --url "https://example.com/" \
  "Verify the homepage loads and the product name is visible"

Watch the browser while steps run (local only; default is headless for CI):

smokeops create --headed --url "https://example.com/" \
  "Verify the homepage loads and the product name is visible"

Optional: raise the SPA settle budget for create (env or flag; soft timeout — prune continues after):

smokeops create --prune-wait-ms 20000 --url "https://example.com/" \
  "Verify the homepage loads and the product name is visible"

Presence-only checks pause briefly between asserts when headed. When the browser work finishes, press Enter to close the window, then approve as usual.

To also export a Playwright script right after accept:

smokeops create --url "https://example.com/" --export \
  "Verify the homepage loads and the product name is visible"

Presence vs journey

Write prompts for the kind of check you want:

• Presence — the page is up and expected copy or branding is visible. Example: Verify the homepage loads and PeakAir heating and cooling branding is visible
• Journey — interact with the page (fill, click, select), then assert an outcome. Example: On the signup page, submit the form without filling any fields and verify a required-field validation error is shown

More presence, journey, and combined scenarios for PeakAir are in examples/peakair-hvac/README.md.

Approve

After a successful proposal you get a short plain-English summary, then choose:

• accept — saves monitors/<name>.json (machine steps plus human summary)
• reject — discards the draft; nothing is written under monitors/
• edit — opens the draft in $VISUAL or $EDITOR (fallback vi); after you save and quit, SmokeOps re-validates and re-runs the steps before you can accept
• skip — available when creating from a suite file; leaves that check unchanged

Drafts live under .smokeops/drafts/ (gitignored). Set an editor if needed:

export EDITOR=nano
# or
export EDITOR="code --wait"

Show a check summary

Print the stored plain-English summary and light metadata (name, URL, prompt, step count) without opening the JSON.

smokeops show
smokeops show monitors/<name>.json

Omit the path in an interactive terminal to pick from monitors/*.json. Pass a path in non-interactive contexts.

Run an approved check

smokeops run executes the approved JSON through SmokeOps’ step executor. There is no LLM call and no export required.

smokeops run
smokeops run monitors/<name>.json

Omit the path in an interactive terminal to pick a check. Pass a path in CI or cron.

Watch mode:

smokeops run --headed monitors/<name>.json

Press Enter to close the browser when finished (skipped automatically in non-interactive CI).

On failure, SmokeOps writes a screenshot and Playwright trace under .smokeops/artifacts/ and prints those paths. Passing runs do not leave artifact files. Artifact capture applies to approved JSON checks (and create / edit re-verify). Exported *.spec.py files run as plain Playwright scripts without SmokeOps artifact capture.

Export a Playwright smoke file (optional)

For teams that want a standalone Playwright file they can own and run independently:

smokeops export monitors/<name>.json

By default this writes monitors/<name>.spec.py. Choose a custom path with --out:

smokeops export monitors/<name>.json --out exports/smoke.py

Schedule with GitHub Actions

GitHub Actions is the timer. The smoke logic stays in your approved check files under monitors/.

1. Copy examples/workflows/smokeops-cron.yml into your app repo as .github/workflows/smokeops-cron.yml.
2. Commit and push your approved check JSON files under monitors/.
3. The template installs the SmokeOps package (git URL until PyPI; switch to uv pip install smokeops after publish). It does not vendor SmokeOps source.
4. Adjust the cron expression if you want a different schedule.

By default the workflow runs every monitors/*.json file with smokeops run.

Scheduled run does not need an LLM API key. If any check fails, the job fails. Target URLs must be reachable from GitHub-hosted runners (not localhost — use a public host or a tunnel).

When a scheduled run fails

Open the failed run from the Actions tab. The log shows which checks passed or failed. On failure, SmokeOps also writes a screenshot and Playwright trace under .smokeops/artifacts/ on the runner.

Those files disappear when the job ends unless uploaded. The template uploads them as a GitHub Actions artifact named smokeops-failure-artifacts. On the run page, open Artifacts, download the zip, and open the PNG or trace.

Run only some checks

Edit the workflow to use a matrix instead of looping all files:

strategy:
  fail-fast: false
  matrix:
    check:
      - monitors/homepage.json
      - monitors/signup.json
steps:
  # ... setup steps ...
  - name: Run approved smoke check
    run: smokeops run ${{ matrix.check }}

Logging

Optional logging control via LOG_LEVEL or SMOKEOPS_LOG_LEVEL (DEBUG, INFO, WARNING, ERROR, CRITICAL, or OFF):

LOG_LEVEL=DEBUG smokeops version
LOG_LEVEL=OFF smokeops version

Contributing

To work on SmokeOps itself (dev setup, tests, LLM eval gate, PyPI publish), see CONTRIBUTING.md.

License

Licensed under the Apache License, Version 2.0. See NOTICE for copyright.