SmokeOps

Turn a plain-language prompt into a smoke check you own — local-first synthetic monitoring with Playwright.

What is SmokeOps?

SmokeOps helps you verify that critical user flows work correctly by turning natural language descriptions into automated smoke checks. You write what matters to test, SmokeOps proposes steps, you review and approve them, and you get back a JSON file you keep in your repository. Approved checks can be run or scheduled without ever calling an LLM again.

Why this matters: Most people either skip synthetic monitoring entirely, write fragile Playwright scripts by hand, or rely on hosted services they don't control. SmokeOps gives you AI-powered creation paired with a durable artifact (JSON in your repo) that you can review, edit, and schedule.

SmokeOps is alpha (actively developed, real-world friendly, some rough edges). You provide an LLM API key for the create step only; scheduled runs need no external service.

Installation

Requires Python 3.10 or later. Install the CLI into your environment:

pip install smokeops
# or
uv tool install smokeops

Confirm the installation:

smokeops version

On your first create or run, SmokeOps installs Chromium automatically if it's missing. In CI environments, you need to keep an explicit playwright install step in your workflows (examples are provided below).

Five-Minute Quickstart

This walkthrough shows the entire workflow in about 5 minutes using the included PeakAir demo site (a sample react app).

1. Start the demo site (optional)

The quickstart below uses PeakAir, a sample app included in this repository. Clone the repo and start it locally:

git clone https://github.com/gate3/SmokeOps.git
cd SmokeOps/examples/peakair-hvac
npm install
npm run dev

PeakAir runs at http://localhost:3000. Leave this terminal running.

If you installed SmokeOps from PyPI and do not want to clone the repo, skip this step. In step 3, replace the URL and prompt with a site you can reach (for example your own app or a public staging URL).

2. Set up LLM credentials

In a new terminal, work from the directory where you want monitors/ to live (for this walkthrough, the SmokeOps repo root: cd SmokeOps).

Create a .env file or export environment variables. SmokeOps routes LLM calls through LiteLLM and supports Gemini (default), OpenAI, and other providers:

export LLM_PROVIDER=gemini
export DEFAULT_MODEL=gemini-3.6-flash
export GEMINI_API_KEY=your_key_here

3. Create your first check

Tell SmokeOps what to test:

smokeops create --url "http://localhost:3000/" \
  "Verify the homepage loads and the product name PeakAir is visible"

By default, create runs headless: Playwright drives Chromium in the background with no visible browser window. You follow progress in the terminal (navigation, DOM pruning, step results). When all steps pass, SmokeOps shows a summary and waits for your approval.

To watch the browser while steps run:

smokeops create --headed --url "http://localhost:3000/" \
  "Verify the homepage loads and the product name PeakAir is visible"

--headed opens a real browser window, prints each step as it runs, and pauses briefly between asserts so you can follow along. When the run finishes, press Enter to close the window, then approve or reject as usual.

4. Approve the check

After SmokeOps runs the steps, you'll see a summary. Choose one of:

• accept — saves to monitors/<name>.json in your current directory (the check name is a slug from your prompt; edit the draft to rename before accept)
• reject — discards the draft
• edit — opens the JSON in your editor to adjust the name or summary

Accept this one. Note the path printed after accept (for example monitors/verify-the-homepage-loads-and-the-product-name-peakair-is-visible.json).

5. Run the approved check

Now run the check without the LLM, using the path from the accept step:

smokeops run monitors/<name>.json

Or omit the path to pick from monitors/*.json interactively:

smokeops run

No LLM is needed from this point. The check runs offline, using only the Playwright steps you approved. If it passes, you see PASS. If it fails, SmokeOps saves a screenshot and trace under .smokeops/artifacts/ for debugging.

Try a bundled journey check (clone only): this repo also includes an approved multipage PeakAir check you can run without creating one:

smokeops run monitors/complete-the-signup-form-with-sample-contact-details-choose-the-seasonal-tune-up.json

That's the core loop: create (LLM proposes) → approve (you decide) → run (offline verification, on schedule).

Why Choose SmokeOps?

What SmokeOps is NOT

• Not a full E2E test framework — for comprehensive test suites with hundreds of scenarios, use Playwright Test
• Not a visual regression tool — focuses on flow verification and assertions, not pixel-perfect comparisons
• Not designed for localhost in CI — use a public staging URL or tunnel service for GitHub Actions
• Not a monitoring platform — no dashboards, alerts, or historical trend analysis; use tools like DataDog or Grafana for that

Usage Guides

The quickstart runs through that loop once. These sections go deeper on each step.

Create

Describe what to test in plain language:

On the signup page, fill in an email, submit the form, and verify the account-created message appears.

When you run smokeops create, SmokeOps:

1. Navigates to the URL you provide
2. Finds interactive elements on the page (buttons, inputs, links, form controls)
3. Asks the LLM to propose Playwright steps that match your prompt
4. Runs those steps in a real browser to verify they work
5. Shows you a summary and waits for your decision

By default, create runs headless (no visible browser window). Add --headed to watch the browser and see each step in the terminal.

Single check

smokeops create --url "https://example.com/signup" \
  "Complete a signup with email and password, then verify the dashboard loads"

smokeops create --headed --url "https://example.com/signup" \
  "Complete a signup with email and password, then verify the dashboard loads"

With --headed, SmokeOps opens a browser window, prints each step as it runs, and prompts you to press Enter to close the window before approval.

Suite file (batch)

For multiple checks, use a YAML suite file:

# suite.yml
defaults:
  base_url: https://example.com

checks:
  - name: homepage-loads
    url: /
    prompt: |
      Verify the homepage loads and the company name is visible

  - name: user-signup
    url: /signup
    prompt: |
      Complete the signup form with sample contact details,
      accept the terms, and verify the account-created confirmation

  - name: user-login
    url: /login
    prompt: |
      Log in with a valid email and password, and verify the dashboard appears

Create all checks at once:

smokeops create --from suite.yml

SmokeOps walks through each check in order, proposing and waiting for your approval (accept/reject/edit/skip) for each. Approved checks are saved to monitors/. If a check file already exists, it is skipped.

smokeops create --from suite.yml --headed

With --headed, press Enter after each check completes to close the browser, then approve or reject.

Approve & edit

After create proposes steps, you see a plain-English summary and are prompted:

Choose a, r, e, or s.

• a (accept) — saves monitors/<name>.json
• r (reject) — discards the draft
• e (edit) — opens the draft JSON in your editor ($EDITOR or $VISUAL, fallback vi)
• s (skip) — available only with --from (suite file)

An accepted check is saved as monitors/<name>.json with:

• A plain-English summary (what you asked for and what the check does)
• The Playwright steps (action, element locator, assertions)
• The URL and your original prompt for reference

If you choose edit, SmokeOps opens your editor. Make changes (rename the check, adjust the summary, tweak a step), save and quit. SmokeOps re-validates the JSON and re-runs the steps to confirm they still work. Then you can accept or reject.

Set your preferred editor:

export EDITOR=nano
# or
export EDITOR="code --wait"  # VS Code

Run

The approved JSON file is your source of truth. smokeops run executes the stored steps with no LLM call and no API key:

smokeops run monitors/homepage.json

Omit the path in an interactive terminal to pick from monitors/*.json. Pass a path in CI or scripts.

smokeops run --headed monitors/homepage.json

With --headed, the browser window stays open until you press Enter.

On failure, SmokeOps writes a screenshot and Playwright trace under .smokeops/artifacts/ and prints those paths. Passing runs do not leave artifact files.

Exported *.spec.py files can also be run with smokeops run, but --headed is ignored for them.

Scheduled runs with GitHub Actions

Smoke checks run on a timer via GitHub Actions. The workflow handles setup, browser installation, and check execution. Scheduled run does not need an LLM API key.

Setup (one-time)

1. Copy examples/workflows/smokeops-cron.yml from this repository into your app repo as .github/workflows/smokeops-cron.yml:

cp examples/workflows/smokeops-cron.yml .github/workflows/smokeops-cron.yml

If you do not have a SmokeOps checkout, download the file from GitHub and save it at that path in your project.

2. Commit and push your approved check files under monitors/:

git add monitors/*.json
git commit -m "Add approved smoke checks"
git push

3. Adjust the cron schedule if needed. The template default runs every 15 minutes:

on:
  schedule:
    - cron: "*/15 * * * *" # Every 15 minutes. Adjust for your needs.

What runs on each schedule

Each time the schedule triggers:

1. GitHub Actions checks out your repo
2. Installs SmokeOps via PyPI
3. Installs Chromium (cached across runs)
4. Runs smokeops run for each monitors/*.json file
5. If any check fails, the job fails and uploads failure artifacts

Debugging failed runs

When a check fails on schedule:

1. Open the failed workflow run in GitHub Actions
2. The log shows which checks passed and which failed
3. SmokeOps writes a screenshot (PNG) and Playwright trace (.zip) to .smokeops/artifacts/
4. The workflow uploads these as a GitHub Actions artifact named smokeops-failure-artifacts
5. Download the zip, open the PNG in an image viewer, and open the trace with playwright show-trace

Running only some checks

If you have many checks but want to run a subset on schedule, edit the workflow to use a matrix:

strategy:
  fail-fast: false
  matrix:
    check:
      - monitors/homepage.json
      - monitors/login.json
      - monitors/checkout.json

steps:
  # ... setup steps ...
  - name: Run approved smoke check
    run: smokeops run ${{ matrix.check }}

Reaching external URLs

GitHub Actions runners are temporary cloud machines maintained by GitHub. They can reach any public URL on the internet, but they cannot reach:

• localhost or 127.0.0.1 (your local machine)
• Private IP addresses (internal networks)
• VPC-only resources

If your staging environment is not publicly reachable, you have two options:

1. Public staging URL — If you have a public staging or preview URL (for example staging.example.com), point your checks there.
2. Tunnel service — Services like Cloudflare Tunnel, ngrok, or localtunnel expose your local machine to the internet. After setting up a tunnel, you get a public URL (e.g., https://abc123.ngrok.io) that routes to your local machine. Use that URL in your checks.

Configuration

Environment Variables

Set these for create (LLM calls) and optionally run (for logging):

SmokeOps reads from a .env file in the current directory first, then falls back to environment variables.

Example .env

LLM_PROVIDER=gemini
DEFAULT_MODEL=gemini-3.6-flash
GEMINI_API_KEY=your_api_key_here
LOG_LEVEL=INFO

# Optional: increase wait time for SPAs
# SMOKEOPS_PRUNE_WAIT_MS=20000

Using a different LLM provider

To use OpenAI, Anthropic, or any LiteLLM-supported provider, set LLM_PROVIDER and DEFAULT_MODEL:

export LLM_PROVIDER=openai
export DEFAULT_MODEL=gpt-4o-mini
export OPENAI_API_KEY=sk-...

No code changes needed—SmokeOps routes all LLM calls through LiteLLM.

Logging

Control verbosity:

LOG_LEVEL=DEBUG smokeops create --url "..." "..."  # Verbose
LOG_LEVEL=OFF smokeops run monitors/check.json     # Silent

Other Commands

Show a check summary

Print the plain-English summary and metadata for an approved check without running it:

smokeops show                          # Pick interactively
smokeops show monitors/homepage.json   # Specific file

Export to standalone Playwright

If you want to run a check as a plain Playwright script (without SmokeOps), export it:

smokeops export monitors/homepage.json
# Writes: monitors/homepage.spec.py

Or choose a custom path:

smokeops export monitors/homepage.json --out exports/smoke.py

The exported file is a runnable Playwright script with no SmokeOps dependencies.

Contributing

To work on SmokeOps itself (development setup, tests, publishing), see CONTRIBUTING.md.

License

Licensed under the Apache License, Version 2.0.
See NOTICE for copyright.