Testery CLI

A Click-based Python CLI that wraps the Testery REST API and provides a generic test harness for running your tests locally or on Testery's cloud testing grid. Kick off runs from CI/CD, manage environments and schedules, upload build artifacts, and standardize how your tests are invoked and reported.

Installation

Requires Python 3 and pip:

pip install testery

Upgrade with:

pip install testery --upgrade

This installs the testery console script on your PATH.

Authentication

Most commands talk to the Testery API and need an API token. The test command only needs a token when running --remote; local runs need no token.

A token is resolved in this order:

1. --token <token> passed on the command line
2. --profile <name> → the named profile in ~/.testery/credentials
3. the [default] profile in ~/.testery/credentials
4. the TESTERY_API_TOKEN environment variable

The credentials file is INI-style:

[default]
token = your-token-here

[dev]
token = a-different-token

Log in and save a token

The easiest way to populate the credentials file is login, which opens the Testery app in your browser. Navigate to Settings > Integrations to find or create an API token, then paste it back into the prompt (a URL containing ?token=... is also accepted):

testery login                 # saves to [default]
testery login --profile dev   # saves to [dev]

Verify a token

testery verify-token --token <yourTesteryApiToken>

Add the hidden --testery-dev flag to any command to target the Testery dev API (https://api.dev.testery.io) instead of production.

testery test — the generic test harness

test is a standardized wrapper around your test framework's runner. It runs your tests locally by default (on Windows, Linux, or macOS) or on Testery with --remote, using the same options either way. It autodetects the framework, installs the runner if needed, controls the output format, and reports a normalized pass/fail summary.

test replaces and deprecates create-test-run. create-test-run continues to work unchanged for existing pipelines.

Run tests locally

# Autodetect the framework in the current directory and run it
testery test

# Point at a project directory and pick the framework explicitly
testery test --working-dir ./web --framework playwright

# Filter by tags, choose how many workers, and request JUnit + JSON artifacts
testery test --framework playwright \
  --include-tags "@smoke" \
  --runners 4 \
  --output-format junit --output-format json

Supported frameworks today: playwright and cucumber-js (more to come). With --framework autodetect (the default) the harness inspects package.json and config files to choose the runner — a project with a cucumber.js config is run with cucumber-js even when Playwright is also a dependency.

If the runner or its browsers aren't installed, the harness installs them (npm install / npx playwright install). Pass --no-install to skip this.

Output formats (--output-format)

Repeatable. Controls the result artifacts produced for local runs:

Artifacts are written to --output-dir (default testery-results/). A normalized pass/fail summary is always printed using --output (pretty, json, or teamcity).

testery test --output-format native --output-format junit --output-dir ./reports

Passing framework-specific arguments

Two ways, designed so new runner flags never require a CLI change:

# 1. Everything after `--` is forwarded verbatim to the runner
testery test --framework playwright -- --grep @smoke --workers 4 --headed

# 2. Or as a single quoted string
testery test --framework playwright --framework-params "--grep @smoke --workers 4"

Injecting variables

--variable KEY=VALUE (repeatable) is injected into the runner's process environment for local runs (and sent as run variables for --remote):

testery test --variable TARGET_URL=https://staging.example.com --variable DEBUG=1

Run tests on Testery (remote)

--remote submits a run to the Testery cloud. This path is identical to create-test-run and requires --token, --project, and --environment:

testery test --remote \
  --token <yourTesteryApiToken> \
  --project <projectKey> \
  --environment <environmentKey> \
  --build-id <uniqueBuildId> \
  --wait-for-results --fail-on-failure

Key options

Remote-only options accepted for a uniform interface: --git-ref/--commit, --git-branch/--branch, --test-name, --status-name, --test-suite, --latest-deploy, --copies, --build-id, --include-all-tags, --parallelize-by-file, --parallelize-by-test, --timeout-minutes, --test-timeout-seconds, --skip-vcs-updates, --deploy-id, --apply-test-selection-rules.

Running tests (other commands)

create-test-run (legacy)

Submits a test run to Testery. Superseded by testery test --remote, but kept for backward compatibility.

testery create-test-run --token <token> --project <projectKey> \
  --build-id <buildId> --environment <environmentKey> --wait-for-results

--fail-on-failure returns exit code 1 if there are test failures. Output formats: teamcity, pretty, json, appveyor, octopus.

run-test-plan

Submits a saved Test Plan run.

testery run-test-plan --token <token> --test-plan-key <planKey> --environment-key <envKey>

upload-test-run

Uploads results from a JUnit XML file as a Testery test run.

testery upload-test-run --token <token> --project-key <projectKey> \
  --environment-key <envKey> --path ./results.xml

Monitoring and reporting

# Follow a single run until it finishes
testery monitor-test-run --token <token> --test-run-id <id> --fail-on-failure

# Watch all active runs for N minutes
testery monitor-test-runs --token <token> --duration 10

# List currently-active runs
testery list-active-test-runs --token <token> --output json

# List recent runs (optionally filtered)
testery list-test-runs --token <token> --limit 25 --filter <filter> --output json

# Write per-test results to a file (e.g. SonarQube format)
testery report-test-run --token <token> --test-run-id <id> --output sonarcube --outfile results.xml

# Cancel runs
testery cancel-test-run --token <token> --test-run-id <id>
testery cancel-test-plan-run --token <token> --test-plan-key <planKey> --test-plan-run-id <id>

Environments

# Create
testery create-environment --token <token> --key <key> --name <name> \
  --variable "KEY1=FOO1" --variable "secure:KEY3=SECRET" --pipeline-stage <stage>

# Update (or create with --create-if-not-exists)
testery update-environment --token <token> --key <key> --name <name> --variable "KEY1=FOO1"

# Add variables from a .env file
testery add-env-vars-from-file --token <token> --environment-key <key> --env-file ./.env --overwrite

# Upload a file that tests can read from the working directory
testery upload-environment-file --token <token> --environment-key <key> \
  --file-name config.json --source-path ./config.json

# List / delete
testery list-environments --token <token> --pipeline-stage <stage> --show-archived
testery delete-environment --token <token> --key <key>

Pipeline stages

# Update or create a pipeline stage with variables
testery update-pipeline-stage --token <token> --name <name> \
  --variable "KEY1=FOO1" --variable "secure:KEY2=SECRET" --create-if-not-exists

# Add variables from a .env file
testery add-stage-vars-from-file --token <token> --name <name> --env-file ./.env --overwrite

Schedules

# Run on a cron interval
testery create-schedule --token <token> --schedule-name "Nightly" --schedule-type interval \
  --cron "0 2 * * *" --project-key <projectKey> --environment-key <envKey>

# Run when a project is deployed
testery create-schedule --token <token> --schedule-name "On Deploy" --schedule-type deploy \
  --project-key <projectKey> --environment-key <envKey> --deploy-project <projectKey>

# List / delete
testery list-schedules --token <token> --output json --show-archived
testery delete-schedule --token <token> --name "Nightly"

Deploys

Notify Testery of a deploy (triggers any ON_DEPLOY schedules).

testery create-deploy --token <token> --project <projectKey> --environment <envKey> \
  --build-id <buildId> --commit <gitSha> --wait-for-results --fail-on-failure

Build artifacts

# Upload a file or directory of artifacts tied to a build id
testery upload-build-artifacts --token <token> --project <projectKey> \
  --build-id <buildId> --path ./dist

# Attach a file to an existing test run
testery add-file --token <token> --test-run-id <id> --kind DotCover ./coverage.json

Account

testery verify-token --token <token>          # check a token is valid
testery login                                 # save a token to ~/.testery/credentials
testery load-users --token <token> --user-file ./users.txt   # bulk-add users (one email per line)

Development

pip install -r requirements.txt    # install dev + runtime deps
pip install --editable .           # local editable install (provides `testery` on PATH)
python setup.py bdist_wheel        # build a wheel

Testing

pytest tests/test_unit.py          # unit tests (no network/token needed)
flake8                             # lint

• Unit tests mock requests.* and the local runner subprocess, and drive commands with Click's CliRunner — see tests/test_unit.py.
• Integration tests in tests/test_integration.py hit the real API and require TESTERY_TOKEN (loaded from .env). Set USE_TESTERY_DEV=True to target dev.

Docker harness (testing the test command on Linux)

tests/e2e/Dockerfile builds an image that pip-installs the CLI and runs testery test against cucumber-js and Playwright fixture projects, validating the install and the local-execution path on Linux:

docker build -f tests/e2e/Dockerfile -t testery-harness-smoke .
docker run --rm testery-harness-smoke
# or, as a pytest (skipped when Docker is unavailable):
pytest tests/e2e/test_docker_harness.py