Testery CLI
A Click-based Python CLI that wraps the Testery REST API and provides a generic test harness for running your tests locally or on Testery's cloud testing grid. Kick off runs from CI/CD, manage environments and schedules, upload build artifacts, and standardize how your tests are invoked and reported.
Installation
Requires Python 3 and pip:
pip install testery
Upgrade with:
pip install testery --upgrade
This installs the testery console script on your PATH.
Authentication
Most commands talk to the Testery API and need an API token. The test command
only needs a token when running --remote; local runs need no token.
A token is resolved in this order:
--token <token>passed on the command line--profile <name>→ the named profile in~/.testery/credentials- the
[default]profile in~/.testery/credentials - the
TESTERY_API_TOKENenvironment variable
The credentials file is INI-style:
[default]
token = your-token-here
[dev]
token = a-different-token
Log in and save a token
The easiest way to populate the credentials file is login, which opens the Testery
app in your browser. Navigate to Settings > Integrations to find or create an API
token, then paste it back into the prompt (a URL containing ?token=... is also
accepted):
testery login # saves to [default]
testery login --profile dev # saves to [dev]
Verify a token
testery verify-token --token <yourTesteryApiToken>
Add the hidden
--testery-devflag to any command to target the Testery dev API (https://api.dev.testery.io) instead of production.
testery test — the generic test harness
test is a standardized wrapper around your test framework's runner. It runs your
tests locally by default (on Windows, Linux, or macOS) or on Testery with
--remote, using the same options either way. It autodetects the framework,
installs the runner if needed, controls the output format, and reports a normalized
pass/fail summary.
testreplaces and deprecatescreate-test-run.create-test-runcontinues to work unchanged for existing pipelines.
Run tests locally
# Autodetect the framework in the current directory and run it
testery test
# Point at a project directory and pick the framework explicitly
testery test --working-dir ./web --framework playwright
# Filter by tags, choose how many workers, and request JUnit + JSON artifacts
testery test --framework playwright \
--include-tags "@smoke" \
--runners 4 \
--output-format junit --output-format json
Supported frameworks today: playwright and cucumber-js (more to come).
With --framework autodetect (the default) the harness inspects package.json and
config files to choose the runner — a project with a cucumber.js config is run with
cucumber-js even when Playwright is also a dependency.
If the runner or its browsers aren't installed, the harness installs them
(npm install / npx playwright install). Pass --no-install to skip this.
Output formats (--output-format)
Repeatable. Controls the result artifacts produced for local runs:
| Format | Behavior |
|---|---|
native |
(default) Streams exactly what the underlying runner prints — identical to running e.g. npx playwright test yourself. |
junit |
Writes a JUnit XML report to the output directory. |
json |
Writes a JSON report to the output directory. |
Artifacts are written to --output-dir (default testery-results/). A normalized
pass/fail summary is always printed using --output (pretty, json, or teamcity).
testery test --output-format native --output-format junit --output-dir ./reports
Passing framework-specific arguments
Two ways, designed so new runner flags never require a CLI change:
# 1. Everything after `--` is forwarded verbatim to the runner
testery test --framework playwright -- --grep @smoke --workers 4 --headed
# 2. Or as a single quoted string
testery test --framework playwright --framework-params "--grep @smoke --workers 4"
Injecting variables
--variable KEY=VALUE (repeatable) is injected into the runner's process
environment for local runs (and sent as run variables for --remote):
testery test --variable TARGET_URL=https://staging.example.com --variable DEBUG=1
Run tests on Testery (remote)
--remote submits a run to the Testery cloud. This path is identical to
create-test-run and requires --token, --project, and --environment:
testery test --remote \
--token <yourTesteryApiToken> \
--project <projectKey> \
--environment <environmentKey> \
--build-id <uniqueBuildId> \
--wait-for-results --fail-on-failure
Key options
| Option | Applies to | Description |
|---|---|---|
--local / --remote |
both | Run on this machine (default) or submit to Testery. |
--working-dir, --project-dir |
local | Project directory to run. Defaults to the current directory. |
--framework |
local | playwright, cucumber-js, or autodetect (default). |
--output-format, --output-formats |
local | native (default), junit, json. Repeatable. |
--output-dir |
local | Where junit/json artifacts are written. |
--framework-params |
local | Extra runner arguments as a single quoted string. |
--install / --no-install |
local | Auto-install the runner/browsers if missing (default: install). |
--include-tags / --exclude-tags |
both | Comma-separated tag filters. |
--test-filter-regex |
both | Filter tests by regular expression. Repeatable. |
--runner-count, --runners |
both | Parallel runners (remote) / workers (local). |
--playwright-project |
both | The Playwright project to run. |
--retry-failed-tests |
both | Retry failed tests once. |
--variable |
both | KEY=VALUE (use secure:KEY=VALUE to encrypt for remote). Repeatable. |
--output |
both | Summary format: pretty (default), json, teamcity. |
--fail-on-failure |
both | Exit non-zero if any test fails. |
--wait-for-results |
remote | Poll until the remote run completes. |
Remote-only options accepted for a uniform interface: --git-ref/--commit,
--git-branch/--branch, --test-name, --status-name, --test-suite,
--latest-deploy, --copies, --build-id, --include-all-tags,
--parallelize-by-file, --parallelize-by-test, --timeout-minutes,
--test-timeout-seconds, --skip-vcs-updates, --deploy-id,
--apply-test-selection-rules.
Running tests (other commands)
create-test-run (legacy)
Submits a test run to Testery. Superseded by testery test --remote, but kept for
backward compatibility.
testery create-test-run --token <token> --project <projectKey> \
--build-id <buildId> --environment <environmentKey> --wait-for-results
--fail-on-failure returns exit code 1 if there are test failures. Output formats:
teamcity, pretty, json, appveyor, octopus.
run-test-plan
Submits a saved Test Plan run.
testery run-test-plan --token <token> --test-plan-key <planKey> --environment-key <envKey>
upload-test-run
Uploads results from a JUnit XML file as a Testery test run.
testery upload-test-run --token <token> --project-key <projectKey> \
--environment-key <envKey> --path ./results.xml
Monitoring and reporting
# Follow a single run until it finishes
testery monitor-test-run --token <token> --test-run-id <id> --fail-on-failure
# Watch all active runs for N minutes
testery monitor-test-runs --token <token> --duration 10
# List currently-active runs
testery list-active-test-runs --token <token> --output json
# List recent runs (optionally filtered)
testery list-test-runs --token <token> --limit 25 --filter <filter> --output json
# Write per-test results to a file (e.g. SonarQube format)
testery report-test-run --token <token> --test-run-id <id> --output sonarcube --outfile results.xml
# Cancel runs
testery cancel-test-run --token <token> --test-run-id <id>
testery cancel-test-plan-run --token <token> --test-plan-key <planKey> --test-plan-run-id <id>
Environments
# Create
testery create-environment --token <token> --key <key> --name <name> \
--variable "KEY1=FOO1" --variable "secure:KEY3=SECRET" --pipeline-stage <stage>
# Update (or create with --create-if-not-exists)
testery update-environment --token <token> --key <key> --name <name> --variable "KEY1=FOO1"
# Add variables from a .env file
testery add-env-vars-from-file --token <token> --environment-key <key> --env-file ./.env --overwrite
# Upload a file that tests can read from the working directory
testery upload-environment-file --token <token> --environment-key <key> \
--file-name config.json --source-path ./config.json
# List / delete
testery list-environments --token <token> --pipeline-stage <stage> --show-archived
testery delete-environment --token <token> --key <key>
Pipeline stages
# Update or create a pipeline stage with variables
testery update-pipeline-stage --token <token> --name <name> \
--variable "KEY1=FOO1" --variable "secure:KEY2=SECRET" --create-if-not-exists
# Add variables from a .env file
testery add-stage-vars-from-file --token <token> --name <name> --env-file ./.env --overwrite
Schedules
# Run on a cron interval
testery create-schedule --token <token> --schedule-name "Nightly" --schedule-type interval \
--cron "0 2 * * *" --project-key <projectKey> --environment-key <envKey>
# Run when a project is deployed
testery create-schedule --token <token> --schedule-name "On Deploy" --schedule-type deploy \
--project-key <projectKey> --environment-key <envKey> --deploy-project <projectKey>
# List / delete
testery list-schedules --token <token> --output json --show-archived
testery delete-schedule --token <token> --name "Nightly"
Deploys
Notify Testery of a deploy (triggers any ON_DEPLOY schedules).
testery create-deploy --token <token> --project <projectKey> --environment <envKey> \
--build-id <buildId> --commit <gitSha> --wait-for-results --fail-on-failure
Build artifacts
# Upload a file or directory of artifacts tied to a build id
testery upload-build-artifacts --token <token> --project <projectKey> \
--build-id <buildId> --path ./dist
# Attach a file to an existing test run
testery add-file --token <token> --test-run-id <id> --kind DotCover ./coverage.json
Account
testery verify-token --token <token> # check a token is valid
testery login # save a token to ~/.testery/credentials
testery load-users --token <token> --user-file ./users.txt # bulk-add users (one email per line)
Development
pip install -r requirements.txt # install dev + runtime deps
pip install --editable . # local editable install (provides `testery` on PATH)
python setup.py bdist_wheel # build a wheel
Testing
pytest tests/test_unit.py # unit tests (no network/token needed)
flake8 # lint
- Unit tests mock
requests.*and the local runner subprocess, and drive commands with Click'sCliRunner— seetests/test_unit.py. - Integration tests in
tests/test_integration.pyhit the real API and requireTESTERY_TOKEN(loaded from.env). SetUSE_TESTERY_DEV=Trueto target dev.
Docker harness (testing the test command on Linux)
tests/e2e/Dockerfile builds an image that pip-installs the CLI and runs
testery test against cucumber-js and Playwright fixture projects, validating the
install and the local-execution path on Linux:
docker build -f tests/e2e/Dockerfile -t testery-harness-smoke .
docker run --rm testery-harness-smoke
# or, as a pytest (skipped when Docker is unavailable):
pytest tests/e2e/test_docker_harness.py
1 maintainer
Metadata
1 maintainer
Release files for testery 1.19.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| testery-1.19.0.tar.gz | 63.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| testery-1.19.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 128.1 kB
Release files / testery-1.19.0.tar.gz
| Download URL | testery-1.19.0.tar.gz |
|---|---|
| Size | 63.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4d1a545316369467d384250f61b753635abd4faf0740d233a346daf89d9cde2a
|
|
BLAKE2b-256 checksum How to use checksums |
053654b6ce4175c53bd02da62e10182b6b39309289e9c35abb1fab2bc1d80a2f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.12
|
Release files / testery-1.19.0-py3-none-any.whl
| Download URL | testery-1.19.0-py3-none-any.whl |
|---|---|
| Size | 64.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e49e63478472708b1d19a2627ff3ea115406d2729a12717f3d15b647339a187e
|
|
BLAKE2b-256 checksum How to use checksums |
e5f20805692175cceb81f1a06594e6d6ed558abd3985716faf1c970064298bd5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.12
|
