newm is the customer-facing client for NewMeasure. It uploads task directories, starts runs in the cloud, shows results, and downloads stored artifacts. Requires Python 3.11 or later and a NewMeasure customer API key.

Install

uv tool install newmeasure-cli

pipx install newmeasure-cli
# or, inside a virtual environment
pip install newmeasure-cli

Usage

Sign in with your customer API key:

newm login

The default API is https://api.newmeasure.cc. Configuration is stored in ~/.newm/config.json with owner-only permissions.

Run a task:

# Upload and run a local task directory
newm run ./my-task --no-wait

# Or run an already uploaded task, several times
newm run t_abc123 --n-trials 5 --no-wait

--no-wait returns the run ID immediately; the run continues in the cloud. Without it, the CLI waits for completion. Check status with:

newm result <run-id>

Browser tasks

Browser tasks test a web product: an AI agent uses it in a cloud browser. Put the application login in a local env file, using the field names the task declares in credential_fields:

username=YOUR_APPLICATION_EMAIL
password=YOUR_APPLICATION_PASSWORD

Keep that file out of version control, and submit it with the run:

newm run ./my-task --credentials-file credentials.env --no-wait

Only the declared fields are sent, over HTTPS (plain HTTP is allowed for localhost). Cloud and model credentials are managed by NewMeasure.

Download a trial's artifacts using the trial ID from newm result:

newm artifacts <run-id> <trial-id> --output ./artifacts/<trial-id>

Downloads include outputs/ (the files the product generated, such as PDF and HTML exports), score.json (the reward and per-check breakdown), and manifest.json. The CLI checks each file's size and SHA-256.

A browser task bundle contains task.toml, instruction.md, and, when present, scenario.yaml, product.md (or ../../shared/product.md), and the environment/ and tests/ folders.

Commands

newm login                      # store API token
newm upload ./my-task           # validate and upload a task
newm run <task-id | dir>        # run an eval
newm result <run-id>            # run details and trial results
newm artifacts <run-id> <trial-id> [--output DIR]
newm tasks                      # list uploaded tasks
newm runs [<task-id>]           # list runs
newm harbor ...                 # pass through to the Harbor CLI
newm --help

Development

From this directory:

uv sync
uv run newm --help
uv run pytest