NewMeasure CLI

Upload evaluation tasks, start cloud runs, check results, and download artifacts from NewMeasure. Requires Python 3.11 or later and a NewMeasure customer API key.

Install

uv tool install newmeasure-cli

Alternatively, use pipx install newmeasure-cli, or install with pip install newmeasure-cli inside a virtual environment. The executable is newm.

Sign in

newm login

Enter your customer API key when prompted. The default API is https://api.newmeasure.cc. Local configuration is stored in ~/.newm/config.json with owner-only permissions.

Run a task

# Upload and run a local task directory
newm run ./my-task --no-wait

# Or run an already uploaded task
newm run t_abc123 --no-wait

--no-wait returns the run ID immediately; execution continues in the cloud. Without it, the CLI waits for completion. Check status with:

newm result <run-id>

For browser tasks, provide application credentials in a local environment file using the field names declared by the task, for example:

username=YOUR_APPLICATION_EMAIL
password=YOUR_APPLICATION_PASSWORD

Keep that file out of version control. Submit it for the run:

newm run ./my-task --credentials-file credentials.env --no-wait

The CLI sends the declared credential values over HTTPS for that run. Cloud provider and model credentials are managed by the NewMeasure service.

Download browser artifacts

Use the trial ID printed by newm result:

newm artifacts <run-id> <trial-id> --output ./artifacts/<trial-id>

Downloads include the available exports, evidence, and manifest. The CLI verifies file sizes and SHA-256 checksums. Browser smoke completion is an operational result, not a content-quality score. Inspect the manifest for cleanup status before repeating a failed task.

Other commands

newm upload ./my-task
newm tasks
newm runs
newm runs <task-id>
newm --help

Task IDs persist across uploads; each execution gets a fresh trial ID. Browser task bundles include the instruction, task settings, and optional scenario and product context. HTTP-agent tasks also require their seed and verifier files.

Development

From the CLI source directory:

uv sync
uv run newm --help
uv run pytest