veris-cli 2.15.1

CLI to connect local agents to the Veris backend

Veris CLI

Connect your existing agent to Veris simulations.

The Veris CLI lets you package your agent as a Docker image and run it against simulation scenarios.

Installation

uv tool install veris-cli

Or from source:

uv tool install git+https://github.com/veris-ai/veris-cli.git

Quick Start

1. Login

# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY

This saves your credentials to ~/.veris/config.yaml.

2. Initialize Your Project

veris init

This creates a .veris/ folder with template files:

• Dockerfile.sandbox — Docker image template for your agent
• veris.yaml — Simulation configuration (services, persona, agent settings)
• .dockerignore — Build context exclusions

3. Create an Environment

veris env create --name "my-agent"

This registers your environment on the backend and saves the environment ID to .veris/config.yaml.

4. Configure Your Agent

Edit .veris/Dockerfile.sandbox and .veris/veris.yaml to match your agent. Set secrets:

veris env vars set OPENAI_API_KEY=sk-your-key --secret

5. Build and Push

veris env push

6. Generate Scenarios

veris scenarios create --num 10
veris scenarios status <SET_ID> --watch

7. Run Simulations

veris simulations create --scenario-set-id <SET_ID>
veris simulations status <SIM_RUN_ID> --watch

8. Evaluate Results

veris evaluations create --sim-run-id <SIM_RUN_ID>
veris evaluations status <SIM_RUN_ID> <EVAL_RUN_ID> --watch

9. Generate Report

veris reports create <SIM_RUN_ID>
veris reports status <REPORT_ID> --watch
veris reports get <REPORT_ID> -o results.html

Full Pipeline (One Command)

# Interactive — prompts for each step
veris run

# CI — all flags, markdown summary to stdout
veris run --scenario-set-id <SET_ID> --grader-id <GRADER_ID> --report

Command Reference

Top-Level Commands

veris login [API_KEY]              # Authenticate (browser or API key)
  --backend-url URL                # Custom backend URL
  --console-url URL                # Custom console URL
  --org ORG_ID                     # Scope to organization

veris init                         # Scaffold .veris/ directory (local only)

veris run                          # Full pipeline: simulations → evaluations → reports
  --scenario-set-id ID             # Scenario set (prompts if omitted)
  --grader-id ID                   # Grader (prompts if omitted)
  --env-id ID                      # Environment (uses config if omitted)
  --concurrency N                  # Parallel jobs (default: 10)
  --image-tag TAG                  # Image tag (default: latest)
  --simulation-timeout N           # Timeout per sim in seconds
  --report                         # Generate HTML report after evaluation

Environment (veris env)

veris env create --name NAME       # Register environment on backend
veris env push [--tag TAG]         # Build and push image (Cloud Build)
veris env list                     # List environments
veris env delete ENV_ID            # Delete environment

Environment Variables (veris env vars)

veris env vars set K=V [K=V ...]   # Set variables
  --secret                         # Mark as secret
  --env-id ID                      # Override environment
veris env vars list                # List variables
veris env vars rm KEY              # Remove a variable

Scenarios (veris scenarios)

veris scenarios create             # Generate scenario set + grader
  --num N                          # Number of scenarios (default: 5)
  --env-id ID                      # Environment
  --image-tag TAG                  # Image tag
veris scenarios status SET_ID      # Check generation progress
  --watch                          # Poll until done
veris scenarios list               # List scenario sets (with grader column)
veris scenarios get SET_ID         # Open in console browser
veris scenarios delete SET_ID      # Delete scenario set

Simulations (veris simulations)

veris simulations create           # Create simulation run (interactive)
  --scenario-set-id ID             # Scenario set
  --env-id ID                      # Environment
  --concurrency N                  # Parallel jobs
  --simulation-timeout N           # Timeout per sim
  --image-tag TAG                  # Image tag
veris simulations status SIM_RUN_ID  # Run progress + sim list
  --watch                          # Poll until done
  --log                            # Append event stream
veris simulations list             # List simulation runs
  --status STATUS                  # Filter by status
  --env-id ID                      # Filter by environment
veris simulations cancel SIM_RUN_ID  # Cancel a run

Evaluations (veris evaluations)

veris evaluations create           # Trigger grading (interactive)
  --sim-run-id ID                  # Simulation run
  --grader-id ID                   # Grader (pre-selected by scenario set)
veris evaluations status SIM_RUN_ID EVAL_RUN_ID  # Eval progress
  --watch                          # Poll until done
veris evaluations list [SIM_RUN_ID]  # List eval runs
                                   # Without args: all evals for current env
                                   # With arg: evals for specific run
veris evaluations get SIM_RUN_ID EVAL_RUN_ID  # Open in console browser

Reports (veris reports)

veris reports create [SIM_RUN_ID]  # Trigger report generation
  --eval-run-id ID                 # Specific eval run
veris reports status REPORT_ID     # Report progress
  --watch                          # Poll until done
veris reports list                 # List reports
veris reports get REPORT_ID        # Download report HTML
  -o PATH                          # Output path

Profiles (veris profile)

veris profile list                 # List all profiles
veris profile get                  # Show active profile
veris profile use [NAME]           # Set active profile + org (interactive)
  --org ORG_ID                     # Set organization
veris profile delete NAME          # Remove a profile

# Use a profile for any command
veris --profile staging env list

CI/CD Integration

name: Veris Simulation
on:
  pull_request:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    environment: veris-sim-ci
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install veris-cli
      - name: Build & push
        env:
          VERIS_API_KEY: ${{ secrets.VERIS_API_KEY }}
        run: |
          veris login "$VERIS_API_KEY"
          veris env push --tag ${{ github.sha }}
      - name: Run pipeline
        run: veris run --image-tag ${{ github.sha }} --report > summary.md
      - uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: summary.md

How It Works

veris init                    → Scaffold .veris/ directory
veris env create              → Register environment on backend
veris env push                → Build and push Docker image
veris scenarios create        → Generate scenarios + grader
veris simulations create      → Run agent against scenarios
veris evaluations create      → Grade simulation results
veris reports create          → Generate failure analysis report

Or use veris run to chain all steps interactively.

Configuration Files

~/.veris/config.yaml

Global config with named profiles:

active_profile: default
profiles:
  default:
    api_key: vrs_abc123
    backend_url: https://sandbox.api.veris.ai
    console_url: https://console.veris.ai
  staging:
    api_key: vrs_staging
    backend_url: https://sandbox.api.veris.ai
    organization_id: org_abc123

.veris/config.yaml

Project config (auto-generated):

profiles:
  default:
    environment_id: env_abc123
    environment_name: my-agent

Profiles

The CLI supports named profiles for managing multiple environments. Each profile stores its own API key, backend URL, console URL, organization, and environment ID.

# Login to a named profile
veris --profile acme login <api-key> --org org_abc123

# Switch active profile
veris profile use acme

# Override for a single command
veris --profile production env list

Profile resolution: --profile flag → active_profile in config → "default".

Existing flat configs (without profiles key) auto-migrate on first write.

Development

git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli
uv sync
uv run pytest
uv tool install --force -e .

Troubleshooting

• "No API key found" — Run veris login
• Cloud Build fails — Check build logs in the output. Ensure Dockerfile.sandbox is valid.
• Image push fails — Credentials are fetched automatically. Just retry.

Support

• GitHub Issues: https://github.com/veris-ai/veris-cli/issues
• Email: [e-mail]

License

MIT