veris-cli 2.22.0

CLI to connect local agents to the Veris backend

Veris CLI

Connect your existing agent to Veris simulations.

The Veris CLI lets you package your agent and run it against simulation scenarios.

Installation

uv tool install veris-cli

Or from source:

uv tool install git+https://github.com/veris-ai/veris-cli.git

Quick Start

1. Login

# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY

This saves your credentials to ~/.veris/config.yaml.

2. Create an Environment

veris env create --name "my-agent"

This scaffolds a .veris/ directory and registers your environment on Veris:

• Dockerfile.sandbox — Agent image definition
• veris.yaml — Simulation configuration (services, persona, agent settings)
• .dockerignore — Files excluded from image build

3. Configure Your Agent

Edit .veris/Dockerfile.sandbox and .veris/veris.yaml to match your agent. Set secrets:

veris env vars set OPENAI_API_KEY=sk-your-key --secret

4. Build and Push

veris env push

5. Generate Scenarios

veris scenarios create --num 10
veris scenarios status <SET_ID> --watch

6. Run Simulations

veris simulations create --scenario-set-id <SET_ID>
veris simulations status <SIM_RUN_ID> --watch

7. Evaluate Results

veris evaluations create --sim-run-id <SIM_RUN_ID>
veris evaluations status <SIM_RUN_ID> <EVAL_RUN_ID> --watch

8. Generate Report

veris reports create <SIM_RUN_ID>
veris reports status <REPORT_ID> --watch
veris reports get <REPORT_ID> -o results.html

Full Pipeline (One Command)

# Interactive — prompts for each step
veris run

# CI — all flags, markdown summary to stdout
veris run --scenario-set-id <SET_ID> --grader-id <GRADER_ID> --report

Command Reference

Top-Level Commands

veris login [API_KEY]              # Authenticate (browser or API key)
  --profile NAME                   # Profile to log in to (sets it active)
  --backend-url URL                # Custom backend URL
  --console-url URL                # Custom console URL
  --org ORG_ID                     # Scope to organization

veris run                          # Full pipeline: simulations → evaluations → reports
  --scenario-set-id ID             # Scenario set (prompts if omitted)
  --grader-id ID                   # Grader (prompts if omitted)
  --env-id ID                      # Environment (uses config if omitted)
  --image-tag TAG                  # Image tag (default: latest)
  --simulation-timeout N           # Timeout per sim in seconds
  --report                         # Generate HTML report after evaluation

Environment (veris env)

veris env create --name NAME       # Scaffold .veris/ + register environment
veris env push [--tag TAG]         # Build and push image to Veris
veris env list                     # List environments
veris env delete ENV_ID            # Delete environment

Environment Config (veris env config)

veris env config push              # Upload veris.yaml to backend (no image build)
  --file PATH                      # Custom veris.yaml path (default: .veris/veris.yaml)
  --env-id ID                      # Override environment

Environment Variables (veris env vars)

veris env vars set K=V [K=V ...]   # Set variables
  --secret                         # Mark as secret
  --env-id ID                      # Override environment
veris env vars list                # List variables
veris env vars rm KEY              # Remove a variable

Scenarios (veris scenarios)

veris scenarios create             # Generate scenario set + grader
  --num N                          # Number of scenarios (default: 5)
  --env-id ID                      # Environment
  --image-tag TAG                  # Image tag
veris scenarios status SET_ID      # Check generation progress
  --watch                          # Poll until done
veris scenarios list               # List scenario sets (with grader column)
veris scenarios get SET_ID         # Open in console browser
veris scenarios delete SET_ID      # Delete scenario set

Simulations (veris simulations)

veris simulations create           # Create simulation run (interactive)
  --scenario-set-id ID             # Scenario set
  --env-id ID                      # Environment
  --simulation-timeout N           # Timeout per sim
  --image-tag TAG                  # Image tag
veris simulations status SIM_RUN_ID  # Run progress + sim list
  --watch                          # Poll until done
  --log                            # Append event stream
veris simulations list             # List simulation runs
  --status STATUS                  # Filter by status
  --env-id ID                      # Filter by environment
veris simulations cancel SIM_RUN_ID  # Cancel a run

Evaluations (veris evaluations)

veris evaluations create           # Trigger grading (interactive)
  --sim-run-id ID                  # Simulation run
  --grader-id ID                   # Grader (pre-selected by scenario set)
veris evaluations status SIM_RUN_ID EVAL_RUN_ID  # Eval progress
  --watch                          # Poll until done
veris evaluations list [SIM_RUN_ID]  # List eval runs
                                   # Without args: all evals for current env
                                   # With arg: evals for specific run
veris evaluations get SIM_RUN_ID EVAL_RUN_ID  # Open in console browser

Reports (veris reports)

veris reports create [SIM_RUN_ID]  # Trigger report generation
  --eval-run-id ID                 # Specific eval run
veris reports status REPORT_ID     # Report progress
  --watch                          # Poll until done
veris reports list                 # List reports
veris reports get REPORT_ID        # Download report HTML
  -o PATH                          # Output path

Profiles (veris profile)

veris profile list                 # List all profiles
veris profile get                  # Show active profile settings
veris profile use [NAME]           # Set active profile (interactive)
veris profile delete NAME          # Remove a profile
veris profile login [API_KEY]      # Alias for 'veris login' (same options)
veris profile org set ORG_ID       # Set organization for active profile
veris profile org rm               # Remove organization from active profile

CI/CD Integration

name: Veris Simulation
on:
  pull_request:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    environment: veris-sim-ci
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install veris-cli
      - name: Build & push
        env:
          VERIS_API_KEY: ${{ secrets.VERIS_API_KEY }}
        run: |
          veris login "$VERIS_API_KEY"
          veris env push --tag ${{ github.sha }}
      - name: Run pipeline
        run: veris run --image-tag ${{ github.sha }} --report > summary.md
      - uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: summary.md

How It Works

veris env create              → Scaffold .veris/ + register environment
veris env push                → Build and push agent image
veris scenarios create        → Generate scenarios + grader
veris simulations create      → Run agent against scenarios
veris evaluations create      → Grade simulation results
veris reports create          → Generate failure analysis report

Or use veris run to chain all steps interactively.

Configuration Files

~/.veris/config.yaml

Global config with named profiles:

active_profile: default
profiles:
  default:
    api_key: vrs_abc123
    backend_url: https://sandbox.api.veris.ai
    console_url: https://console.veris.ai
  staging:
    api_key: vrs_staging
    backend_url: https://sandbox.api.veris.ai
    organization_id: org_abc123

.veris/config.yaml

Project config (created by veris env create):

profiles:
  default:
    environment_id: env_abc123
    environment_name: my-agent

Profiles

The CLI supports named profiles for managing multiple environments. Each profile stores its own API key, backend URL, console URL, organization, and environment ID.

# Login to a named profile (creates it if new, sets it active)
veris login --profile acme --org org_abc123

# Switch active profile
veris profile use acme

# Manage organization
veris profile org set org_abc123
veris profile org rm

Profile resolution: active_profile in config → "default".

Existing flat configs (without profiles key) auto-migrate on first write.

Development

git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli
uv sync
uv run pytest
uv tool install --force -e .

Troubleshooting

• "No API key found" — Run veris login
• Image build fails — Check build logs in the output. Ensure .veris/Dockerfile.sandbox is valid.
• Image push fails — Credentials are fetched automatically. Just retry.

Support

• GitHub Issues: https://github.com/veris-ai/veris-cli/issues
• Email: [e-mail]

License

MIT