veris-cli 2.24.0

CLI to connect local agents to the Veris backend

Veris CLI

Connect your existing agent to Veris simulations.

The Veris CLI lets you package your agent and run it against simulation scenarios.

Installation

uv tool install veris-cli

Or from source:

uv tool install git+https://github.com/veris-ai/veris-cli.git

Quick Start

1. Login

# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY

This saves your credentials to ~/.veris/config.yaml.

2. Create an Environment

veris env create --name my-agent-env --agent-name "My Agent"

This scaffolds a .veris/ directory and registers your environment on Veris:

• Dockerfile.sandbox — Agent image definition
• veris.yaml — Simulation configuration (services, persona, agent settings)
• .dockerignore — Files excluded from image build

The --name you pass is the agent's display name. The CLI slugifies it to create a target/env name: "My Agent" becomes my-agent-env. This name is used as the top-level key in veris.yaml and as the backend environment name.

3. Configure Your Agent

Edit .veris/Dockerfile.sandbox and .veris/veris.yaml to match your agent. Set secrets:

veris env vars set OPENAI_API_KEY=sk-your-key --secret

4. Build and Push

veris env push

When only one target is defined, the CLI uses it automatically — no flags needed.

5. Generate Scenarios

veris scenarios create --num 10
veris scenarios status <SET_ID> --watch

6. Run Simulations

veris simulations create --scenario-set-id <SET_ID>
veris simulations status <SIM_RUN_ID> --watch

7. Evaluate Results

veris evaluations create --sim-run-id <SIM_RUN_ID>
veris evaluations status <SIM_RUN_ID> <EVAL_RUN_ID> --watch

8. Generate Report

veris reports create <SIM_RUN_ID>
veris reports status <REPORT_ID> --watch
veris reports get <REPORT_ID> -o results.html

Full Pipeline (One Command)

# Interactive — prompts for each step
veris run

# CI — all flags, markdown summary to stdout
veris run --scenario-set-id <SET_ID> --grader-id <GRADER_ID> --report

Command Reference

Top-Level Commands

veris login [API_KEY]              # Authenticate (browser or API key)
  --profile NAME                   # Profile to log in to (sets it active)
  --backend-url URL                # Custom backend URL
  --console-url URL                # Custom console URL
  --org ORG_ID                     # Scope to organization

veris run                          # Full pipeline: simulations → evaluations → reports
  --scenario-set-id ID             # Scenario set (prompts if omitted)
  --grader-id ID                   # Grader (prompts if omitted)
  --env-id ID                      # Environment (uses config if omitted)
  --target NAME                    # Target (auto-detected if only one)
  --image-tag TAG                  # Image tag (default: latest)
  --simulation-timeout N           # Timeout per sim in seconds
  --report                         # Generate HTML report after evaluation

Environment (veris env)

veris env create --name NAME       # Scaffold .veris/ + register environment (NAME = env/target name)
  --agent-name NAME                # Agent display name (stored in veris.yaml agent.name)
veris env push [--tag TAG]         # Build and push image to Veris
  --target NAME                    # Target to push (auto-detected if only one)
veris env list                     # List environments
veris env delete ENV_ID            # Delete environment

Active Target (veris env targets)

veris env targets get               # Show the active target
veris env targets set NAME          # Set the active target for this project
veris env targets list              # List configured targets (from veris.yaml)
veris env targets clear             # Clear the active target

Environment Config (veris env config)

veris env config push              # Upload veris.yaml to backend (no image build)
  --file PATH                      # Custom veris.yaml path (default: .veris/veris.yaml)
  --env-id ID                      # Override environment
  --target NAME                    # Target to upload

Environment Variables (veris env vars)

veris env vars set K=V [K=V ...]   # Set variables
  --secret                         # Mark as secret
  --env-id ID                      # Override environment
  --target NAME                    # Target to set variables for
veris env vars list                # List variables
  --target NAME
veris env vars rm KEY              # Remove a variable
  --target NAME

Scenarios (veris scenarios)

veris scenarios create             # Generate scenario set + grader
  --num N                          # Number of scenarios (default: 5)
  --env-id ID                      # Environment
  --image-tag TAG                  # Image tag
veris scenarios status SET_ID      # Check generation progress
  --watch                          # Poll until done
veris scenarios list               # List scenario sets (with grader column)
veris scenarios get SET_ID         # Open in console browser
veris scenarios delete SET_ID      # Delete scenario set

Simulations (veris simulations)

veris simulations create           # Create simulation run (interactive)
  --scenario-set-id ID             # Scenario set
  --env-id ID                      # Environment
  --simulation-timeout N           # Timeout per sim
  --image-tag TAG                  # Image tag
veris simulations status SIM_RUN_ID  # Run progress + sim list
  --watch                          # Poll until done
  --log                            # Append event stream
veris simulations list             # List simulation runs
  --status STATUS                  # Filter by status
  --env-id ID                      # Filter by environment
veris simulations cancel SIM_RUN_ID  # Cancel a run

Evaluations (veris evaluations)

veris evaluations create           # Trigger grading (interactive)
  --sim-run-id ID                  # Simulation run
  --grader-id ID                   # Grader (pre-selected by scenario set)
veris evaluations status SIM_RUN_ID EVAL_RUN_ID  # Eval progress
  --watch                          # Poll until done
veris evaluations list [SIM_RUN_ID]  # List eval runs
                                   # Without args: all evals for current env
                                   # With arg: evals for specific run
veris evaluations get SIM_RUN_ID EVAL_RUN_ID  # Open in console browser

Reports (veris reports)

veris reports create [SIM_RUN_ID]  # Trigger report generation
  --eval-run-id ID                 # Specific eval run
veris reports status REPORT_ID     # Report progress
  --watch                          # Poll until done
veris reports list                 # List reports
veris reports get REPORT_ID        # Download report HTML
  -o PATH                          # Output path

Profiles (veris profile)

veris profile list                 # List all profiles
veris profile get                  # Show active profile settings
veris profile use [NAME]           # Set active profile (interactive)
veris profile delete NAME          # Remove a profile
veris profile login [API_KEY]      # Alias for 'veris login' (same options)
veris profile org set ORG_ID       # Set organization for active profile
veris profile org rm               # Remove organization from active profile

CI/CD Integration

name: Veris Simulation
on:
  pull_request:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    environment: veris-sim-ci
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install veris-cli
      - name: Build & push
        env:
          VERIS_API_KEY: ${{ secrets.VERIS_API_KEY }}
        run: |
          veris login "$VERIS_API_KEY"
          veris env push --tag ${{ github.sha }}
      - name: Run pipeline
        run: veris run --image-tag ${{ github.sha }} --report > summary.md
      - uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: summary.md

How It Works

veris env create              → Scaffold .veris/ + register environment
veris env push                → Build and push agent image
veris scenarios create        → Generate scenarios + grader
veris simulations create      → Run agent against scenarios
veris evaluations create      → Grade simulation results
veris reports create          → Generate failure analysis report

Or use veris run to chain all steps interactively.

Configuration Files

~/.veris/config.yaml

Global config with named profiles:

active_profile: default
profiles:
  default:
    api_key: vrs_abc123
    backend_url: https://sandbox.api.veris.ai
    console_url: https://console.veris.ai
  staging:
    api_key: vrs_staging
    backend_url: https://sandbox.api.veris.ai
    organization_id: org_abc123

.veris/config.yaml

Project config (created by veris env create). Each target's environment_id is stored under the profile:

active_target: my-cool-agent-env
profiles:
  default:
    targets:
      my-cool-agent-env:
        environment_id: env_abc123
        environment_name: my-cool-agent-env

.veris/veris.yaml

Simulation configuration. Every target is a top-level key — the target name is the backend environment name:

version: "1.0"

my-cool-agent-env:
  services:
    - name: postgres
      config:
        SCHEMA_PATH: /agent/schemas/schema.sql
    - name: slack
      dns_aliases:
        - slack.com
  persona:
    modality:
      type: http
      url: http://localhost:8008/chat
  agent:
    name: My Cool Agent
    code_path: /agent
    entry_point: uv run app
    port: 8008
    environment:
      DATABASE_URL: postgresql://postgres:postgres@localhost:5432/SIMULATION_ID

When only one target is defined, all commands auto-select it.

Multiple Targets (Monorepo)

A single repo can manage multiple agents/environments. Each veris env create adds a target whose name is the backend environment name.

Creating targets

# First agent
veris env create --name my-cool-agent-env --agent-name "My Cool Agent"

# Second agent — appended to the same veris.yaml
veris env create --name customer-support-env --agent-name "Customer Support"

After two creates, .veris/veris.yaml looks like:

version: "1.0"

my-cool-agent-env:
  services: [...]
  persona: ...
  agent:
    name: My Cool Agent
    ...

customer-support-env:
  services: [...]
  persona: ...
  agent:
    name: Customer Support
    ...

Pushing and running

# With one target — auto-detected, no flag needed
veris env push

# With multiple targets — specify which one
veris env push --target customer-support-env
veris run --target customer-support-env
veris env vars set API_KEY=sk-... --target customer-support-env --secret

Setting an active target

Avoid typing --target on every command:

veris env targets set customer-support-env
veris env push                    # pushes customer-support-env
veris run                         # runs customer-support-env
veris env targets get              # shows: customer-support-env
veris env targets list             # lists all targets from veris.yaml
veris env targets clear            # clears the active target

Target resolution

Priority	Source
1	--target flag
2	active_target in .veris/config.yaml
3	Auto-detect: sole target in veris.yaml
4	Error if ambiguous

Per-target Dockerfile

Targets can override the default Dockerfile:

customer-support-env:
  dockerfile: .veris/Dockerfile.support
  agent:
    name: Customer Support
    ...

Falls back to .veris/Dockerfile.sandbox when not specified.

Cross-profile behavior

When you switch profiles (e.g. from dev to staging) and push, the CLI detects that the target exists on another profile and offers to create it on the current one.

Profiles

The CLI supports named profiles for managing multiple backend accounts. Each profile stores its own API key, backend URL, console URL, organization, and environment IDs.

# Login to a named profile (creates it if new, sets it active)
veris login --profile acme --org org_abc123

# Switch active profile
veris profile use acme

# Manage organization
veris profile org set org_abc123
veris profile org rm

Profile resolution: active_profile in config → "default".

Existing flat configs (without profiles key) auto-migrate on first write.

Development

git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli
uv sync
uv run pytest
uv tool install --force -e .

Troubleshooting

• "No API key found" — Run veris login
• Image build fails — Check build logs in the output. Ensure .veris/Dockerfile.sandbox is valid.
• Image push fails — Credentials are fetched automatically. Just retry.

Support

• GitHub Issues: https://github.com/veris-ai/veris-cli/issues
• Email: [e-mail]

License

MIT