veris-cli 2.14.0

CLI to connect local agents to the Veris backend

Veris CLI

Connect your existing agent to Veris simulations.

The Veris CLI lets you package your agent as a Docker image and run it against simulation scenarios.

Installation

uv tool install veris-cli

Or from source:

uv tool install git+https://github.com/veris-ai/veris-cli.git

Quick Start

🚀 To run locally Skip to Local Development & Testing to run scenarios on your machine without cloud deployment.

1. Login

# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY

This saves your credentials to ~/.veris/config.yaml.

Note: Login is only required for cloud runs. For local testing with veris run local, you only need Docker and your agent's required environment variables in a .env file.

2. Initialize Your Project

In your agent's project directory:

veris init

This will:

1. Create a .veris/ folder with configuration files
2. Prompt you for an environment name (e.g., "my-customer-support-agent")
3. Create the environment in Veris
4. Save the environment ID to .veris/config.yaml

Files created:

• Dockerfile.sandbox - Docker image template for your agent
• veris.yaml - Simulation configuration (services, persona, agent settings)
• config.yaml - Environment ID (auto-generated, don't edit)

3. Configure Your Agent

Edit the generated files to match your agent:

.veris/Dockerfile.sandbox - Update paths to your agent code:

# Copy your agent's dependencies
COPY pyproject.toml uv.lock /agent/
COPY your_agent_module /agent/your_agent_module

# Install dependencies
WORKDIR /agent
RUN uv sync --frozen --no-dev

.veris/veris.yaml - Configure your agent's entry point, port, and environment variables:

agent:
  code_path: /agent
  entry_point: your_agent_module.main:app  # Update this!
  port: 8000  # Update if your agent uses a different port
  environment:
    DATABASE_URL: postgresql://postgres:postgres@localhost:5432/SIMULATION_ID

For secrets (API keys), use veris env set for cloud runs or a .env file at project root for local runs:

# Cloud runs: stored securely, injected at runtime
veris env set OPENAI_API_KEY=sk-your-key --secret

# Local runs: create .env at project root
echo 'OPENAI_API_KEY=sk-your-key' > .env

4. Build and Push Your Agent Image

# Build and push in one command
veris env push

# Or build only (for testing)
veris env build

This will:

1. Use the environment created during veris init
2. Generate push credentials for the latest tag
3. Automatically build your Docker image (handles buildx on Mac)
4. Automatically push to the registry

Note: On macOS, this uses docker buildx for multi-platform builds targeting linux/amd64 (GKE platform).

5. Generate Scenarios (Optional)

You can write scenarios by hand (see Local Development) or generate them automatically:

# Generate 5 scenarios + graders using Claude Code
veris scenarios generate --num 5

This launches a K8s job that explores your agent's source code and produces test scenarios and graders. Poll status with:

veris scenarios list

6. List Available Scenarios

veris scenarios list

7. Create and Run a Simulation

# Interactive mode (prompts for scenario and environment)
veris run create

# Or specify directly
veris run create --scenario-set-id scenset_abc123 --env-id env_xyz789

8. Monitor Your Run

# Check status
veris run status run_abc123

# Watch status (updates every 3 seconds)
veris run status run_abc123 --watch

# View logs
veris run logs run_abc123

# Follow logs (like tail -f)
veris run logs run_abc123 --follow

9. Evaluate Results (Optional)

Once a run completes and graders are available:

# List available graders
veris eval list

# Trigger evaluation (interactive prompts for run and grader)
veris evaluation-runs create

# Check evaluation status
veris evaluation-runs list --run-id run_abc123
veris evaluation-runs status evalrun_abc123 --run-id run_abc123

10. Cancel a Run (if needed)

veris run cancel run_abc123

Complete Command Reference

Authentication

# Browser-based Google login (recommended)
veris login

# Login with API key (for CI/scripts)
veris login <api-key>

# Login and scope to an organization
veris login --org <org-id>

# Specify a custom backend URL (for developers)
veris login [--backend-url https://sandbox.api.veris.ai]

# Login to a specific profile
veris --profile staging login <api-key>

Project Setup

# Initialize .veris/ directory and create environment
veris init [--name "my-agent"]

# If name not provided, you'll be prompted interactively

Profiles

# List all profiles
veris profile list

# Show active profile's settings
veris profile show

# Set the active profile
veris profile use <name>

# Set organization scope for the active profile
veris profile set-org <org-id>

# Clear organization scope (revert to personal)
veris profile set-org

# Delete a profile
veris profile delete <name>

# Use a specific profile for any command
veris --profile <name> <command>

Environment Management

# Build Docker image only
veris env build [--tag latest] [--no-cache]

# Build and push Docker image (locally or via Cloud Build)
veris env push [--tag latest] [--no-cache] [--remote]

# List all environments
veris env list [--status ready]

# Set environment variables (for cloud runs)
veris env set KEY=VALUE [...] [--secret] [--env-id <id>]

# List environment variables
veris env vars [--env-id <id>]

# Remove an environment variable
veris env rm KEY [--env-id <id>]

Scenarios

# List scenario sets
veris scenarios list [--env-id <id>]

# Generate scenarios + graders via K8s job
veris scenarios generate [--env-id <id>] [--num 5] [--image-tag latest]

# View scenario set details
veris scenarios get <set-id>

Eval (Graders)

# List graders for an environment
veris eval list [--env-id <id>]

Evaluation Runs

# Trigger grading on a completed run
veris evaluation-runs create [--run-id <id>] [--grader-id <id>]

# List evaluation runs for a run
veris evaluation-runs list --run-id <id>

# Get evaluation run status and results
veris evaluation-runs status <eval-run-id> --run-id <id> [--watch]

CI

# Run simulation + evaluation pipeline (config-driven)
veris ci run [--scenario-set-id <id>] [--env-id <id>] [--concurrency <n>] [--image-tag <tag>] [--simulation-timeout <seconds>]

Runs

# List runs
veris run list [--status <status>] [--env-id <id>]

# Create run (interactive or with flags)
veris run create [--scenario-set-id <id>] [--env-id <id>] [--concurrency 10] [--simulation-timeout <seconds>]

# Get run status
veris run status <run-id> [--watch]

# Get run logs
veris run logs <run-id> [--follow]

# Cancel run
veris run cancel <run-id>

# Run scenarios locally in Docker (no cloud deployment needed)
veris run local [scenario...] [--skip-build] [--image <name>] [--platform <platform>] [--scenarios-dir <path>] [--concurrency <n>]

Simulations

# List simulations for a run
veris simulations get <run-id>

# Get a specific simulation result
veris simulations get <run-id> <sim-id>

Reports

# Create a failure analysis report
veris reports create [--env-id <id>] [--eval-run-id <id>]

# List reports
veris reports list [--env-id <id>]

# Check report status (with optional polling)
veris reports status <report-id> [--watch]

# Download report HTML to file
veris reports get <report-id> [-o <output-path>]

Local Development & Testing

You can run scenarios locally without deploying to Veris cloud infrastructure. This is useful for:

• Testing your agent during development
• Debugging scenarios offline
• Running simulations without network dependency

Prerequisites

1. Docker installed and running
2. .veris/Dockerfile.sandbox and .veris/veris.yaml configured (run veris init first)
3. Any required API keys or environment variables set in .env file or environment (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.)

Quick Start

# Create .env file with your API keys
echo 'OPENAI_API_KEY=sk-your-key' > .env

# Run all scenarios in ./scenarios/ directory
veris run local

# Run specific scenarios
veris run local checkout payment

# Skip Docker build (faster, use existing image)
veris run local --skip-build

Command Options

veris run local [SCENARIO...] [OPTIONS]

Arguments:
  SCENARIO            One or more scenario IDs (default: all scenarios in scenarios-dir)

Options:
  --skip-build           Skip building the Docker image
  --image TEXT           Docker image name (default: veris-sandbox)
  --platform TEXT        Docker platform (default: linux/amd64)
  --scenarios-dir PATH   Path to scenarios folder (default: ./scenarios)
  --concurrency INT      Max parallel containers (default: unbounded)

Examples

# Run all scenarios in ./scenarios/
veris run local

# Run two specific scenarios in parallel
veris run local checkout payment

# Use custom scenarios directory
veris run local --scenarios-dir ./tests/scenarios

# Skip build and use custom image name
veris run local --skip-build --image my-agent-sandbox

# Limit to 2 parallel containers
veris run local --concurrency 2

# Use different platform (e.g., for Intel Macs)
veris run local --platform linux/amd64

How Scenario Resolution Works

If you don't specify scenario arguments, the CLI scans your --scenarios-dir (default: ./scenarios/):

• Files: checkout.yaml → scenario ID checkout
• Directories: checkout/ → scenario ID checkout
• Fallback: If no scenarios found, uses default customer_browse_and_purchase

The scenario ID is passed to the container via SCENARIO_ID environment variable.

Output

After running, you'll see:

1. Summary table: scenario → sim_id → exit code → logs path
2. Complete logs: All .log files from each simulation run

Logs are saved to .veris/logs/<sim_id>/ for each run.

Environment Variables

The command loads environment variables from two sources (in order):

1. agent.environment in .veris/veris.yaml — non-secret config shared with cloud runs
2. .env at project root — local-only secrets (takes precedence)

Each container also receives a SCENARIO_ID environment variable.

Docker Details

Each scenario runs in an isolated container with:

• Mounted .veris/veris.yaml at /config/veris.yaml (read-only)
• Mounted scenarios folder at /scenarios (read-only)
• Mounted logs directory at /sessions (for output)
• Environment variables from .env plus SCENARIO_ID

CI/CD Integration

Run simulations automatically on every pull request and post results as a PR comment.

Setup

1. Run veris ci run interactively once to select a scenario set — this saves the config to .veris/config.yaml:

veris ci run

2. Commit .veris/config.yaml to your repo (make sure it's not gitignored).

3. Add a VERIS_API_KEY secret to your repo (Settings → Secrets → Actions).

GitHub Actions Workflow

name: Veris Simulation
on:
  pull_request:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    # Restrict to maintainers via GitHub environment protection rules
    environment: veris-sim-ci
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install veris-cli
        run: pip install veris-cli

      - name: Build & push agent image
        env:
          VERIS_API_KEY: ${{ secrets.VERIS_API_KEY }}
        run: |
          veris login "$VERIS_API_KEY"
          veris env push --tag ${{ github.sha }} --remote

      - name: Run simulation & evaluation
        run: |
          veris ci run --image-tag ${{ github.sha }} > veris-summary.md

      - name: Comment on PR
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: veris-summary.md

The environment: veris-sim-ci line uses GitHub environment protection rules — secrets are only exposed to approved maintainers, even on public repos.

CLI Usage

# Everything from config — zero flags needed
veris ci run

# Override image tag (common in CI for PR builds)
veris ci run --image-tag $(git rev-parse --short HEAD)

# Override everything
veris ci run --scenario-set-id X --env-id Y --concurrency 5

Progress is printed to stderr, clean markdown to stdout. The command exits non-zero if the run or evaluation fails.

How It Works

┌─────────────────────────────────────────────────────────────┐
│  1. One-Time Setup                                          │
│     veris init → creates environment + config files         │
│                → saves env_id to .veris/config.yaml         │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  2. Push Updates                                            │
│     Edit code → veris env push → docker build & push        │
│     (can push multiple versions using --tag v1, v2, etc.)   │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  3. Generate Scenarios (optional)                           │
│     veris scenarios generate → Claude Code explores agent   │
│                              → produces scenarios + graders │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  4. Run Simulations                                         │
│     veris run create → Veris spawns your agent in K8s       │
│                     → Runs scenarios against it             │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  5. Evaluate Results (optional)                             │
│     veris evaluation-runs create → grades simulation traces │
│     veris evaluation-runs status → view grading results     │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  6. Generate Reports (optional)                             │
│     veris reports generate → failure analysis report        │
│     veris reports get → download HTML report                │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  7. Monitor & Analyze                                       │
│     veris run status → check progress                       │
│     veris run list → list all runs                          │
│     veris simulations get → view per-simulation results     │
└─────────────────────────────────────────────────────────────┘

Profiles

The CLI supports named profiles for managing multiple Veris environments (e.g., dev, staging, production) — similar to AWS CLI profiles. Each profile stores its own API key, backend URL, organization, and environment ID.

Setting Up Profiles

# Login to the default profile (same as before)
veris login <api-key>

# Login to a named profile with an organization
veris --profile acme login <api-key> --org org_abc123

# Initialize a project under a specific profile
veris --profile acme init --name my-agent

Organizations

Profiles can be scoped to an organization. When set, all created assets (environments, runs) belong to that org, and list commands only show that org's assets.

# Set org during login
veris login --org org_abc123

# Or set/change org on an existing profile
veris profile set-org org_abc123

# Clear org (use personal scope)
veris profile set-org

# View current profile settings including org
veris profile show

You can find your organization ID in the Veris Console under the Admin page.

Switching Profiles

# Set the active profile (persists across commands)
veris profile use acme

# Override the active profile for a single command
veris --profile production env list

# See which profiles are configured
veris profile list

Profile Resolution

Profiles are resolved in priority order:

1. --profile <name> CLI flag (per-command override)
2. active_profile field in ~/.veris/config.yaml (set via veris profile use)
3. "default" (implicit fallback)

Backwards Compatibility

Existing flat config files (without profiles key) continue to work. On first write, the flat format is automatically migrated to the profiled format — no manual migration needed.

Configuration Files

~/.veris/config.yaml

Created by veris login. Supports named profiles:

active_profile: default
profiles:
  default:
    api_key: vrs_abc123xyz
    backend_url: https://sandbox.api.veris.ai
  acme:
    api_key: vrs_acme_key
    backend_url: https://sandbox.api.veris.ai
    organization_id: org_abc123

.veris/config.yaml

Project configuration (auto-generated by veris init). Each profile can have its own environment:

profiles:
  default:
    environment_id: env_abc123
    environment_name: my-agent
  staging:
    environment_id: env_def456
    environment_name: my-agent-staging

.veris/Dockerfile.sandbox

Template for building your agent's Docker image. Important: Build context is project root, so COPY paths are relative to your project root, not .veris/.

.veris/veris.yaml

Simulation configuration including:

• Services your agent uses (with DNS aliases)
• Persona modality (http/ws/email)
• Agent entry point and port

Development

# Clone repo
git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli

# Install dependencies
uv sync

# Run tests
uv run pytest

# Install locally for testing
uv tool install --force .

Troubleshooting

"No API key found"

Run veris login to authenticate via Google, or veris login <your-api-key> with an API key.

Docker build fails

• Make sure Docker is running
• On macOS, Docker Desktop must be installed (required for docker buildx)
• Try veris env build --no-cache to force a clean build

Image push fails

Check that Docker is running and try again. Credentials are fetched automatically — you don't need to run docker login manually.

Support

• GitHub Issues: https://github.com/veris-ai/veris-cli/issues
• Email: [e-mail]

License

MIT