veris-cli 2.9.0

CLI to connect local agents to the Veris backend

Veris CLI

Connect your existing agent to Veris simulations.

The Veris CLI lets you package your agent as a Docker image and run it against simulation scenarios.

Installation

uv tool install veris-cli

Or from source:

uv tool install git+https://github.com/veris-ai/veris-cli.git

Quick Start

🚀 To run locally Skip to Local Development & Testing to run scenarios on your machine without cloud deployment.

1. Login

# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY

This saves your credentials to ~/.veris/config.yaml.

Note: Login is only required for cloud runs. For local testing with veris run local, you only need Docker and your agent's required environment variables in a .env file.

3. Initialize Your Project

In your agent's project directory:

veris init

This will:

1. Create a .veris/ folder with configuration files
2. Prompt you for an environment name (e.g., "my-customer-support-agent")
3. Create the environment in Veris
4. Save the environment ID to .veris/config.yaml

Files created:

• Dockerfile.sandbox - Docker image template for your agent
• veris.yaml - Simulation configuration (services, persona, agent settings)
• config.yaml - Environment ID (auto-generated, don't edit)

4. Configure Your Agent

Edit the generated files to match your agent:

.veris/Dockerfile.sandbox - Update paths to your agent code:

# Copy your agent's dependencies
COPY pyproject.toml uv.lock /agent/
COPY your_agent_module /agent/your_agent_module

# Install dependencies
WORKDIR /agent
RUN uv sync --frozen --no-dev

.veris/veris.yaml - Configure your agent's entry point, port, and environment variables:

agent:
  code_path: /agent
  entry_point: your_agent_module.main:app  # Update this!
  port: 8000  # Update if your agent uses a different port
  environment:
    DATABASE_URL: postgresql://postgres:postgres@localhost:5432/SIMULATION_ID

For secrets (API keys), use veris env set for cloud runs or a .env file at project root for local runs:

# Cloud runs: stored securely, injected at runtime
veris env set OPENAI_API_KEY=sk-your-key --secret

# Local runs: create .env at project root
echo 'OPENAI_API_KEY=sk-your-key' > .env

5. Build and Push Your Agent Image

# Build and push in one command
veris env push

# Or build only (for testing)
veris env build

This will:

1. Use the environment created during veris init
2. Generate push credentials for the latest tag
3. Automatically build your Docker image (handles buildx on Mac)
4. Automatically push to the registry

Note: On macOS, this uses docker buildx for multi-platform builds targeting linux/amd64 (GKE platform).

6. Generate Scenarios (Optional)

You can write scenarios by hand (see Local Development) or generate them automatically:

# Generate 5 scenarios + graders using Claude Code
veris scenarios generate --num 5

This launches a K8s job that explores your agent's source code and produces test scenarios and graders. Poll status with:

veris scenarios list

7. List Available Scenarios

veris scenarios list

8. Create and Run a Simulation

# Interactive mode (prompts for scenario and environment)
veris run create

# Or specify directly
veris run create --scenario-set-id scenset_abc123 --env-id env_xyz789

9. Monitor Your Run

# Check status
veris run status run_abc123

# Watch status (updates every 3 seconds)
veris run status run_abc123 --watch

# View logs
veris run logs run_abc123

# Follow logs (like tail -f)
veris run logs run_abc123 --follow

10. Evaluate Results (Optional)

Once a run completes and graders are available:

# List available graders
veris eval list

# Trigger evaluation (interactive prompts for run and grader)
veris evaluation-runs create

# Check evaluation status
veris evaluation-runs list --run-id run_abc123
veris evaluation-runs status evalrun_abc123 --run-id run_abc123

11. Cancel a Run (if needed)

veris run cancel run_abc123

Complete Command Reference

Authentication

# Browser-based Google login (recommended)
veris login

# Login with API key (for CI/scripts)
veris login <api-key>

# Specify a custom backend URL (for developers)
veris login [--backend-url https://sandbox.api.veris.ai]

# Login to a specific profile
veris --profile staging login <api-key>

Project Setup

# Initialize .veris/ directory and create environment
veris init [--name "my-agent"]

# If name not provided, you'll be prompted interactively

Profiles

# List all profiles
veris profile list

# Show active profile's settings
veris profile show

# Set the active profile
veris profile use <name>

# Delete a profile
veris profile delete <name>

# Use a specific profile for any command
veris --profile <name> <command>

Environment Management

# Build Docker image only
veris env build [--tag latest] [--no-cache]

# Build and push Docker image
veris env push [--tag latest] [--no-cache]

# List all environments
veris env list [--status ready]

Scenarios

# List scenario sets
veris scenarios list [--env-id <id>]

# Generate scenarios + graders via K8s job
veris scenarios generate [--env-id <id>] [--num 5] [--image-tag latest]

# View scenario set details
veris scenarios get <set-id>

# View a specific scenario within a set
veris scenarios view <set-id> <scenario-id>

Eval (Graders)

# List graders for an environment
veris eval list [--env-id <id>]

Evaluation Runs

# Trigger grading on a completed run
veris evaluation-runs create [--run-id <id>] [--grader-id <id>]

# List evaluation runs for a run
veris evaluation-runs list --run-id <id>

# Get evaluation run status and results
veris evaluation-runs status <eval-run-id> --run-id <id> [--watch]

Runs

# List runs
veris run list [--status <status>]

# Create run (interactive or with flags)
veris run create [--scenario-set-id <id>] [--env-id <id>] [--concurrency 10]

# Get run status
veris run status <run-id> [--watch]

# Get run logs
veris run logs <run-id> [--follow]

# List simulations for a run
veris run simulations <run-id>

# Get simulation result
veris run simulation <run-id> <sim-id>

# Cancel run
veris run cancel <run-id>

# Run scenarios locally in Docker (no cloud deployment needed)
veris run local [scenario...] [--skip-build] [--image <name>] [--platform <platform>] [--scenarios-dir <path>] [--concurrency <n>]

Reports

# Generate a failure analysis report
veris reports generate [--env-id <id>] [--eval-run-id <id>]

# List reports
veris reports list [--env-id <id>]

# Check report status (with optional polling)
veris reports status <report-id> [--watch]

# Download report HTML to file
veris reports get <report-id> [-o <output-path>]

Local Development & Testing

You can run scenarios locally without deploying to Veris cloud infrastructure. This is useful for:

• Testing your agent during development
• Debugging scenarios offline
• Running simulations without network dependency

Prerequisites

1. Docker installed and running
2. .veris/Dockerfile.sandbox and .veris/veris.yaml configured (run veris init first)
3. Any required API keys or environment variables set in .env file or environment (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.)

Quick Start

# Create .env file with your API keys
echo 'OPENAI_API_KEY=sk-your-key' > .env

# Run all scenarios in ./scenarios/ directory
veris run local

# Run specific scenarios
veris run local checkout payment

# Skip Docker build (faster, use existing image)
veris run local --skip-build

Command Options

veris run local [SCENARIO...] [OPTIONS]

Arguments:
  SCENARIO            One or more scenario IDs (default: all scenarios in scenarios-dir)

Options:
  --skip-build           Skip building the Docker image
  --image TEXT           Docker image name (default: veris-sandbox)
  --platform TEXT        Docker platform (default: linux/arm64)
  --scenarios-dir PATH   Path to scenarios folder (default: ./scenarios)
  --concurrency INT      Max parallel containers (default: unbounded)

Examples

# Run all scenarios in ./scenarios/
veris run local

# Run two specific scenarios in parallel
veris run local checkout payment

# Use custom scenarios directory
veris run local --scenarios-dir ./tests/scenarios

# Skip build and use custom image name
veris run local --skip-build --image my-agent-sandbox

# Limit to 2 parallel containers
veris run local --concurrency 2

# Use different platform (e.g., for Intel Macs)
veris run local --platform linux/amd64

How Scenario Resolution Works

If you don't specify scenario arguments, the CLI scans your --scenarios-dir (default: ./scenarios/):

• Files: checkout.yaml → scenario ID checkout
• Directories: checkout/ → scenario ID checkout
• Fallback: If no scenarios found, uses default customer_browse_and_purchase

The scenario ID is passed to the container via SCENARIO_ID environment variable.

Output

After running, you'll see:

1. Summary table: scenario → sim_id → exit code → logs path
2. Complete logs: All .log files from each simulation run

Logs are saved to .veris/logs/<sim_id>/ for each run.

Environment Variables

The command loads environment variables from two sources (in order):

1. agent.environment in .veris/veris.yaml — non-secret config shared with cloud runs
2. .env at project root — local-only secrets (takes precedence)

Each container also receives a SCENARIO_ID environment variable.

Docker Details

Each scenario runs in an isolated container with:

• Mounted .veris/veris.yaml at /config/veris.yaml (read-only)
• Mounted scenarios folder at /scenarios (read-only)
• Mounted logs directory at /sessions (for output)
• Environment variables from .env plus SCENARIO_ID

How It Works

┌─────────────────────────────────────────────────────────────┐
│  1. One-Time Setup                                          │
│     veris init → creates environment + config files         │
│                → saves env_id to .veris/config.yaml         │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  2. Push Updates                                            │
│     Edit code → veris env push → docker build & push        │
│     (can push multiple versions using --tag v1, v2, etc.)   │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  3. Generate Scenarios (optional)                           │
│     veris scenarios generate → Claude Code explores agent   │
│                              → produces scenarios + graders │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  4. Run Simulations                                         │
│     veris run create → Veris spawns your agent in K8s       │
│                     → Runs scenarios against it             │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  5. Evaluate Results (optional)                             │
│     veris evaluation-runs create → grades simulation traces │
│     veris evaluation-runs status → view grading results     │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  6. Generate Reports (optional)                             │
│     veris reports generate → failure analysis report        │
│     veris reports get → download HTML report                │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  7. Monitor & Analyze                                       │
│     veris run status → check progress                       │
│     veris run list → list all runs                          │
│     veris run simulations → view per-simulation results     │
└─────────────────────────────────────────────────────────────┘

Profiles

The CLI supports named profiles for managing multiple Veris environments (e.g., dev, staging, production) — similar to AWS CLI profiles. Each profile stores its own API key, backend URL, and environment ID.

Setting Up Profiles

# Login to the default profile (same as before)
veris login <api-key>

# Login to a named profile
veris --profile staging login <staging-api-key> --backend-url https://staging.api.veris.ai

# Initialize a project under a specific profile
veris --profile staging init --name my-agent-staging

Switching Profiles

# Set the active profile (persists across commands)
veris profile use staging

# Override the active profile for a single command
veris --profile production env list

# See which profiles are configured
veris profile list

Profile Resolution

Profiles are resolved in priority order:

1. --profile <name> CLI flag (per-command override)
2. active_profile field in ~/.veris/config.yaml (set via veris profile use)
3. "default" (implicit fallback)

Backwards Compatibility

Existing flat config files (without profiles key) continue to work. On first write, the flat format is automatically migrated to the profiled format — no manual migration needed.

Configuration Files

~/.veris/config.yaml

Created by veris login. Supports named profiles:

active_profile: default
profiles:
  default:
    api_key: vrs_abc123xyz
    backend_url: https://sandbox.api.veris.ai
  staging:
    api_key: vrs_staging_key
    backend_url: https://staging.api.veris.ai

.veris/config.yaml

Project configuration (auto-generated by veris init). Each profile can have its own environment:

profiles:
  default:
    environment_id: env_abc123
    environment_name: my-agent
  staging:
    environment_id: env_def456
    environment_name: my-agent-staging

.veris/Dockerfile.sandbox

Template for building your agent's Docker image. Important: Build context is project root, so COPY paths are relative to your project root, not .veris/.

.veris/veris.yaml

Simulation configuration including:

• Services your agent uses (with DNS aliases)
• Persona modality (http/ws/email)
• Agent entry point and port

Development

# Clone repo
git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli

# Install dependencies
uv sync

# Run tests
uv run pytest

# Install locally for testing
uv tool install --force .

Troubleshooting

"No API key found"

Run veris login to authenticate via Google, or veris login <your-api-key> with an API key.

Docker build fails

• Make sure Docker is running
• On macOS, Docker Desktop must be installed (required for docker buildx)
• Try veris env build --no-cache to force a clean build

Image push fails

Check that Docker is running and try again. Credentials are fetched automatically — you don't need to run docker login manually.

Support

• GitHub Issues: https://github.com/veris-ai/veris-cli/issues
• Email: [e-mail]

License

MIT