veris-cli 2.2.0

CLI to connect local agents to the Veris backend

Veris CLI

Connect your existing agent to Veris simulations.

The Veris CLI lets you package your agent as a Docker image and run it against simulation scenarios.

Installation

uv tool install veris-cli

Or from source:

uv tool install git+https://github.com/veris-ai/veris-cli.git

Quick Start

🚀 To run locally Skip to Local Development & Testing to run scenarios on your machine without cloud deployment.

1. Login

# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY

This saves your credentials to ~/.veris/config.yaml.

Note: Login is only required for cloud runs. For local testing with veris run local, you only need Docker and your agent's required environment variables in a .env file.

3. Initialize Your Project

In your agent's project directory:

veris init

This will:

1. Create a .veris/ folder with configuration files
2. Prompt you for an environment name (e.g., "my-customer-support-agent")
3. Create the environment in Veris
4. Save the environment ID to .veris/config.yaml

Files created:

• Dockerfile.sandbox - Docker image template for your agent
• veris.yaml - Simulation configuration (services, persona, agent settings)
• .env.simulation - Environment variables for your agent
• config.yaml - Environment ID (auto-generated, don't edit)

4. Configure Your Agent

Edit the generated files to match your agent:

.veris/Dockerfile.sandbox - Update paths to your agent code:

# Copy your agent's dependencies
COPY pyproject.toml uv.lock /agent/
COPY your_agent_module /agent/your_agent_module

# Install dependencies
WORKDIR /agent
RUN uv sync --frozen --no-dev

.veris/veris.yaml - Configure your agent's entry point and port:

agent:
  code_path: /agent
  entry_point: your_agent_module.main:app  # Update this!
  port: 8000  # Update if your agent uses a different port

.veris/.env.simulation - Add your agent's environment variables:

OPENAI_API_KEY=sk-your-key
DATABASE_URL=postgresql://...

5. Build and Push Your Agent Image

# Build and push in one command
veris env push

# Or build only (for testing)
veris env build

This will:

1. Use the environment created during veris init
2. Generate push credentials for the latest tag
3. Automatically build your Docker image (handles buildx on Mac)
4. Automatically push to the registry

Note: On macOS, this uses docker buildx for multi-platform builds targeting linux/amd64 (GKE platform).

6. Generate Scenarios (Optional)

You can write scenarios by hand (see Local Development) or generate them automatically:

# Generate 5 scenarios + graders using Claude Code
veris scenarios generate --num 5

This launches a K8s job that explores your agent's source code and produces test scenarios and graders. Poll status with:

veris scenarios list

7. List Available Scenarios

veris scenarios list

8. Create and Run a Simulation

# Interactive mode (prompts for scenario and environment)
veris run create

# Or specify directly
veris run create --scenario-set-id scenset_abc123 --env-id env_xyz789

9. Monitor Your Run

# Check status
veris run status run_abc123

# Watch status (updates every 3 seconds)
veris run status run_abc123 --watch

# View logs
veris run logs run_abc123

# Follow logs (like tail -f)
veris run logs run_abc123 --follow

10. Evaluate Results (Optional)

Once a run completes and graders are available:

# List available graders
veris eval list

# Trigger evaluation (interactive prompts for run and grader)
veris evaluation-runs create

# Check evaluation status
veris evaluation-runs list --run-id run_abc123
veris evaluation-runs status evalrun_abc123 --run-id run_abc123

11. Cancel a Run (if needed)

veris run cancel run_abc123

Complete Command Reference

Authentication

# Browser-based Google login (recommended)
veris login

# Login with API key (for CI/scripts)
veris login <api-key>

# Specify a custom backend URL (for developers)
veris login [--backend-url https://sandbox.api.veris.ai]

Project Setup

# Initialize .veris/ directory and create environment
veris init [--name "my-agent"]

# If name not provided, you'll be prompted interactively

Environment Management

# Build Docker image only
veris env build [--tag latest] [--no-cache]

# Build and push Docker image
veris env push [--tag latest] [--no-cache]

# List all environments
veris env list [--status ready]

Scenarios

# List scenario sets
veris scenarios list [--env-id <id>]

# Generate scenarios + graders via K8s job
veris scenarios generate [--env-id <id>] [--num 5] [--image-tag latest]

Eval (Graders)

# List graders for an environment
veris eval list [--env-id <id>]

Evaluation Runs

# Trigger grading on a completed run
veris evaluation-runs create [--run-id <id>] [--grader-id <id>]

# List evaluation runs for a run
veris evaluation-runs list --run-id <id>

# Get evaluation run status and results
veris evaluation-runs status <eval-run-id> --run-id <id> [--watch]

Runs

# Create run (interactive or with flags)
veris run create [--scenario-set-id <id>] [--env-id <id>] [--concurrency 10]

# Get run status
veris run status <run-id> [--watch]

# Get run logs
veris run logs <run-id> [--follow]

# Cancel run
veris run cancel <run-id>

# Run scenarios locally in Docker (no cloud deployment needed)
veris run local [scenario...] [--skip-build] [--image <name>] [--platform <platform>] [--scenarios-dir <path>] [--concurrency <n>]

Local Development & Testing

You can run scenarios locally without deploying to Veris cloud infrastructure. This is useful for:

• Testing your agent during development
• Debugging scenarios offline
• Running simulations without network dependency

Prerequisites

1. Docker installed and running
2. .veris/Dockerfile.sandbox and .veris/veris.yaml configured (run veris init first)
3. Any required API keys or environment variables set in .env file or environment (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.)

Quick Start

# Create .env file with your API keys
echo 'OPENAI_API_KEY=sk-your-key' > .env

# Run all scenarios in ./scenarios/ directory
veris run local

# Run specific scenarios
veris run local checkout payment

# Skip Docker build (faster, use existing image)
veris run local --skip-build

Command Options

veris run local [SCENARIO...] [OPTIONS]

Arguments:
  SCENARIO            One or more scenario IDs (default: all scenarios in scenarios-dir)

Options:
  --skip-build           Skip building the Docker image
  --image TEXT           Docker image name (default: veris-sandbox)
  --platform TEXT        Docker platform (default: linux/arm64)
  --scenarios-dir PATH   Path to scenarios folder (default: ./scenarios)
  --concurrency INT      Max parallel containers (default: unbounded)

Examples

# Run all scenarios in ./scenarios/
veris run local

# Run two specific scenarios in parallel
veris run local checkout payment

# Use custom scenarios directory
veris run local --scenarios-dir ./tests/scenarios

# Skip build and use custom image name
veris run local --skip-build --image my-agent-sandbox

# Limit to 2 parallel containers
veris run local --concurrency 2

# Use different platform (e.g., for Intel Macs)
veris run local --platform linux/amd64

How Scenario Resolution Works

If you don't specify scenario arguments, the CLI scans your --scenarios-dir (default: ./scenarios/):

• Files: checkout.yaml → scenario ID checkout
• Directories: checkout/ → scenario ID checkout
• Fallback: If no scenarios found, uses default customer_browse_and_purchase

The scenario ID is passed to the container via SCENARIO_ID environment variable.

Output

After running, you'll see:

1. Summary table: scenario → sim_id → exit code → logs path
2. Complete logs: All .log files from each simulation run

Logs are saved to .veris/logs/<sim_id>/ for each run.

Environment Variables

The command automatically:

• Loads .env from project root if it exists
• Passes all environment variables from .env to containers
• Adds SCENARIO_ID environment variable for each scenario

Docker Details

Each scenario runs in an isolated container with:

• Mounted .veris/veris.yaml at /config/veris.yaml (read-only)
• Mounted scenarios folder at /scenarios (read-only)
• Mounted logs directory at /sessions (for output)
• Environment variables from .env plus SCENARIO_ID

How It Works

┌─────────────────────────────────────────────────────────────┐
│  1. One-Time Setup                                          │
│     veris init → creates environment + config files         │
│                → saves env_id to .veris/config.yaml         │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  2. Push Updates                                            │
│     Edit code → veris env push → docker build & push        │
│     (can push multiple versions using --tag v1, v2, etc.)   │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  3. Generate Scenarios (optional)                           │
│     veris scenarios generate → Claude Code explores agent   │
│                              → produces scenarios + graders │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  4. Run Simulations                                         │
│     veris run create → Veris spawns your agent in K8s       │
│                     → Runs scenarios against it             │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  5. Evaluate Results (optional)                             │
│     veris evaluation-runs create → grades simulation traces │
│     veris evaluation-runs status → view grading results     │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  6. Monitor & Analyze                                       │
│     veris run status → check progress                       │
│     veris run logs → view events                            │
└─────────────────────────────────────────────────────────────┘

Configuration Files

~/.veris/config.yaml

Created by veris login:

api_key: vrs_abc123xyz
backend_url: https://sandbox.api.veris.ai

.veris/config.yaml

Project configuration (auto-generated by veris init):

environment_id: env_abc123
environment_name: my-agent

.veris/Dockerfile.sandbox

Template for building your agent's Docker image. Important: Build context is project root, so COPY paths are relative to your project root, not .veris/.

.veris/veris.yaml

Simulation configuration including:

• Services your agent uses (with DNS aliases)
• Persona modality (http/ws/email)
• Agent entry point and port

.veris/.env.simulation

Environment variables loaded into your agent container at runtime.

Development

# Clone repo
git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli

# Install dependencies
uv sync

# Run tests
uv run pytest

# Install locally for testing
uv tool install --force .

Troubleshooting

"No API key found"

Run veris login to authenticate via Google, or veris login <your-api-key> with an API key.

Docker build fails

• Make sure Docker is running
• On macOS, Docker Desktop must be installed (required for docker buildx)
• Try veris env build --no-cache to force a clean build

Image push fails

Check that Docker is running and try again. Credentials are fetched automatically — you don't need to run docker login manually.

Support

• GitHub Issues: https://github.com/veris-ai/veris-cli/issues
• Email: [e-mail]

License

MIT