salvo-ai 0.1.1

Test framework for multi-step AI agents

SALVO

Test your AI agents like you test your code.

Define scenarios in YAML. Run them N times. Get pass rates, costs, and failure breakdowns. When runs fail, replay recorded traces without spending another cent.

pip install salvo-ai[openai]

Your agent passes once. Does it pass 50 times in a row?

Why Salvo · Quick Start · How It Works · Assertions · Commands · Adapters

---

Why Salvo

You changed a system prompt. You tweaked the tools. You swapped models. Does your agent still work?

Single prompt-response evals don't catch multi-step failures. Your agent might call tools in the wrong order, hallucinate arguments, or bail out after one turn. You won't know until a user hits it.

Salvo fixes that. Write a YAML scenario, give it tools with mock responses, define what "correct" looks like with assertions, and run it 50 times. Get a pass rate. Sleep at night.

$ salvo run scenarios/ -n 10

  coding-assistant   gpt-4o   10/10 passed   score: 1.00   $0.12   avg 2.3s
  search-agent       gpt-4o    8/10 passed   score: 0.85   $0.08   avg 1.8s
  data-pipeline      gpt-4o    9/10 passed   score: 0.94   $0.15   avg 3.1s

  3 scenarios | 27/30 passed | avg score: 0.93

When something breaks, don't re-run and re-pay. Replay the recorded trace and re-evaluate with updated assertions. Zero API calls. Zero cost.

---

Quick Start

1. Install

pip install salvo-ai[openai]       # For OpenAI models
pip install salvo-ai[anthropic]    # For Anthropic models
pip install salvo-ai[all]          # Both

2. Initialize

salvo init

Creates a starter project:

my-project/
  salvo.yaml              # Project config
  scenarios/
    example.yaml          # Example scenario with tools and assertions
  tools/
    example_tool.yaml     # Reusable tool definition

3. Write a Scenario

description: "Agent fixes a bug by reading source, finding the issue, and applying a fix"

adapter: openai
model: gpt-4o

system_prompt: |
  You are a senior software engineer. Read the failing test,
  inspect the source, write a fix, then run tests to verify.

prompt: |
  The test in tests/test_math.py is failing. Please investigate and fix the bug.

tools:
  - name: file_read
    description: "Read the contents of a file"
    parameters:
      type: object
      properties:
        path:
          type: string
      required: [path]
    mock_response: |
      def add(a, b):
          return a - b  # BUG: should be a + b

  - name: file_write
    description: "Write contents to a file"
    parameters:
      type: object
      properties:
        path: { type: string }
        content: { type: string }
      required: [path, content]
    mock_response: "File written successfully"

  - name: run_tests
    description: "Run the test suite"
    parameters:
      type: object
      properties:
        test_path: { type: string }
    mock_response: "1 passed, 0 failed"

assertions:
  - type: tool_called
    tool: file_read
    required: true

  - type: tool_sequence
    mode: in_order
    sequence: [file_read, file_write, run_tests]
    weight: 2.0

  - type: output_contains
    value: "+"

threshold: 0.8

4. Run

# Run once
salvo run scenarios/example.yaml

# Run 10 times for reliability stats
salvo run scenarios/ -n 10

# Record traces for replay
salvo run scenarios/ -n 10 --record

---

How It Works

 YAML Scenario          Adapter            N-Trial Runner
┌──────────────┐    ┌──────────────┐    ┌──────────────────┐
│ system_prompt │───>│   OpenAI /   │───>│  Run scenario N  │
│ prompt        │    │  Anthropic   │    │  times, collect  │
│ tools + mocks │    │              │    │  traces + usage  │
│ assertions    │    └──────────────┘    └────────┬─────────┘
└──────────────┘                                  │
                                                  v
                    ┌──────────────┐    ┌──────────────────┐
                    │   Results    │<───│  Assertion Engine │
                    │              │    │                   │
                    │ pass rates   │    │ tool_sequence     │
                    │ scores       │    │ jmespath queries  │
                    │ cost / time  │    │ LLM judge (k=3)  │
                    │ breakdowns   │    │ cost/latency caps │
                    └──────────────┘    └──────────────────┘

Scenarios define what to test: a system prompt, user prompt, available tools with mock responses, and assertions that define "correct."

Adapters send the conversation to your model. Tools are presented to the model, and mock responses are returned when the model calls them. Multi-turn loops continue until the model stops calling tools or hits the safety limit.

Assertions evaluate the trace: Did it call the right tools? In the right order? Did the output contain what we expected? Did it stay under budget?

Traces capture everything: every message, every tool call, token counts, latency, cost. Record them to replay later without API calls.

---

Assertions

Salvo ships with 7 assertion types that cover the full evaluation spectrum.

Quick Assertions

The simplest way to check common things:

# Did the agent call a specific tool?
- type: tool_called
  tool: file_read
  required: true          # Hard fail if not called

# Does the output contain a string?
- type: output_contains
  value: "fixed"

Tool Sequence

Verify your agent follows the correct workflow:

- type: tool_sequence
  mode: in_order          # exact | in_order | any_order
  sequence:
    - file_read
    - file_write
    - run_tests
  weight: 2.0             # Count this assertion double

Mode	Behavior
exact	Actual calls must match expected exactly
in_order	Expected tools appear in order (extra calls OK)
any_order	All expected tools called (any order, extras OK)

JMESPath Queries

Query the full trace data structure with JMESPath:

# Check a tool was called with specific arguments
- type: jmespath
  expression: "tool_calls[?name=='file_write'].arguments.path | [0]"
  operator: exists

# Check token usage
- type: jmespath
  expression: "metadata.total_tokens"
  operator: lt
  value: 5000

# Check the model's final response
- type: jmespath
  expression: "response.content"
  operator: contains
  value: "successfully"

Available operators

Operator	Description
eq	Equal
ne	Not equal
gt	Greater than
gte	Greater than or equal
lt	Less than
lte	Less than or equal
contains	String/list contains value
regex	Matches regular expression
exists	Value is not null/missing

Queryable trace structure

{
  "response": {
    "content": "final assistant message",
    "finish_reason": "stop"
  },
  "turns": [
    { "role": "assistant", "content": "...", "tool_calls": [...] },
    { "role": "tool_result", "content": "...", "tool_name": "file_read" }
  ],
  "tool_calls": [
    { "name": "file_read", "arguments": { "path": "test.py" } },
    { "name": "file_write", "arguments": { "path": "src.py", "content": "..." } }
  ],
  "metadata": {
    "model": "gpt-4o",
    "provider": "openai",
    "cost_usd": 0.012,
    "latency_seconds": 2.3,
    "input_tokens": 1500,
    "output_tokens": 500,
    "total_tokens": 2000,
    "turn_count": 3,
    "finish_reason": "stop"
  }
}

Cost & Latency Limits

Hard caps on spend and time:

- type: cost_limit
  max_usd: 0.50

- type: latency_limit
  max_seconds: 30.0

LLM Judge

When assertions can't be expressed as rules, use an LLM judge with k-vote consensus:

- type: judge
  criteria:
    - name: correctness
      description: "Did the agent fix the actual bug?"
      weight: 2.0
    - name: methodology
      description: "Did the agent read before writing?"
      weight: 1.0
  judge_model: gpt-4o-mini   # Cheap model for judging
  k: 3                        # 3 votes, majority wins
  threshold: 0.7

---

Commands

Command	What it does
salvo init	Create a starter project with example scenario
salvo validate <path>	Validate scenario files (catches typos, bad schemas)
salvo run <path> [-n N]	Run scenarios N times, show pass rates and scores
salvo report <run-id>	Show detailed breakdown of a completed run
salvo replay [trace-id]	Replay a recorded trace without API calls
salvo reeval [trace-id]	Re-evaluate a trace with updated assertions

Key Flags

# Run 20 trials with trace recording
salvo run scenarios/ -n 20 --record

# Run with a specific model
salvo run scenarios/ --model gpt-4o-mini

# Validate before running
salvo validate scenarios/

# Replay the latest recorded trace
salvo replay

# Re-evaluate with a modified scenario
salvo reeval --scenario scenarios/updated.yaml

---

Adapters

Salvo supports OpenAI and Anthropic out of the box, with a plugin system for custom adapters.

Adapter	Models	Install
openai	gpt-4o, gpt-4o-mini, o1, o3, etc.	pip install salvo-ai[openai]
anthropic	claude-4, claude-sonnet, etc.	pip install salvo-ai[anthropic]
Custom	Any model	Subclass BaseAdapter

Custom Adapters

# In your scenario
adapter: my_package.adapters.LlamaAdapter
model: llama-3.1-70b

# my_package/adapters.py
from salvo.adapters.base import BaseAdapter

class LlamaAdapter(BaseAdapter):
    async def run(self, scenario, resolved_extras):
        # Your implementation here
        ...

---

Configuration

salvo.yaml

Project-level defaults that scenarios can override:

# Default model settings
default_adapter: openai
default_model: gpt-4o

# Where to find scenarios
scenarios_dir: scenarios

# Where to store run data
storage_dir: .salvo

# Judge defaults
judge:
  adapter: openai
  model: gpt-4o-mini
  k: 3
  temperature: 0.0
  default_threshold: 0.8

Reusable Tools

Extract tool definitions into YAML files and include them across scenarios:

# tools/file_tools.yaml
name: file_read
description: "Read file contents"
parameters:
  type: object
  properties:
    path: { type: string }
  required: [path]
mock_response: "file contents here"

# In your scenario
tools:
  - !include tools/file_tools.yaml
  - name: inline_tool
    description: "Defined right here"
    mock_response: "ok"

---

Trace Recording & Replay

Record complete traces during runs, then replay and re-evaluate without making API calls.

# Record traces
salvo run scenarios/ --record

# Replay the last recorded trace (no API calls)
salvo replay

# Replay a specific trace
salvo replay abc123

# Re-evaluate with updated assertions (no API calls)
salvo reeval abc123 --scenario scenarios/stricter.yaml

This is the money feature for iteration. Change your assertions, re-evaluate the same trace, see if your new scoring logic works. No API costs. Instant feedback.

---

Validation

Salvo catches problems before you spend money on API calls:

$ salvo validate scenarios/bad.yaml

  scenarios/bad.yaml
    [error] Line 1: Unknown field 'modle'. Did you mean 'model'?
    [error] Assertion 0: Unknown type 'tool_calld'. Did you mean 'tool_called'?

  0/1 scenarios valid

Validates schema, field names (with typo suggestions), assertion types, tool definitions, and threshold ranges.

---

Architecture

src/salvo/
  adapters/          # OpenAI, Anthropic, custom adapter registry
  cli/               # Typer CLI: init, validate, run, report, replay, reeval
  evaluation/        # Assertion engine: normalizer, scorer, 5 evaluators
  execution/         # Multi-turn runner, N-trial runner, trace capture
  loader/            # YAML loading, !include resolution, validation
  models/            # Pydantic v2 models: Scenario, RunResult, RunTrace
  recording/         # Trace recorder, replayer, redaction
  scaffold/          # Project templates for salvo init
  storage/           # JSON store for runs, traces, manifests

---

License

MIT License. See LICENSE for details.

---

Your agent works once. Salvo tells you if it works every time.