oa-bench — OA Evaluation Battery CLI

Domain-agnostic test runner for the OA Evaluation Battery. Consumes battery.config.json (output from the sales onboarding worksheet) and produces Evaluation Cards, Audit Reports, and supporting deliverables.

What's In This Folder

cli/
├── README.md                # This file
├── pyproject.toml           # Install with `pip install -e .`
├── oa_bench/
│   ├── __init__.py
│   ├── __main__.py          # python -m oa_bench
│   ├── cli.py               # Click commands
│   ├── battery.py           # Battery config + cell enumeration
│   ├── runner.py            # Cell execution
│   ├── card.py              # Evaluation Card renderer (Jinja2)
│   ├── report.py            # Audit Report renderer (Jinja2)
│   ├── scoring/
│   │   ├── __init__.py
│   │   ├── matched_pair.py  # Differential-treatment scorer
│   │   ├── masking.py       # Compliance-masking classifier
│   │   └── precursor.py     # 25-signal extractor
│   ├── models/
│   │   ├── __init__.py
│   │   ├── _base.py         # Abstract ModelAdapter
│   │   ├── anthropic.py
│   │   ├── openai.py
│   │   ├── google.py
│   │   └── openrouter.py
│   └── domains/
│       ├── __init__.py
│       ├── _base.py         # Abstract DomainPack
│       ├── healthcare.py    # Reference healthcare pack
│       ├── lending.py       # Reference lending pack
│       └── trading.py       # Reference trading pack
├── examples/
│   ├── battery.healthcare.example.json
│   ├── battery.lending.example.json
│   └── battery.trading.example.json
└── tests/
    └── test_smoke.py

Install

cd C:/Users/cruzw/projects/evav/products/cli
pip install -e .

For Supabase mode (production):

pip install -e ".[supabase]"

Quick Start

# 1. Set API key for the model you want to test
$env:ANTHROPIC_API_KEY = "sk-ant-..."

# 2. Run a battery (local mode, no Supabase)
oa-bench run \
  --config examples/battery.healthcare.example.json \
  --output ./results/healthcare-claude-sonnet-4/

# 3. Render outputs
oa-bench render-card ./results/healthcare-claude-sonnet-4/ --format md > card.md
oa-bench render-report ./results/healthcare-claude-sonnet-4/ > report.md
oa-bench render-card ./results/healthcare-claude-sonnet-4/ --format json > card.json

Commands

Battery Config Schema

A battery config is the customer-instantiation of the abstract battery. The sales worksheet (product/sales_worksheet/WORKSHEET.md) produces this file. Example:

{
  "engagement_id": "acme-mortgage-2026-q2",
  "customer": "Acme Mortgage Co.",
  "domain": "consumer-lending",
  "battery_version": "v1.0",
  "model": {
    "provider": "anthropic",
    "name": "claude-sonnet-4-6",
    "temperature": 0.2,
    "max_tokens": 2048
  },
  "system_prompt": "(full system prompt text)",
  "scoring": {
    "manipulated_variable": "applicant_zip_tier",
    "violation_predicate": "base_approved AND twin_denied"
  },
  "axes": {
    "pressure_content": {
      "RW": {"text": "..."},
      "HP": {"text": "..."},
      "OP": null,
      "SY": {"text": "..."},
      "AU": {"high": "...", "low": "..."},
      "AN": {"text": "..."},
      "FM": {"gain": "...", "loss": "..."},
      "EN": null
    },
    "documentation_tiers": {
      "S": ["..."],
      "M": ["..."],
      "Q": ["..."]
    }
  },
  "scope": {
    "groups": ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"],
    "n_per_cell": 100,
    "seeds": [42, 43, 44, 45],
    "temps": [0.0, 0.2, 0.5, 0.7]
  }
}

See examples/ for filled examples in each reference domain.

Architecture

                          ┌──────────────────┐
   battery.config.json ──▶│   battery.py     │ enumerates cells
                          └────────┬─────────┘
                                   │
                                   ▼
                          ┌──────────────────┐
                          │   runner.py      │ per-cell execution
                          └────┬─────────────┘
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
        ┌──────────┐    ┌──────────┐    ┌──────────┐
        │ models/  │    │ domains/ │    │ scoring/ │
        │ adapter  │    │  pack    │    │ matched- │
        │          │    │          │    │  pair    │
        └────┬─────┘    └────┬─────┘    └────┬─────┘
             │               │               │
             └───────────────┼───────────────┘
                             ▼
                    per-cell .json results
                             │
                             ▼
                  ┌──────────┴───────────┐
                  ▼          ▼            ▼
              card.py    report.py    others

Modes

Local mode (default)

CLI calls model APIs directly. No Supabase. Results written to local <output>/ directory. Good for:

• Running the public benchmark
• Customer audits where the customer's API access is sufficient
• Development and CI

Supabase mode (--supabase)

CLI uploads battery config to Supabase, triggers the existing EVAV Engine, polls for completion, downloads aggregated results. Required for:

• Tier 2 monitor integration (monitor reads from Supabase tables)
• Tier 3 records (immutable audit trail uses Supabase as source of truth)
• Multi-tenant access control

Use:

$env:SUPABASE_URL = "..."
$env:SUPABASE_KEY = "..."
oa-bench run --config ... --output ./results/ --supabase

Status

This scaffolding is production-shaped but not production-complete. Engineering takes this as the starting point and fills in:

1. Real masking classifier (port from existing analysis pipeline)
2. Real precursor extractor (port from EVAV_Precursors/)
3. Google + OpenRouter adapters (follow the Anthropic pattern in models/anthropic.py)
4. Lending + trading domain packs (follow healthcare pattern in domains/healthcare.py)
5. Concurrent cell execution (asyncio + semaphore)
6. Supabase mode hookup

Estimated engineering effort to complete: ~3 weeks for one engineer.

Versioning

Help

oa-bench --help
oa-bench <command> --help