verified-executable-search (VES)

Verifier-first executable search runtime: AI candidates produce raw artifacts, the host verifies them into structured Evidence, and search only acts on verified facts. Distribution name: verified-executable-search (import package: ves).

What VES is

• Verifier-first: candidate programs are untrusted; every quality number is recomputed by a host EvidenceVerifier inside a VerificationContext. Candidate self-reported scores are ignored.
• Verification ≠ Judgment: the pipeline establishes facts (VerificationStatus / VerificationResult); feasibility and ranking live in the typed Judge layer (Gate / ObjectiveSpec / JudgeSpec).
• Search on verified facts only: AnchorPolicy selects improvement anchors from VerifiedCandidates; candidates keep Candidate lineage (draft/improve) for audit and repair.

What works in v0.1 (stable API)

• ArtifactContract / SafeArtifactLoader — trusted host specs + race-safe, symlink-safe artifact reads
• VerificationContext / Evidence / EvidenceVerifier — facts, not scores
• Judge / Gate / ObjectiveSpec / JudgeSpec — typed preferences
• VerificationPipeline / VerificationRecord / VerifiedCandidate — verified results and auditable records (schema v1 with fingerprints)
• CLI ves verify (built-in examples + --problem) and ves replay (REPRODUCIBLE / MISMATCH closed loop)

Experimental in v0.1

• SearchEngine + AnchorPolicy (GreedyTop1Policy ~ A arm, BranchDiverseTopKPolicy ~ B arm; interface may evolve in v0.2)
• Real Docker + LLM search adapters (examples/search_demo.py) — source checkout only, not part of the wheel

What remains experimental

• Pareto search semantics (SearchEngine rejects rule=PARETO; v0.1 supports total-order selection only)
• Stochastic verification replay determinism (deferred to v0.2)

Out of scope for v0.1

• Task decomposition / multi-agent orchestration
• MCTS / rollout-based planning
• Full mathematical modeling (mechanism design, statistical significance)

Install

# Python 3.11+
pip install verified-executable-search

For development or the real-LLM/Docker adapters, clone the repository and install from source: pip install .[dev,ml,search].

Quick example

from ves.artifact import ArtifactContract
from ves.judge import Direction, JudgeSpec, ObjectiveSpec

contract = ArtifactContract(
    filename="answer.json",
    media_type="application/json",
    required_fields=("value",),
    numeric_fields=("value",),
)
spec = JudgeSpec(
    objectives=(
        ObjectiveSpec(observation="error", direction=Direction.MINIMIZE),
    ),
)

See docs/writing-a-verifier.md for a complete from-scratch example.

CLI

ves verify solution.json --example knapsack --out record.json
ves replay record.json        # REPRODUCIBLE / MISMATCH

Demos (no LLM required)

python -m examples.demo_adversarial   # six adversarial verdicts
python -m examples.demo_search --mock # mock search evolution

Real search needs VES_LLM_BASE_URL (or AIDE_BASE_URL fallback) + Docker; see docs/ves-demo.md and docs/ves-search.md.

Documentation

• docs/writing-a-verifier.md — new-user tutorial
• docs/ves-cli.md — CLI + replay + schema v1
• docs/ves-demo.md / docs/ves-search.md — hero demos + SearchEngine
• docs/ves-release-roadmap.md — release checklist
• SECURITY.md — threat model, sandbox boundaries, private reporting

CI status

See CI badge / workflow runs — no hand-maintained test counts.

License

MIT