eval-mock 0.0.1

A flight simulator for AI agents - deterministic, stateful simulated worlds for agent evaluation.

Eval Mock

English | 简体中文

A flight simulator for AI agents. A deterministic, stateful simulated business world where your agent is tested — before it ever touches a real customer, a real order, or a real inbox.

Status: design complete, implementation starting — no release yet (pre-0.1.0).

The 60-second version

You own an AI agent — say, customer support. Tomorrow you ship a new version (new model, new prompt). Your boss asks: "Will it mishandle refunds?"

Today you have three bad options:

• Test against real systems — you can't. A test refund is a real refund; a test email lands in a real customer's inbox.
• Build a staging environment — expensive, slow, and still incomplete: the agent needs orders and email and CRM behaving as one coherent world.
• Click through by hand and pray — and a pass today proves nothing, because you can't tell agent flakiness from environment flakiness.

Eval Mock gives your agent an exam hall instead:

• Fake, but alive. A simulated inbox, order system, and CRM that actually behave: when the agent sends an email, the simulated customer receives it; when it cancels an order, the order state changes. Nothing real is touched. No API bills.
• Game saves. Every test starts from an identical save. Run the same scenario 100 times — the world is exactly the same every time, so failures measure your agent's variance (pass^k), not infrastructure noise.
• Auto-grading. Task completed? Touched data it shouldn't have? Retried after the payment API "timed out"? Graded from the world's final state and the full call journal — no human watching replays, no second LLM as judge.
• Disaster drills. Declaratively inject timeouts, rate limits, and permission errors, and see whether the agent recovers — before your customers run that experiment for you.

What you get before every release is a scorecard:

50 scenarios × 10 runs each: v1 passes 92%, v2 passes 88% — the regression is entirely "forgets to retry after a timeout", first diverging at step 3.

Ship or hold, you now decide with numbers.

Why a third path

Evaluating an agent requires a world it can act in. The existing paths all fall short:

• Real systems: side effects, cost, and no reproducibility.
• Record & replay: only replays trajectories that already happened; breaks the moment the agent takes a different (equally valid) path.
• Traditional API mocks: stateless canned responses — an email "sent" through the Email tool can never be found by the CRM tool; the same query returns the same answer before and after a cancellation. There is no world, only isolated answers.

Eval Mock is the third path: the world comes first; tools are projections of the world. The agent acts through standard protocols (MCP, plus a native function-calling adapter); every step is journaled, assertable, and exactly replayable.

How it works

1. World Store — one entity store shared by all tools. seed / snapshot / restore / diff are first-class: the same seed rebuilds the same world, any number of times, verified by canonical digests across machines and OSes.
2. Scenario — one declarative YAML file = one exam question: the task given to the agent, the initial world, enabled tools, injected faults, scripted multi-turn user follow-ups, and assertions. Versioned together; distributable; a colleague reruns your exact evaluation with one command.
3. Trajectory & Verdict — a complete tool-call journal plus a machine-readable verdict: final-state assertions, collateral-damage checks (nothing changed outside an allowlist), earliest-failure attribution, aggregated pass rates across repeated runs. OTel span export optional.

Design rule: LLMs may help author environments; at runtime the environment is zero-LLM — pure deterministic code, so results are reproducible by construction. Scripted multi-turn is deterministic too: no LLM plays the user.

An illustrative scenario (DSL may evolve until 0.1.0):

task:
  prompt: "The customer wants a refund for order #1042."
world:
  shop:
    orders:
      - { id: 1042, status: delivered, days_since_delivery: 3 }
tools: [shop, email]
faults:
  - { tool: shop.refund, error: timeout, after_calls: 1 }   # first attempt times out
turns:
  - user: "Actually, store credit is fine too."
assertions:
  - id: refund-completed
    expr: world.shop.orders.exists(o, o.id == 1042 && o.status == "refunded")
  - id: no-collateral
    collateral: { allow_changes: [shop.orders, shop.refunds, email.sent] }

What Eval Mock is not

• Not a model mock. We simulate the tools and the world around your agent; the model under test is your real model.
• Not a code-execution sandbox. SWE-bench-style "run the repo's tests in a container" is a different, well-served problem.
• Not an LLM user simulator. Multi-turn users are scripted and deterministic — by design.
• Not a scoring platform. Verdicts are plain files that feed pytest, CI gates, DeepEval, LangSmith, or your own judge.

Roadmap (v1 scope)

• Determinism kernel: seeded world, virtual clock, canonical digests, snapshot/restore
• Core loop: scenario → MCP server + function-calling adapter → journal → verdict
• Domain packs: Email / Calendar / CRM — plus declarative BYO packs (define your own domain in YAML, no code)
• Fault injection, scripted multi-turn, repeat/pass^k CI gating, third-party verify replay
• 0.1.0 on PyPI

Promises

• Zero telemetry. Runs fully offline — no accounts, no API keys, no network required.
• The open-source version is the full product. No feature-gated enterprise edition.

License

TBD (Apache 2.0 planned).