AsAsked

Verifiable Intent proves an agent stayed inside its formal limits. It does not check whether the agent bought what the person actually asked for.

AsAsked closes that gap before the money moves. Between "the cart is ready" and "pay", a guard compares the cart with the person's own words and answers either proceed or ask — with the reasons and a ready question for the person ("You asked to be able to return them; this pair is final sale. Should I still go ahead?"). Checking after the purchase is only useful for disputes; preventing the wrong purchase is the point.

This repository contains the guard, the open sandbox store and benchmark it is measured in, and the original demonstration of the gap in the standard.

---

The pre-checkout guard

pip install asasked

Setup for Claude Code, Claude Desktop, Cursor, VS Code, Codex CLI, Gemini CLI and agent frameworks, model keys and downloads: docs/install.md.

from asasked import Guard

guard = Guard("Black ASICS in size 42, I need to be able to return them.",
              today="Monday, 2 November 2026")
guard.check([{"name": "ASICS Gel-Nimbus 26", "brand": "ASICS", "color": "Black/White",
              "size": "42", "price": 159.00, "quantity": 1,
              "returnable": False, "delivery_days": 3}])
# {"decision": "ask",
#  "reasons": ["Asked to be able to return it, the item is final sale: ASICS Gel-Nimbus 26."],
#  "question": "Before I buy this, please check: ... Should I still go ahead?",
#  "notes": []}

• The cart is plain dicts; only name is required. A fact the person stated that the cart says nothing about (no returnable, no delivery_days) is not guessed: it lands in notes as "the order doesn't say".
• An item that renews and charges again (a subscription, a free trial that turns paid) carries "recurring": {"every": "month", "price": 12.99}; price is the renewal price and defaults to the item's price. Such an item is asked about unless the person asked for a subscription or agreed to one, and an amount limit counts it at the higher of today's and the renewal price: a $0 trial at $10.99 a month is over a $10 limit.
• Colours match by word, in English or Russian ("Triple Black", "Чёрный"). The first colour of a multi-colour item is its main one: "Black/White" is black, "White/Black" is not. Sizes ignore the size system unless both sides name one: "EU 42" is 42, "US 9" is not "EU 9".
• today defaults to the machine's date. Pass it when the person's day differs; deadlines like "by Friday" count from it.
• A Verifiable Intent mandate is optional: guard.check(cart, mandate) adds its amount limits; without one, only the person's own words are checked.
• With the person's signed L2 mandate, Guard.from_mandate(l2, user_jwk) takes the request from its prompt_summary instead of from the agent, so an agent cannot reword it: a changed request breaks the signature. The key is cnf.jwk of an L1 your chain verification has already checked. A mandate with no request in it fails closed: every check asks. Note that the Mastercard Python SDK (356c296) does not write prompt_summary into the signed L2, so its mandates carry no request; the TypeScript implementation does. Pass the agent's own account of the request as claimed= and the guard compares the two fact by fact: where the agent says something the person did not sign (another colour, no returns), the check adds a note. The cart is checked against the signed request either way.
• In AP2 v0.2, asasked.ap2.protect_presentation(guard, create_checkout_presentation, details=...) checks the merchant's checkout JWT before the agent signs the closed Checkout Mandate over it. AP2 v0.2 mandates carry no request, so the person's words come from the conversation, passed by your code; colour, size and returns come from the merchant's catalog through details(item_id).
• Facts are extracted by every model you have a key for (CLAUDE_API_KEY with asasked[anthropic], LLM_API_KEY for Gemini, OPENAI_COMPAT_*), or by Guard(..., providers=("anthropic",)). One model works; two are safer, because a fact they disagree on is visible. A disputed detail can only warn, never block. The exceptions lean towards asking: if either model heard a deadline, a price limit or an item that the order lacks, it counts — models miss these far more often than they invent them.
• The decision is plain deterministic code over those facts and the cart, so every ask can be checked by hand.
• It fails closed: if the request cannot be checked, it asks.
• In the sandbox store it runs in three modes: off, advisory (the agent gets a check_before_checkout tool) and enforced (checkout holds the order until the agent has asked the person).

As an MCP server. pip install "asasked[mcp]" adds asasked-mcp (stdio) with one tool, check_order(request, cart, answers). Its instructions tell the agent to call it before paying. On ask the agent shows the person the question as it is, then checks again with their reply in answers. Facts are extracted once per request, so re-checking a corrected cart is instant. Model keys come from the environment, as above. A server can only answer when asked, so it cannot stop an agent that never calls it. It also has to take the agent's word for everything it is sent: the request and the person's replies in answers both come from the agent, so an agent could reword the request or answer for the person. The MCP server is a second opinion for an agent that wants one, not a lock. If payment goes through your own code, call Guard there instead.

{"mcpServers": {"asasked": {"command": "asasked-mcp",
                            "env": {"CLAUDE_API_KEY": "..."}}}}

Configs for each app: docs/install.md.

In front of your checkout. To make sure the agent can't pay around the guard, put the guard into the payment step itself. While the cart doesn't match the request, checkout is not called. Instead, the agent gets "NOT PAID… show the person this question" along with the question. The person's reply comes from your code, guard.answer(text), never from the agent, so the agent can't agree on the person's behalf. cart turns the checkout arguments into a cart of dicts, as in Guard.check.

pay = guard.protect(checkout, cart=lambda items: items)       # any function, sync or async

Hooks for popular agent frameworks are thin layers over the same gate:

Framework	Install	Hook
Claude Agent SDK	asasked[claude-agent]	hooks={"PreToolUse": [checkout_hook(guard, cart, tool="mcp__shop__checkout")]} from asasked.integrations.claude_agent
OpenAI Agents SDK	asasked[openai-agents]	@function_tool(tool_input_guardrails=[checkout_guardrail(guard, cart)]) from asasked.integrations.openai_agents
LangChain 1.x	asasked[langchain]	create_agent(..., middleware=[CheckoutGuard(guard, cart)]) from asasked.integrations.langchain

The benchmark runs the same agent with and without the guard and reports what the guard caught, what it missed, and how often it stopped a right purchase for nothing — the cost of a guard is as important as its benefit.

python scripts/run_agents.py --tasks all --agents naive,gpt-6-sol --guard both

---

The gap

Verifiable Intent (draft v0.1) cryptographically binds a purchase to a delegation chain: the item is on an approved list, the amount is inside a band, the merchant is allowed. All of that is real and it verifies.

What the person actually said is carried in a single field, prompt_summary. In the spec it is:

• OPTIONAL (spec/credential-format.md)
• annotated "for audit trail" (spec/README.md)
• never read by any verification code — grep prompt_summary across src/verifiable_intent/verification/ returns nothing

So the person's own words are present in the mandate, and nothing compares them to the constraints or to the final cart.

This is not only a prompt_summary problem. The constraint types the standard defines are:

Constraint	Expresses
AllowedMerchantConstraint / AllowedPayeeConstraint	who may be paid
CheckoutLineItemsConstraint	which SKUs, how many
PaymentAmountConstraint / PaymentBudgetConstraint	how much
PaymentRecurrenceConstraint / AgentRecurrenceConstraint	how often
ReferenceConstraint	linkage

There is no way to express "in black", "size 42", "returnable if it doesn't fit", or "here by Saturday" — so a purchase can satisfy every constraint in the standard and still be the wrong purchase.

Why this matters for liability

Issue #15 asks who absorbs the loss when an agent's error produces a purchase the user disputes. The issue body records this answer:

The merchant is liable as they have verified L3 cart_jwt against the L2's user intent

Its provenance is worth stating precisely. The answer is marked [CY] and preceded by a request for @fahads9 to verify it; the issue has no comments, so it has not been confirmed. It is not an official position of the standard, and the same answer notes that a liability shift is still being worked out in the Agent Pay Acceptance framework.

It is nonetheless the answer on record, and it is the intuitive one: someone must reconcile the cart against intent, and the merchant is the obvious candidate. The problem is that the standard gives merchants no mechanism to do it. The duty is assigned; the instrument does not exist.

---

Reproduce it

git clone https://github.com/agent-intent/verifiable-intent.git   # next to this repo
python3 -m venv .venv && source .venv/bin/activate
pip install -e ../verifiable-intent -e ".[dev]"
python scripts/run.py

scripts/run.py builds a genuine VI chain for each scenario — real ES256 signatures, real selective disclosure, verified by the Mastercard SDK, nothing mocked — and then runs the intent check against the same purchase.

The VI SDK is located via VI_SDK_PATH, then ../verifiable-intent, then ./verifiable-intent.

scripts/run.py also needs a model for intent extraction (see .env.example). Three providers are supported and interchangeable: Gemini through Google's own API (LLM_API_KEY, no extra dependencies), Claude (pip install -e ".[anthropic]", then CLAUDE_API_KEY, plus CLAUDE_BASE_URL for a compatible proxy), and any OpenAI-compatible /chat/completions endpoint (OPENAI_COMPAT_*). LLM_PROVIDER picks the default; LLM_AGREE_PROVIDERS picks the pair that is compared. build_conformance_vectors.py does not need a model — it exercises only the cryptographic path.

scripts/run_stress_test.py --provider both runs every phrase through both models and reports the fields where they disagree.

scripts/run_reply_test.py checks the other side: what the person answers when the guard asks. Each of the 48 answers in stress_replies.py follows the guard's question on a v3 task, and the expected decisions were written before the first run. On 0.1.9, 37 of the first 38 went as expected (data/runs/replies-v3-0.1.9.jsonl.gz). Ten conditional answers were added since ("ok, but only if the bag is black"). Once the person has answered, two models that read the answer differently no longer turn a stop into a warning: in an earlier run three refusals got through that way. With that rule, 42 of 48 went as expected and no refusal let a payment through; the other 6 asked again where the person had agreed, mostly to an added bag (data/runs/replies-v3-strict.jsonl.gz).

scripts/run_subscription_test.py runs the 16 hidden-subscription cases of stress_subscription.py: 16 of 16 went as expected (data/runs/subscription.jsonl.gz).

---

What the scenarios show

Eight purchases. In each one the agent stays strictly inside its mandate.

Scenario	Person asked for	Agent bought	VI	Intent
ok-baseline	cheap beginner Babolat	Babolat Drive Junior	passes	matches
price-top	"cheap, for a beginner"	top-of-band pro racket	passes	diverges
brand-swap	"Babolat, I'm used to them"	HEAD	passes	diverges
variant	black ASICS, size 42	white ASICS, size 44	passes	diverges
quantity	"a couple of cans"	six	passes	diverges
refund	"so I can return it"	non-returnable	passes	diverges
deadline	"by Saturday, tournament"	ships in 21 days	passes	diverges
vi-catches	beginner racket under 15 000	out-of-allowlist pro racket	rejects	diverges

Six of eight purchases satisfy Verifiable Intent completely while contradicting what the person asked for. The last row is the control: given a real violation, the standard catches it on its own and this project adds nothing.

In variant the price matches to the cent. In refund and deadline the person's condition cannot be written as a constraint at all.

---

How the check is built

src/asasked/
  catalog.py                    products with attributes VI constraints cannot express
  scenarios.py                  8 scenarios: request -> constraints -> purchase
  vi_runner.py                  builds a real VI chain, verifies it with the Mastercard SDK
  guard.py                      the pre-checkout guard: proceed or ask, with the question
  mandate.py                    the person's request read from their signed L2 mandate
  ap2.py                        the guard before an AP2 agent signs the closed Checkout Mandate
  coherence.py                  extraction + deterministic checking + pre-purchase decision
  llm_extract.py                the language model call — the only AI in the project
  stress_phrases.py             37 real-speech phrases in Russian for testing extraction
  stress_phrases_en.py          40 real-speech phrases in English, same categories
  stress_replies.py             48 free-form answers to the guard's question, with expected decisions
scripts/
  run_stress_test.py            runs them, auto-checking where an answer is unambiguous
  run_reply_test.py             runs the answers through the guard on v3 tasks
  run.py                        8 scenarios, after the fact
  run_prepurchase.py            same scenarios, decided before checkout
  build_conformance_vectors.py  generates conformance_vectors.json
  conformance_ts/               checks them with the TypeScript implementation (Node.js)
  store/                        the sandbox store: catalog_v1.py, store.py (tools), http.py
  harness/                      agent runs: tasks.py (labelled pilot and hard sets), agent.py
                                (loop), user_sim.py (simulated person), run.py (scoring)
scripts/run_agents.py           real models shop in the store, results to runs/*.jsonl
scripts/demo_store.py           a naive agent in the store, no model needed
scripts/demo_reworded.py        an agent that retells the request; the signed mandate catches it
scripts/serve_store.py          the store over HTTP
tests/                          the deterministic check, VI chain, store; no model needed

Extraction and checking are deliberately separate

1. Extraction turns a person's words into facts. This needs a language model, and it is confined to llm_extract.py.
2. Checking compares those facts to the cart. This is ordinary deterministic code with no model in it.

The split is the point, not decoration. "The AI decided so" is worth nothing in a dispute. "The person asked to be able to return it; the item is non-returnable" is worth something, because a human can verify it by hand. The same inputs always produce the same findings — 20 identical calls give one identical result.

Soft preferences

Stress-testing exposed a second gap, this one in our own schema rather than the standard's. People hedge:

черные кроссовки, а вообще если только белые есть — ну ладно, разница не принципиальная (black shoes, though if only white are available — fine, not a big deal)

A schema of hard facts records color=black and the checker blocks a perfectly acceptable purchase. So the model additionally returns soft_fields: which extracted facts the person themselves marked as non-binding. A mismatch on a soft field produces a warning instead of a block, and does not stop the agent. Softness on one field does not weaken the others.

Requirements no field holds

The fact schema is closed: brand, colour, size, price, quantity, returns, delivery. "Gluten-free", "fits an iPhone 15", "not made in China" fit none of them, and a closed schema would drop them without a trace. So the model also returns other_conditions: such requirements in the person's own words, each marked required or optional by the same signals as soft_fields. The model never judges whether the cart meets them. The shop's data has nothing to check them against, so the checker asks the person about each required one and turns each optional one into a note. If only one of the models heard a condition, it becomes a note too. On the 77 stress phrases these appear only where the person really set such a condition ("not black or white", "no worse than HEAD"). Fallbacks the fields already hold ("43 if there's no 42") sometimes show up as optional, and those produce a note, not a stop.

Before the purchase, not after

decide_before_purchase() runs the same check while the agent is still deciding. Blocking findings stop it and produce a question for the person; warnings do not. This is the earlier and more useful entry point — the buyer owns the agent, and being asked beforehand beats discovering the problem in a chargeback.

---

The sandbox store

The eight scenarios show that the gap exists. To measure how often agents fall into it, they need somewhere to shop. asasked.store is a deterministic sandbox store an agent drives through tools: search_products, get_product, add_to_cart, remove_from_cart, view_cart, ask_user, checkout.

• Catalog v1: 456 products (rackets, shoes, balls, bags) built from product families × colour × size × merchant. Traps come from the structure, as in a real store, rather than being planted: the same item in another colour at the same price to the cent, the same item cheaper at a merchant with no returns, or with two-week delivery, a neighbouring model of another level, and variants that are out of stock.
• Search shows name, price and merchant only. The return policy, delivery time and stock are on the product page, so an agent has to look.
• Every order goes through a real Verifiable Intent chain built from the person's mandate (allowed SKUs, amount band, merchants). An order outside the mandate is declined, as a payment network would decline it.
• Every call is logged, so a run can be replayed and published.

python scripts/demo_store.py        # a naive agent: VI passes, the purchase is wrong
python scripts/demo_reworded.py     # the agent drops "black" and "returnable" when it retells the request
python scripts/serve_store.py       # the same store over HTTP on localhost

Running agents in the store

asasked.harness runs a real model as the shopping agent. Every model gets the same neutral system prompt, the same tools and the same step limit (30), so differences in results are about the model, not the scaffold. Claude runs through the Anthropic SDK; anything else through an OpenAI-compatible /chat/completions.

Each task carries a human label: which SKUs and quantity would be the right purchase, or that buying nothing is right, and whether the agent should have asked first. Agents are scored against that label, not against the AsAsked engine. The engine's verdict on the same purchase is recorded next to it, so its own accuracy can be measured separately.

python scripts/run_agents.py --agents claude-sonnet-5,gpt-6-sol,deepseek-v4-pro

Each run is one JSON line in runs/ with the full tool-call transcript. A failed transport (a proxy error, an empty reply) is recorded as an error and not counted as the model's result. --parallel N runs several at once.

Task sets. pilot has 8 tasks, one per trap; frontier models pass almost all of them, so it is a smoke test, not a benchmark. hard has 12 tasks closer to real requests: muddled wording, several conditions at once, soft preferences ("black ideally, white is fine"), multi-item carts that must come from one merchant, a stated budget below the mandate, a request in Russian. Each hard task's label is tested to accept a right purchase and reject the tempting wrong one. The full labelled dataset is still to come.

Dataset v1. 200 tasks built from 13 trap templates over catalog v1 (returnable variant, deadline, cheapest-but- returnable, cheapest-in-time, soft colour, out of stock everywhere, over budget, cans vs packs, brand exclusion, two sizes in one order, vague requests, a starter kit, and Russian requests), split by hash into 100 open and 100 hidden tasks; data/tasks_v1.jsonl holds the open half. Labels are data, not code (harness/labels.py): which items and how many, a total limit, buy nothing, ask first. Every task is checked to have a right purchase that is in stock, inside the mandate and from one merchant, and at least one trap the mandate allows. The generator only proposes, and one person reviewed it. 44 tasks from 5 templates were read row by row (ok / fix / drop; the open half's sheet is data/review_v1_cos.csv); for each remaining template the reviewer answered a concrete "which purchase is right?" question, and the template's label was checked against the answer (data/review_v1_templates.csv). The other 156 tasks were not read one by one. The review changed labels in three templates: two-pair shoe orders now say and require that both pairs are returnable, and "cheapest" requests now require the cheapest listing that meets the other conditions, not any of them. English tasks were read alongside a machine translation made by Claude Sonnet 5 next to the original; labels are translated by code, not by the model. Whether the English requests sound natural is checked by a model, not a person (scripts/check_naturalness.py, results in data/naturalness_v1_model.csv). Dataset v1 has a single reviewer, so no inter-annotator agreement is reported for it; scripts/build_dataset.py --agreement computes it when a second, independent review exists. Its tasks and review sheets for the hidden half live outside the repository, in ../hidden or wherever ASASKED_HIDDEN_DIR points, under the same file names as in data/.

The hidden half of v1 is not really hidden. The generator's seed and the open/hidden split rule are both in this repository, so anyone can rebuild the hidden tasks with scripts/build_dataset.py. Treat all 200 v1 tasks as public: they are for development, not for held-out scores. Dataset v2 draws its hidden tasks from a seed that is kept outside the repository.

python scripts/build_dataset.py
python scripts/run_agents.py --tasks v1-open --agents naive --guard both

Simulated person. A fixed reply to every question made one honest agent fail: it asked "reply yes to confirm" and never got a yes. In hard tasks, ask_user is answered by a model that knows the person's full intent, including what they did not say, and answers only what is asked, as in tau-bench. It does not judge; scoring is still against the label. An agent that ends its turn with a question in plain text instead of calling ask_user ("the cheaper pair is final sale — or the pricier one with returns?") has asked the person, not given up: the question goes to the same simulated person and the run continues. The product page states delivery as "arrives within N days of ordering"; a bare ships_in_days was read by one model as dispatch time, and it would not buy against a deadline.

Results: with and without the guard

Each agent did every task twice: guard off, and guard on with checkout asking it first (--guard both, the guard extracting facts with models, as in normal use). One run per task, scored against the labels. "Wrong" means the payment went through and VI accepted it, but the purchase was not what the person asked for. The v1 runs are from 31 October to 1 November 2026, with AsAsked 0.1.3; the v2 runs are from 3 November 2026, with AsAsked 0.1.4 and the stricter simulated person of commit c3f15a2; the hidden v2 half was run on 4–5 November 2026 with AsAsked 0.1.5 and the same simulator.

The run logs are published: data/runs/v1-open-0.1.3.jsonl.gz and data/runs/v2-open-0.1.4.jsonl.gz, one JSON line per run with the request, the full tool-call transcript, the order, the score and every guard check. The tables below are computed from them:

python scripts/summarize_runs.py data/runs/v1-open-0.1.3.jsonl.gz --markdown
python scripts/summarize_runs.py data/runs/v2-open-0.1.4.jsonl.gz --markdown

A run through Claude Code with the plugin is published as data/runs/claude-code-v2-open-0.1.7.jsonl.gz (scripts/run_claude_code.py, Sonnet, the 50 open v2 tasks, AsAsked 0.1.7). Here the guard is not enforced: the agent decides itself whether to call check_order. It called it before payment in 50 of 50 runs, and all 50 purchases matched the request. In 52 checks the guard stopped 3 right carts and had no wrong ones to catch.

In the v1 run 445 runs of 1,200 failed on the network (the machine lost its connection), all of minimax-m3 and nemotron-3.5-lightning and 45 of deepseek-v4-flash. They were run again, and the v1 log holds the new run in place of each failed one; failed runs were replaced whatever their task, not chosen by result. No run in either log is an error.

Dataset v1, open half, 100 tasks (off → on):

Agent	As asked	Wrong, VI passed	Bought nothing, should have
naive (first search result)	1 → 9	99 → 3	0 → 88
claude-sonnet-5	100 → 100	0 → 0	0 → 0
deepseek-v4-flash	100 → 100	0 → 0	0 → 0
gpt-6-sol	100 → 99	0 → 0	0 → 1
minimax-m3	100 → 100	0 → 0	0 → 0
nemotron-3.5-lightning	95 → 95	1 → 0	4 → 5

The naive agent buys the trap almost every time. The guard stopped 98 of its 99 wrong carts and missed 1. The naive agent passes the guard's question on to the person; twice the simulated person said yes to a wrong cart (it took "a can of 3 balls" for the four cans it wanted), so 3 wrong purchases went through. Real models almost never fell into v1's traps: 1 wrong purchase in 500 without the guard, 0 with it. Across 460 checks on their carts the guard caught that 1 wrong cart, missed none and stopped 9 right ones.

Dataset v2, open half, 50 tasks (off → on). v1 does not tell the models apart, so v2 keeps only trap templates in which strong models made mistakes in a pilot (7 templates: corrections mid-request, an event deadline, three items from one merchant, cheapest-but-returnable, a brand exclusion, two items, deadlines in Russian). The templates were chosen where the guard has something to catch; read the numbers with that in mind. The open half is data/tasks_v2.jsonl; the hidden half is reported below.

Agent	As asked	Wrong, VI passed	Bought nothing, should have
claude-sonnet-5	49 → 49	1 → 0	0 → 1
deepseek-v4-flash	50 → 50	0 → 0	0 → 0
gpt-6-sol	50 → 50	0 → 0	0 → 0
minimax-m3	49 → 48	0 → 0	1 → 2
nemotron-3.5-lightning	37 → 48	10 → 0	3 → 2

Wrong purchases fell from 11 of 250 to 0 of 250. The guard caught 8 wrong carts and missed none. The cost: in 14 of 255 checks the guard stopped a right cart, and the agent had to ask the person one extra question. With the guard on, 5 of 250 runs ended with nothing bought, 4 without it.

The simulated person in these runs is stricter than in the 0.1.3 runs: it never accepts a missing, deferred or substituted item (commit c3f15a2). In the 0.1.3 runs every wrong purchase left with the guard on was a cart the guard had stopped and the simulator then approved. The stricter rule was written after looking at those open tasks, so it is tuned on them. The hidden half below was not used for tuning.

Dataset v2, hidden half, 47 tasks (off → on), AsAsked 0.1.5, runs of 4–5 November 2026, the same simulated person. Neither the guard's rules nor the simulator were tuned on these tasks. The hidden file has 50 tasks; 3 have the same answer as a pilot task, which can be rebuilt from the open code, so they are not counted (dataset_v2.load_v2). One nemotron-3.5-lightning run in each mode timed out and is not counted, so it has 46.

Agent	As asked	Wrong, VI passed	Bought nothing, should have
claude-sonnet-5	41 → 43	2 → 0	4 → 4
deepseek-v4-flash	47 → 47	0 → 0	0 → 0
gpt-6-sol	47 → 46	0 → 0	0 → 1
minimax-m3	45 → 46	1 → 0	1 → 1
nemotron-3.5-lightning	39 → 43	4 → 0	3 → 3

Wrong purchases fell from 7 of 234 to 0 of 234. The guard caught 5 wrong carts and missed none; the other 2 wrong purchases did not recur in the run with the guard (an agent does not act the same way every run). In 12 of 228 checks of right carts the guard stopped one, and the agent had to ask the person one extra question. With the guard on, 9 of 234 runs ended with nothing bought, 8 without it.

The hidden tasks and their log are not published yet. Publishing them would leave no tasks to measure the next guard version on honestly. They will be published once the next guard version has been measured on them, or once a new hidden set exists, whichever comes first. Until then, their SHA-256, so the published files can be checked against these runs:

ce1da227d8da1daa5bf4eef405777adc1f02d9bb87d2d143de71ec1e5d32898c  tasks_v2.jsonl (hidden, all 50 tasks)
670b5c1cb8fe0254000883bfbbde0496a65c48b8561708727eee471e8e55139f  v2-hidden-0.1.5.jsonl (run log, 470 runs)

Compared with the earlier runs. The first results, from runs of 26–30 October 2026 before the guard changes of 31 October, had on the same open halves: on v1, models bought wrong 1 time in 500 with and without the guard (1 missed, 3 right carts stopped in 403 checks) and gave up far more often (minimax-m3 29 of 100 without the guard, gpt-6-sol 6); that v1 run predates the three test-bench fixes of 27 October. On v2, wrong purchases fell from 12 to 1 of 250, with 5 caught, 1 missed and 19 right carts stopped in 255 checks. The logs of those runs are not published; their numbers are given here only for comparison. The v2 run of 31 October – 1 November with AsAsked 0.1.3 and the earlier simulator is published as data/runs/v2-open-0.1.3.jsonl.gz: wrong purchases fell from 9 to 3 of 250, with 11 caught, 0 missed and 12 right carts stopped in 254 checks; all 3 were carts the guard stopped and the simulator approved.

---

Conformance vectors

The standard ships no cross-implementation test vectors, so a second implementation has no way to show it agrees with the reference one. Second implementations do exist — an open TypeScript library is announced in #30 — but none of the discussions includes shared vectors to test them against.

build_conformance_vectors.py generates them: all 8 scenarios, each with a genuinely signed chain (L1, L2, the L2 payment presentation, L3a, L3b), the public JWKs, and the expected verdict — chain validity, constraint satisfaction, and the exact violations.

The output is not committed. L3 mandates expire five minutes after issuance, so a frozen fixture would test "can you reject something stale" within minutes of being written. Run the generator and validate the result immediately.

The signatures are real: they verify with a plain ECDSA implementation and no Mastercard SDK involved, which is exactly what a second implementation would do.

scripts/conformance_ts/ does that with the TypeScript library from #30 (verifiable-intent 0.1.0 on npm). It verifies each chain twice, as the payment network and as the merchant, checks the cart against the checkout constraints, and compares every verdict with expected. All 8 vectors match:

python scripts/build_conformance_vectors.py
cd scripts/conformance_ts && npm install && npm run check

The library's verifyChain checks the payment constraints only; the cart is checked against line_items by the merchant with checkConstraints, as with the Mastercard SDK.

---

Limitations

Read the numbers with these in mind:

• The eight scenarios were written to expose this gap. "Six of eight" is not a measured failure rate in the wild; it shows the failure mode exists and is easy to construct, not how often it occurs.
• The scenario catalog is a toy — 7 products, 2 merchants. The sandbox store has 456 products and 4 merchants, but it is still one sport and a catalog we built ourselves.
• Two models, two languages, 77 phrases. On 2026-10-25 the full stress set (37 Russian + 40 English phrases) ran through Claude Sonnet 5 and Gemini 3 Flash, each phrase extracted by both: 61 phrases with an unambiguous answer all matched, 0 extraction failures, 16 left for manual reading by design. The models disagreed on at least one field in 12 of 77 phrases, mostly deadlines without a reference date and price preferences one model inferred but the person never stated. One disagreement was a real catch: one model read the Russian idiom "красная цена" ("the most I'd pay") as the colour red. Disputed fields warn instead of blocking. Before that run, the earlier prompt made Claude answer greetings in prose instead of JSON (3 failures). Earlier, Gemini alone on the 37 Russian phrases: 0 failures.
• Extraction is not bit-reproducible. Even at temperature=0 Gemini occasionally differs on borderline phrasing; the Claude API takes no sampling parameters at all. The deterministic half is unaffected, and it is the half that belongs in a dispute.
• Soft-field detection depends on the model noticing that a person hedged.
• The answers to the guard's question were written by a model, not by real customers: 48 of them, in three languages. A yes that adds an item ("take the bag too") is still often asked again: the two models read the answer differently, and after an answer the guard does not let that pass.
• A subscription is seen only when the cart says so. An item without recurring counts as a one-time purchase; a shop that does not mark its subscriptions in the cart data cannot be caught. The subscription cases (stress_subscription.py) are a separate set of 16, not part of the v3 tasks, so the published v3 tasks stay as they are.
• The agent benchmark is small and was run once. Each agent did each task once with the guard and once without, so a difference of one or two tasks is noise. Labels have a single reviewer, the person answering questions is a model, and v2's templates were chosen where models make mistakes. Only the open halves are reported.
• On v1, the guard has little to do for real models: they bought wrong once in 500 runs without it. The clear effect is on a naive agent and on one weaker model in v2.
• The simulated person can approve a wrong cart. All wrong purchases with the guard on in v2, and two of the naive agent's in v1, were carts the guard stopped and the simulated person then said yes to.
• The public history starts on 31 October 2026. The project was developed in a private repository before that; commit ids older than this repository, as in the site's changelog, come from that history.
• This is a prototype. It is on PyPI and the deterministic half is covered by tests (pytest, no model or network needed), but at 0.x the API (Guard, check_order) may still change between versions.

---

Relationship to the standard

This repository works on top of Verifiable Intent and does not modify it. The spec is Apache 2.0 and is expected as a sibling checkout, not vendored here.

The question has been put to the standard in issue #37 (September 2026); as of 3 November 2026 it has no replies. A narrower proposal followed in #45 (November 2026): that the L2 prompt_summary carry the person's own words, in one place, disclosed with the checkout mandate. Before it, a search of every issue, pull request and comment in that repository found none addressing the mismatch between stated intent and encoded constraints. The spec itself is unchanged since April 2026 (commit 356c296).