zeroproof

The ZeroProof Python SDK. One package, two importable modules:

• zeroproof: the platform client. OTLP trace ingest and trace-dataset listing against the token gate.
• zeroproof.simulations: post-training data for an agent. (Was the separate top-level package zeroproof_simulations; that name still imports for two releases with a deprecation warning.) Give it the agent's traces, or its tools and system prompt; it simulates the situations, the people, and the world, plays the agent through multi-turn tool-calling conversations, and returns rows for your grader.

This repo absorbed the zeroproof-simulations package; zeroproof-simulations on PyPI is deprecated in favor of zeroproof.

Releases of zeroproof before 0.3 were an unrelated encrypted agent-to-agent messaging client. That code was removed in 0.04; pin zeroproof<0.3 if you still depend on it.

Two ways in, one engine. Give it the agent's tools and system prompt and it samples situations across everything that agent can be asked. Give it graded traces as well and it aims the budget at the situations that fail in production, so new rows land where the agent is weak and carry both the failure and the fixed version. Every row is a full conversation: user turns, agent turns, tool calls, tool results, scheduled faults. Rows come back ungraded; your grader decides what good means. Default explore: one unique situation per row. How it thinks: docs/simulations.md.

How a row gets made

A situation is drawn across the world axes (from the agent's tools) and the human axes (from a separate writer). It fills a cell in the coverage grid, nudges the five search arms, and the agent plays it against a world that breaks on schedule. The row that comes out splits into Task, Rollout, Judgment, and Marker, and every training target is a projection of some of those four. The interactive version, running on real rows, is at zeroproofai.com/docs/engine.

Overview

simulate() is a pipeline.

1. Read the agent. Tools and system prompt. That is the spec of the world.
2. Build a fake world from those tools. Objects, plausible results, and faults (timeout, deny, junk).
3. Write users. A separate writer (same hosted model, different prompt, no agent policy) samples situations across tools, stance, history, and so on.
4. Pick the diverse ones. Embeddings plus a bit of noise so the batch is not 200 copies of the same prompt.
5. Play the agent. It talks, calls tools, gets results, talks again. All of that is stored: user text, agent text, tool calls, tool results, final_text.
6. Grade. Rows come back ungraded. Grade after with zps.grade(...), pass your own grader=, or grade=True for the deterministic conduct score.

Stop when the row cap or the clock hits.

How to use

pip install zeroproof   # or: uv add zeroproof

Bring your own model. Any OpenAI-compatible chat endpoint that returns tool calls works; it writes the situations and plays the agent, so both run on your key:

export OPENAI_API_KEY=...
export OPENAI_BASE_URL=...   # only for a non-OpenAI endpoint

import zeroproof.simulations as zps

data = zps.simulate(
    agent="openai:gpt-4.1-mini",
    tools=my_tools,
    system_prompt=my_system_prompt,
    output="rollout.jsonl",
)

Or use ZeroProof-hosted Qwen, which is the default when no agent= is given. Ask us for a VLLM_API_KEY; the endpoint is shared and rate limited.

export VLLM_API_KEY=...

No key at all: the situation writer also defaults to hosted Qwen, even when agent= is your own function. Pass simulator=False to use the built-in template writer instead. It needs no model and runs in seconds; the situations are less varied than a model writes, so it is for wiring up your agent and grader, not for a training set.

data = zps.simulate(
    my_agent, tools=my_tools, system_prompt=my_system_prompt, simulator=False, budget=40
)

Working in this repo: uv sync, then uv run pytest after uv sync --extra dev.

One runtime dependency (requests), Python 3.10+. Installing from PyPI rather than a path or a git URL matters if you build a Prime Intellect environment on this: the Environments Hub installs a pushed env with plain pip, so a [tool.uv.sources] git pin resolves locally and then fails on their runtime with a ModuleNotFoundError.

import zeroproof.simulations as zps

data = zps.simulate(tools=my_tools, system_prompt=my_system_prompt, output="rollout.jsonl")
data = zps.simulate(agent=my_agent)

Pass spec= if you have a local tools-and-system-prompt folder. The generated datasets are on Hugging Face, organized by agent type instead of stored in this repo.

What to run

Depends on the use case. How each scenario is built is in The recipe.

zps.simulate(tools=my_tools, system_prompt=my_system_prompt)  # explore
zps.simulate(tools=my_tools, system_prompt=my_system_prompt, mode="sft")
zps.simulate(tools=my_tools, system_prompt=my_system_prompt, mode="rl")
zps.simulate(tools=my_tools, system_prompt=my_system_prompt, mode="adaptive", until="saturation")

Examples

Sign in

zeroproof login

Prints a link and a short code. Open the link, sign in or sign up, press Approve. The key is saved to ~/.zeroproof/credentials.json and every platform call below reads it from there. Interrupted before you approved? Run it again; it resumes the same code. This is the path for coding agents too: tell yours to run zeroproof login and click the link it shows you. zeroproof status shows which key is in use, zeroproof logout removes it.

No account yet, or no browser? One command creates the account and the key. Open the dashboard later by signing in with an email code.

zeroproof signup --email you@example.com

That key is a trial key (25k input and 50k output tokens a day, 100 MB, ten datasets, seven days) until the person signs in once at https://www.zeroproofai.com/sign-in with an email code. zeroproof status shows the tier; zeroproof.account() returns tier, limits and usage.

Store datasets on Zero Proof Labs

Push a run to your Zero Proof Labs account so the optimization framework can iterate on it. Credentials resolve in this order: api_key= argument, ZEROPROOF_DELEGATED_CREDENTIAL (a short-lived zp_dc_... issued from a Clerk session), ZEROPROOF_API_KEY, then the key saved by zeroproof login.

# Runtime path with a delegated credential
# export ZEROPROOF_DELEGATED_CREDENTIAL="zp_dc_..."

# If you need to mint one from a Clerk session token:
# credential = zps.issue_delegated_credential(clerk_token, ttl_seconds=3600)
# export ZEROPROOF_DELEGATED_CREDENTIAL=credential["credential"]

data = zps.simulate(spec="specs/github")
v1 = data.push("github-explore-v1")  # -> {"datasetId": "ds_...", ...}

# iterate, then push the next version with lineage
v2 = data.push("github-explore-v2", parent=v1["datasetId"])

zps.datasets()  # list yours + storage used
rows = zps.pull(v1["datasetId"])  # rows, or pass path= for a file
zps.push_file("rollout.jsonl")  # upload an existing JSONL
zps.delete_dataset(v1["datasetId"])  # permanent

Storage is private per account, 5 GB free. parent= records dataset lineage so iterations show as a family on the platform.

data.push and zps.push_file run a publish gate first (gate=False skips it). Every graded row gets a calibration stamp: its task's pass rate over k repeats, k, and the policy that produced it, so a trainer can build a curriculum or retire solved tasks. An RL-shaped run (repeats of one ask) is refused with PublishGateError when it is ungraded or has no mixed group, because a grouped update would learn nothing from it. The report comes back as entry["gate"], with warnings when out-of-band or unanimous asks are still present; zps.optimize(data, mode="rl") prunes those. zps.publish_gate(rows) runs the same check on any row list. The stamp is the schema's Calibration object: zps.calibration_of(row) reads it back typed, from_row carries it on rollout.extra["calibration"], and to_row writes it out again.

Training rows from export_training / training_rows carry a loss_mask, one 0/1 per message: 1 on the agent's turns, 0 on system, user, and tool-output turns. Tool output is the environment's text, not the policy's, so a trainer should not learn to predict it. mask_mode="final" trains only the last assistant turn, for conversations whose earlier agent turns were scripted or came from another policy; the export report counts trained_messages and masked_messages.

Prune before training

rows, report = zps.optimize(data, mode="rl")  # whole groups, 20%-80% pass rate
rows, report = zps.optimize(data, mode="rl", band=(0.3, 0.7))
rows, report = zps.optimize(data, mode="rl", enforce_band=False)  # rank, do not drop
report["band_dropped"]  # {"too_easy": n, "too_hard": n}

optimize(mode="rl") drops junk rows, duplicate rollouts within an ask (same trajectory twice adds nothing to a group-relative advantage), truncated rollouts, unanimous asks (all pass or all fail: zero advantage), and asks outside the difficulty band, then keeps whole groups round-robin across fault kinds. The band is the offline difficulty filter from the reasoning-model recipes (keep prompts the policy solves 20-80% of the time); it is a heuristic, so it is a parameter.

The report also carries the reward-hack scan: report["correlations"] is corr(reward, feature) for reply length, tool-call count, and assistant turns, and report["hygiene_warnings"] names anything at or above HACK_THRESHOLD (0.3). Reward that tracks length or punishes tool use is a judge problem, so it is flagged, not pruned. The same scan, plus near-duplicate asks and length spread, runs in the publish gate. Standalone: zps.reward_correlations(rows), zps.dedupe_groups(rows), zps.near_duplicate_prompts(rows), zps.length_report(rows).

Train, holdout, eval

data.push("airline-v3", holdout=0.2)  # train set + a linked holdout set, split by task
data.push("airline-evals", purpose="eval")  # a set you measure with
zps.update_dataset("ds_...", purpose="holdout")
zps.preview("ds_...")  # three sample rows + the analyzer report
zps.profile("ds_...")  # pass rate, support, mixed tasks, tool use, per task

The Datasets page groups sets by purpose (train, holdout, eval) and records the simulation mode on each. A push is train unless it says otherwise; ingested traces are eval until training data is cut from them. Holdout is split by scenario_id, so a task is wholly on one side, and the same task lands on the same side every run.

Trust the numbers

Three checks that decide whether a result is believable, all report-only and all over rows you already have.

zps.judge_trust(rows, judge=my_judge)  # is the judge trustworthy?
zps.decontaminate(train_rows, against=[eval_rows])  # 8-gram overlap with the eval set
zps.compare_runs(run_a, run_b)  # paired delta with a 95% interval
zps.delta_report(before, after, target="pass_at_1", must_not_regress=["honest_after_fault"])

Judge trust. Label 30 to 100 rows by hand as gold_reward (0/1). The report gives agreement with a Wilson interval and Cohen's kappa, agreement on two task halves (tune the rubric on one, read the other), judge pass rate on short versus long replies within the same human label (length bias the humans rule out), and, with the judge callable, a re-judge of a sample as-is (consistency) and with neutral filler appended (a flip means the judge reads length). Disagreements come back as a review queue. format_judge_trust(report) prints it.

Decontamination. Word 8-gram overlap between a dataset's prompts and replies and any evaluation source: row lists, JSONL paths, or platform dataset ids. Short prompts fall back to exact match. Returns the clean rows and the first offenders.

Intervals and comparison. Every pass@1 now carries a 95% interval from a bootstrap over tasks (pass_at(rows).ci95), and metric_summary / marker_summary do the same for markers. compare_runs pairs the tasks two runs share, bootstraps the paired difference, and adds a sign-flip permutation p-value; fewer than five shared tasks falls back to an unpaired test and says so. The verdict no_difference_detected means the interval covers zero, not that the runs are equal.

Before and after. delta_report runs compare_runs on pass@1 and every marker both row sets share. target= names the metric the training was meant to move and gives the headline; must_not_regress= names the behaviors whose significant drop fails the report; any other significant drop is a warning. format_delta_report(report) prints one line per metric.

Publish a dataset as a card

data.push(
    "airline-refunds-v3",
    agent="airline-support",
    publish=True,
    description="Graded refund conversations with injected tool faults.",
)
zps.publish("ds_...", agent="airline-support")  # or publish an existing one
zps.catalog()  # every public card, by agent
rows = zps.pull("ds_...")  # public sets need no key
zps.unpublish("ds_...")

Cards live at https://zeroproofai.com/datasets, grouped by agent, with rows, size and the analyzer's numbers on each. A dataset must be finalized and hold rows to publish.

Speed

Two-minute airline runs using ZeroProof-hosted Qwen. Results were measured on the hosted GPU with warm replicas and burst under load.

Parameter reference

Aliases: phrasings= / n= → requests_per_situation; repeats= → rollouts_per_request; unique= → unique_situations; policy= → system_prompt; risk= → fault_rate.

Output

Each row, in data.trajectories and on disk: prompt, messages, steps, final_text, scenario_id. Optional world_state, faults, reward, reason. llm_grade=True adds llm_reward. zps.rank(path) adds quality without changing reward.

After grading, data.pass_at (also on the ScoredData from judge= and evaluate) gives pass@1, pass^k and pass@k off the same groups, one job each: pass@1 is the measurement headline (the agent runs once in production), pass^k is the reliability line (all k repeats pass), and pass@k minus pass@1 (.headroom) is what a grouped RL update has to learn from, the same asks group_signal counts as mixed. k is the smallest group of repeats; below repeats=4 the k-way numbers are None with a note rather than a noisy figure. .per_task is the raw per-prompt pass-rate vector. With an LLM judge, pass@k inflates on false positives and pass^k on false negatives, so pass@1 stays the headline.

scored = data.grade(judge=my_judge)
print(scored.pass_at)  # pass@1 0.61 | pass^8 0.32 | pass@8 0.88 | headroom 0.27 (200 groups, k=8)

simulate(logprobs=True) records, on every agent turn, the summed log-probability of the tokens the policy generated and how many there were (step["logprob"], step["n_tokens"], totals on the row). A trainer that updates on these rollouts later needs that number to form the importance ratio exp(new_logprob - logprob); without it the update is off-policy and nothing says so. zps.logprob_report(rows) says how much was captured and whether reward tracks the policy's confidence, which on a fair judge it should not. Score the same rows under a reference model, put its summed logprob in ref_logprob, and zps.mean_kl(rows) gives the sampled KL per generated token, overall and per task; zps.calibrate(rows, ref="ref_logprob") writes it into each row's calibration.mean_kl. A turn the model cut at the token cap is marked truncated.

The default judge is not the policy. zps.grade grades with hosted Phi-4 (ZEROPROOF_JUDGE overrides; any vllm:/openai: spec or a bare URL works), while rollouts come from hosted Qwen, because a judge grading its own model's writing prefers it. When the judge and the rows' model_version are the same model anyway, the grade report says so (self_judged, warnings).

A judge is a reward model, so two things ride with every label. Provenance: rows graded by zps.grade carry judge_name, judge_status, and judge_meta with the model, prompt hash, temperature, and version (<model>@<prompt sha>); a rubric edit is a new judge and the row says so. run_judge(version=...) records the same for your own judge. Accuracy: hand-label a sample into gold_reward and call zps.judge_agreement(rows) (or scored.agreement()) for agreement, Cohen's kappa, the confusion counts, and pass_when_gold_fail, the gold failures the judge passed. Those are the rows a training run learns the failure from, so that rate matters more than the headline agreement. Pass a second scoring run as gold to measure the judge against itself. Fifty gold rows is the floor; the report says so below it.

Every row carries schema_version ("1"). A row is a projection of four objects in zeroproof.simulations.schema: Task (the situation), Rollout (one episode), Judgment (a scorer's verdict), Marker (a behavior measurement). zps.from_row(row) splits a row into them and zps.to_row(...) flattens them back. The wire contract is zeroproof/simulations/schemas/row-v1.json. Rows written before the stamp are version 0 and load by shape, so older files still work.

The recipe

Each scenario is a draw across the world and the human.

World (from this agent's tools and system prompt)

• objects and tool results that match the spec
• tool outcome: success, timeout, deny, stale, etc.
• world state: exists, missing, already handled, unfinished, etc.
• history: first visit, prior miss, return, etc.
• rules the agent is supposed to follow

Human

• intent: which tool, what they want (randomized sometimes)
• stance: ordinary, ambiguous, adversarial, hurried, etc.
• persona: first time, returning, in a hurry, etc.
• tone: impatient, frustrated, polite, etc.
• typing: standard, lowercase, typo, clipped, etc.

Ordinary asks first, then the edges. On top of that, we embed the openers and add a bit of random noise so the batch stays spread out, not a cluster of near-copies. Spend the row cap and the clock on diversity, not copies.

Package layout

The public surface is the package itself: import zeroproof.simulations as zps. Internals are grouped by stage and may move between releases.

Development

uv sync --extra dev
uv run pytest           # about two minutes, no network
uv run ruff check .     # lint; `--fix` for the mechanical ones
uv run mypy             # type check
pre-commit install      # optional: ruff and whitespace hooks on commit

CI runs the suite on Python 3.10 through 3.13, ruff, mypy, and a plain-pip install of the built wheel into a clean venv.

License

Apache-2.0