zeroproof-simulations

Generate diverse training conversations for any agent, grounded in its tools and system prompt.

The SDK inspects the agent, simulates a world consistent with those tools (objects, results, failures), and samples scenarios across that space. The same model writes the user and plays the agent. Default explore: one unique situation per row.

Overview

simulate() is a pipeline.

1. Read the agent. Tools and system prompt. That is the spec of the world.
2. Build a fake world from those tools. Objects, plausible results, and faults (timeout, deny, junk).
3. Write users. A separate writer (same hosted model, different prompt, no agent policy) samples situations across tools, stance, history, and so on.
4. Pick the diverse ones. Embeddings plus a bit of noise so the batch is not 200 copies of same prompy.
5. Play the agent. It talks, calls tools, gets results, talks again. All of that is stored: user text, agent text, tool calls, tool results, final_text.
6. Grade. Deterministic conduct score by default. Attach an LLM if you want. Or pass your own grade=.

Stop when the row cap or the clock hits.

How to use

uv sync
export VLLM_API_KEY=...

ZeroProof hosts Qwen for simulation: it powers the default user writer and rollout agent. Ask us for a VLLM_API_KEY to try it. The endpoint is shared and rate limited.

From another project:

pip install zeroproof-simulations   # or: uv add zeroproof-simulations

Zero runtime dependencies, Python 3.10+. Installing from PyPI rather than a path or a git URL matters if you build a Prime Intellect environment on this: the Environments Hub installs a pushed env with plain pip, so a [tool.uv.sources] git pin resolves locally and then fails on their runtime with a ModuleNotFoundError.

uv run pytest after uv sync --extra dev.

import zeroproof_simulations as zps

data = zps.simulate(tools=my_tools, system_prompt=my_system_prompt, output="rollout.jsonl")
data = zps.simulate(agent=my_agent)

Pass spec= if you have a local tools-and-system-prompt folder. The generated datasets are on Hugging Face, organized by agent type instead of stored in this repo.

What to run

Depends on the use case. How each scenario is built is in The recipe.

zps.simulate(tools=my_tools, system_prompt=my_system_prompt)                 # explore
zps.simulate(tools=my_tools, system_prompt=my_system_prompt, mode="sft")
zps.simulate(tools=my_tools, system_prompt=my_system_prompt, mode="rl")
zps.simulate(tools=my_tools, system_prompt=my_system_prompt, mode="adaptive", until="saturation")

Examples

Store datasets on Zero Proof Labs

Push a run to your Zero Proof Labs account so the optimization framework can iterate on it. Auth is your zp_ API key from https://www.zeroproofai.com/platform (env ZEROPROOF_API_KEY).

data = zps.simulate(spec="specs/github")
v1 = data.push("github-explore-v1")            # -> {"datasetId": "ds_...", ...}

# iterate, then push the next version with lineage
v2 = data.push("github-explore-v2", parent=v1["datasetId"])

zps.datasets()                                  # list yours + storage used
rows = zps.pull(v1["datasetId"])               # rows, or pass path= for a file
zps.push_file("rollout.jsonl")                 # upload an existing JSONL
zps.delete_dataset(v1["datasetId"])            # permanent

Storage is private per account, 5 GB free. parent= records dataset lineage so iterations show as a family on the platform.

Speed

Two-minute airline runs using ZeroProof-hosted Qwen. Results were measured on the hosted GPU with warm replicas and burst under load.

Parameter reference

Aliases: phrasings= / n= → requests_per_situation; repeats= → rollouts_per_request; unique= → unique_situations; policy= → system_prompt; risk= → fault_rate.

Output

Each row: prompt, messages, steps, final_text, scenario_id. Optional world_state, faults, reward, reason. llm_grade=True adds llm_reward. zps.rank(path) adds quality without changing reward.

The recipe

Each scenario is a draw across the world and the human.

World (from this agent's tools and system prompt)

• objects and tool results that match the spec
• tool outcome: success, timeout, deny, stale, etc.
• world state: exists, missing, already handled, unfinished, etc.
• history: first visit, prior miss, return, etc.
• rules the agent is supposed to follow

Human

• intent: which tool, what they want (randomized sometimes)
• stance: ordinary, ambiguous, adversarial, hurried, etc.
• persona: first time, returning, in a hurry, etc.
• tone: impatient, frustrated, polite, etc.
• typing: standard, lowercase, typo, clipped, etc.

Ordinary asks first, then the edges. On top of that, we embed the openers and add a bit of random noise so the batch stays spread out, not a cluster of near-copies. Spend the row cap and the clock on diversity, not copies.

License

Apache-2.0