TeamRunner

The experimentation platform for agent organizations.

Preview — a soft launch. TeamRunner 0.0.x is published so friends and testers can try it before it is announced. Until the announced 0.1.0 it may change in ways that break teams and run folders made with an earlier version. Comments are very welcome: open an issue or write to hello@teamrunner.ai.

Run a multi-agent system entirely from text files and folders. Change one line and run it again; [v0.2] run it eighty times and get a CSV of what changed.

debate/                      ← a package: this folder gets a debate done, anywhere you copy it
├── team.yaml                ← this folder is a team, and its default topic
├── advocate.md              ← an agent: config on top, prompt as prose
├── critic.md
├── judge.md                 ← no chair.md, so they speak in name order: advocate, critic, judge
└── runs/                    ← every run of this team, logged here (git-ignored)

experiments/advocate_model/  [v0.2]
└── experiment.yaml          ← base: ../../debate — GPT vs. Claude as advocate, 20 replications

Everything unmarked is in the first release, 0.1. [v0.2] arrives with experiments; [v0.3] with richer organizations (managers, private information, subteams, parallel work).

teamrunner run debate --message "Remote work makes teams more productive"

No Python. No graph-building API. No framework vocabulary. If you can edit a text file and run one command, you can run — and study — an organization of agents.

Contents

• Why we built it
• The five ideas
• Tutorial: your first debate in 10 minutes
• Tutorial part 2: your first experiment
• What lands in a run folder
  • What your runs keep, and where your words go
• Going further: the worked examples
• What TeamRunner is not
• Status and roadmap
• Questions, bugs and ideas

Why we built it

1. The people who study organizations are not the people who write agent code

Management scientists, operations researchers, and behavioural economists have spent a century studying how the structure of a team changes its output: hierarchy vs. flat, who reports to whom, who sees which numbers, what happens when one member has a private incentive.

Suddenly there is a laboratory where you can actually run those experiments — teams of LLM agents, cheap, repeatable, fully observable. And the entry price is: learn Python, learn an agent framework, and write a few hundred lines of orchestration code before you can ask your first question.

That is the wrong tax on the wrong people. In TeamRunner an agent is a Markdown file, a team is a folder, and a hierarchy is a folder inside a folder. The org chart is the directory tree. You edit prose, not code.

2. Changing the structure should be a one-line change, not a refactor

In code-first frameworks, the organizational structure is dissolved into the program. "What if the fact-checker were a neutral referee instead of each side's own editor?" is a question about org design; answering it means rewriting your graph.

In TeamRunner the structure lives in files, and experiments are overlays that patch those files. The founding rule:

If it's in a file, it's a factor.

Models, temperatures, prompts, who sees what, the flow itself, even which sub-team does the work — all of them are legal things to vary in a designed experiment, because all of them are just values in text files.

3. Almost nothing published about multi-agent systems is reproducible

Most multi-agent results are n = 1 demos: one run, one prompt, one screenshot. LLMs are stochastic; a single run is an anecdote. And even a careful author can rarely hand you the exact artifact that produced their result.

TeamRunner treats reproducibility as a feature, not a chore, with a deliberately honest standard:

Deterministic where the system controls randomness, statistically rigorous where the model does, fully auditable everywhere.

• Deterministic harness. [v0.2] Every source of randomness we own — condition assignment, turn order, sampling of tasks — is seeded from one master seed. Same seed, same experimental design.
• Statistical, not bitwise, replication. [v0.2] We do not pretend API models can be replayed from a seed (they can't: some providers offer best-effort seeds, some none, and temperature=0 is not determinism). So the unit of a result is a distribution over replications, with confidence intervals and effect sizes. replications: 1 is deliberately awkward to write.
• Total artifact capture. Already in 0.1: every member exactly as it ran (model, settings, full prompt), everything said, the model that actually answered, token counts, timings, costs, and the Python, LiteLLM and Deno versions go into the run folder, so you can compare two runs line by line. [v0.2] Every raw request and response, and a fully expanded copy of the spec.

A team folder is a shareable, forkable, git-diffable package. "Clone this repo and reproduce this study" is the whole point, and it is what 0.2 completes.

4. Orchestration is crowded. Experimentation is empty.

There are many good ways to build a multi-agent application. There is almost no way to run a designed experiment on one. TeamRunner is aimed squarely at the second: factorial designs, seeded replications, an evaluation layer, a canonical event log, and a long-format CSV at the end that goes straight into R, Stata, or pandas. The event log is in 0.1, under every run; the rest arrives with [v0.2].

Who it's for, today: researchers studying agent behaviour and organizational structure. Who it's for next: eval and prompt-optimization teams in industry, who have exactly the same problem with a different vocabulary — they call it a prompt A/B test, we call it a two-level factor.

The five ideas

1. An agent is a Markdown file. YAML frontmatter for configuration, the body is the system prompt as prose. Non-programmers edit prompts in the format prompts actually want to be written in.

---
model: openai/gpt-6-luna
params: {reasoning_effort: high}
output:
  winner: [advocate, critic]
  reasoning: ""
---
You are an impartial judge of a formal debate on the motion:

> {{motion}}

Decide which side argued better — argument quality and logical clarity,
not your own opinion of the motion.

2. A folder is a team if it contains team.yaml. Everything inside is a member, and a member is one of three things: a worker (an agent with a goal), a team ([v0.3] a subfolder with its own team.yaml — just another node in its parent's flow: its turn means running its own internal flow to completion, like a subcommittee: it deliberates on its own whiteboard, and the parent hears only its report — report_from: in its team.yaml says whose words those are), or a toolkeeper (the member who holds the keys to a tool). Complexity is contained by hierarchy, not by syntax. There is no registry file listing your agents; the file browser is the org chart.

Tools are not members. An organization doesn't hand every worker every tool; a factory has a tool crib, with someone at the window who checks who may take what, and everyone else asks there. So the only members who touch tools are toolkeepers — a toolkeeper holds the guardrails (timeouts, no network, which folders), decides by a plain access list rather than a prompt, and its text body is simply what the team is told about it. Who may use a toolkeeper, and on what terms, is up to the toolkeeper's own access list — one toolkeeper can give different members different terms, the way a library gives professors and students different cards. The whole folder — team and toolkeepers — is a package: copy it anywhere and it gets the same job done. The work itself comes from outside: you hand the team a topic when you run it, or it uses the default written in team.yaml.

3. Every team has a chair, and there are no hidden if-statements. Someone always decides who acts next, and TeamRunner makes you say who:

• a chair (chair.md), which follows the script you wrote, or ([v0.3]) a decision table — deterministic, zero model calls;
• a manager (manager.md, [v0.3]) — an LLM whose prompt says "you are the moderator, decide who speaks next";
• or neither, and members simply take turns in alphabetical order, everyone hearing everyone.

Every routing decision is logged as an event attributed to whoever made it. Which means protocol vs. discretion — a script vs. a manager exercising judgment — is itself something you can A/B test by swapping one file for the other ([v0.3]).

4. One whiteboard for who sees what. Everything an agent knows arrives through one of:

Everything said is posted on the whiteboard, open to all by default, so a team with no whiteboard file is a meeting room — and that is all of v0.1. [v0.3] whiteboard.yaml adds the rest: seal the reviewers from each other, or make the team a relay where each person hears only who spoke before them, with a read_by line for what was said — one line each.

Information asymmetry is not a feature we had to invent; it is a narrow read_by. Incentives are numeric items on the whiteboard. Private briefs are items only one party can read.

And one thing that is emphatically not a layer: the run log. It is the researcher's camera, outside the simulation. No agent can ever read or write it — a hard guarantee, without which every asymmetry design leaks.

5. Experiments are overlays. [v0.2] Your package stays clean and runnable. An experiment.yaml lives outside it, names it with base:, and declares factors as patches against it, plus replications and a seed. Single runs and 200-run factorials go through the same engine. When a variant wins, teamrunner promote copies the exact version that ran back into the package — same agent, new version — and export hands you a standalone copy for production.

Tutorial: your first debate in 10 minutes

We'll build a three-agent debate, run it, give it rounds, a fact checker and the web, and then ([v0.2]) turn it into a 40-run experiment. Zero code throughout.

Everything in this tutorial not marked [v0.2] or [v0.3] works in the preview. See Status.

Step 0 — install

pip install teamrunner

Put your provider keys in TeamRunner's key file, ~/.teamrunner/.env (in your home folder, outside every team):

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

Keys stay with you, never with your teams: team folders get copied, zipped and sent to people, and a key inside one goes with it. If you already have a key set in your shell, TeamRunner offers to save it to the key file the first time it needs it (or to use it just this once). If it finds a .env inside a team folder, it offers to move the key out and delete it there. And it will refuse to run if it finds something that looks like an API key inside a spec file.

Models on your own machine work too, free: install Ollama, pull a model (ollama pull llama3.2), and write model: ollama/llama3.2. If Ollama runs somewhere other than this computer, put its address in the key file as OLLAMA_API_BASE. A word of warning: the smallest models often fail at exactly what teams need most — answering in a fixed form and asking toolkeepers properly — so they test the wiring better than they stand in for a real team. Before the first turn of every run, TeamRunner checks that each model your team names is available and lists every problem it finds in one table, so you never pay for half a run. teamrunner validate --online runs the same check without a run.

Step 1 — make a team

teamrunner new debate
cd debate

new only makes files — no model is called. It creates the debate/ folder, a starter team.yaml, and a .gitignore that keeps run logs out of git.

Two other ways to start, when you'd rather not begin from nothing:

• teamrunner new my_debate --from fact_checked copies one of the examples that ship with TeamRunner (or any team folder you name), with fresh ids; each file remembers what it was copied from.
• teamrunner new my_debate --describe "two debaters argue a topic, a fact checker checks each speech, and a judge decides" asks a model to write the team for you. It checks its own work with validate, never overwrites files, tells you what it cost, and ends by printing teamrunner explain, so you can read what it made before you run it. (If you work with an AI coding assistant, teamrunner skill gives it the same instructions: it installs a skill for Claude Code and Codex, and teamrunner skill --agents-md <folder> writes an AGENTS.md for other assistants that read one.)

This tutorial starts from nothing, so you see every file.

team.yaml is the only required file. It marks the folder as a team and says what the team does:

# team.yaml
name: debate
description: Two debaters argue a topic; a judge decides

Two things worth knowing now:

• There is no version line to write. TeamRunner keeps one rule instead: a word in these files never changes its meaning. When something has to mean something new, it gets a new word, and the old one is retired. So if your TeamRunner meets a word it doesn't know — a newer feature, or a retired one — it stops before spending anything and tells you what to do (pip install -U teamrunner, or which word replaced the old one). Every run records the exact TeamRunner version it used, so your 2026 study can be rerun in 2028.
• You never number your own versions. The first time you validate or run, TeamRunner adds an id: line to each file, with a comment saying what it is: a permanent tag that stays the same when you edit the file, rename it, or copy it to another project. Wherever TeamRunner shows an id, it shows the last eight characters after a "…" (judge (…KBMCNDPE)) — search for those to find the full id. That is what later lets you ask "show me every run of this judge, across all five of my projects." Each time you edit a prompt, TeamRunner notices the content changed and records it as a new version of the same agent — every run's log keeps a fingerprint of each member's content — automatically, with nothing for you to write.

Step 2 — write three agents

An agent file is Markdown. Frontmatter configures, body prompts.

<!-- advocate.md -->
---
description: Argues in favor of the topic
model: openai/gpt-6-luna
---
You are debating in favor of this topic:

> {{topic}}

Each round, make your strongest remaining argument in one short
paragraph. From round two onward, also rebut your opponent's most
recent point before adding your own. Be persuasive but honest.

<!-- critic.md -->
---
description: Argues against the topic
model: openai/gpt-6-luna
---
You are debating against this topic:

> {{topic}}

Each round, rebut your opponent's most recent point, then make your
strongest remaining argument against the topic in one short paragraph.
Be persuasive but honest.

<!-- judge.md -->
---
description: Decides who argued better
model: openai/gpt-6-luna
output:
  winner: [advocate, critic]
  reasoning: ""
---
You have watched a debate on:

> {{topic}}

Decide which side argued better — judge argument quality and logical
clarity, not your own opinion of the topic. Announce the winner and
explain your decision in one paragraph.

Four things just happened:

Every agent names its model. The model is part of the agent — it's who you hired. When a whole team uses the same one, team.yaml can say it once (default_model: openai/gpt-6-luna), and a model with its settings can live in a profile, models/careful.yaml, that agents name as model: careful.yaml. teamrunner show judge prints exactly which model and settings the judge will run with. examples/orchestrators_lite does both: its five engineers share one profile, models/engineer.yaml, named once as the team's default_model:.

Alphabetical order is the flow. With no chair.md, members act in alphabetical filename order, one turn each — here advocate → critic → judge. That is also why the files above have no name: line: an agent's name is its filename. The folder listing is the running order, visible in any file browser. Rename to reorder.

{{topic}} is an input, filled in the next step. Prompts stay reusable; the specific question is a controlled variable, not something an agent invents on the fly.

output: makes the judge's verdict machine-readable. The judge now returns typed fields (winner, a choice; reasoning, text) alongside its prose. This matters more than it looks: declared outputs are the only thing routing and metrics are ever allowed to read. TeamRunner never regex-scrapes free text to decide what happens next. If control flow depends on it, it has to be declared.

The one-line forms above (winner: [advocate, critic] means "a choice between these"; reasoning: "" means "text") expand to a long form when you need it: winner: {type: choice, options: [advocate, critic]}. The four types are choice, text, number, flag.

Step 3 — give them something to argue about

Add these lines to the end of team.yaml:

message_goes_to: topic

default_inputs:
  topic: "Remote work makes teams more productive"

default_inputs is what the team argues about when you don't say otherwise. It can hold as many inputs as the prompts use — one line each, say topic: and audience: — and a task file (--task-file) can give several at once. It comes last because a team only needs it if its prompts take inputs. message_goes_to says that a message you send from the command line becomes the topic, so the prompts can keep a name that reads well. The topic itself comes from outside the team: changing it — or making it a factor — is how you check that a finding isn't an artifact of one question.

Step 4 — validate, then run

teamrunner validate .

validate is the compiler for people who don't write code, and we treat its error messages as a headline feature: a typo in a flow arrow, a decision option with no arrow, an agent with no model, a worker holding a tool instead of asking a toolkeeper, an input with neither a value nor a default — all of it is caught before a single token is spent.

teamrunner run . --rehearse

A rehearsal runs the team with no model at all: every member says placeholder words, and each typed answer is picked at random from what its file allows, so decisions and loops really happen. It costs nothing and needs no key. It tells you whether the flow goes where you meant — not whether the answers are any good.

teamrunner run .

teamrunner run . --message "Four-day weeks make teams more productive"

The first runs the default topic; the second replaces it. Either way, the run log records the topic and whether it was given or the default.

What you see while it runs (the words are invented; the layout is the real one):

run · debate · default input topic: "Remote work makes teams more productive"
✓ 1 model · 1 key
advocate                                                     3.1s   $0.0060
    “Remote work lets people do focused work without…”
critic                                                       3.3s   $0.0060
    “Collaboration suffers when nobody shares a room…”
judge                       winner = advocate                2.1s   $0.0040
    “The advocate gave the stronger evidence and…”

completed · 3 turns · 8.5s · ≈ $0.0160
runs/2026-10-12_143201_debate

**judge**

The advocate gave the stronger evidence and answered every point…

- **winner:** advocate
- **reasoning:** stronger evidence

Each turn shows who spoke, what they decided, how long it took, what it cost and the start of what they said. Costs are estimates — providers report tokens, not prices, so TeamRunner multiplies exact token counts by list prices — hence the ≈. If you pay different prices, write yours in ~/.teamrunner/prices.yaml; every run records each price it used and where it came from. --verbose shows everything said in full, with tokens and the exact model used; --quiet prints nothing at all, for scripts and experiments, and reports how the run went through its exit code and the run folder.

Later, teamrunner runs lists every run of this package — when, which team, the input, the result (winner = advocate), status, time and cost — and teamrunner runs latest (or teamrunner runs 14:32) shows one again exactly as it looked while it ran.

That's a working multi-agent system: four text files, one command.

Step 5 — take control of the flow

Alphabetical order got us one turn each. A real debate has rounds. Give the team a chair — add chair.md, which is simply the script:

# Three rounds of advocate and critic, then the judge decides.

rounds(*3) -> judge

rounds = advocate -> critic

The flow grammar is deliberately tiny — it is a whiteboard sketch that happens to be executable:

• a -> b — sequence. b:label attaches a stage annotation, which is recorded in the events and injected into the agent's context so it knows where it is (advocate:opening, critic:closing).
• rounds(*3) — repeat a step exactly three times.
• name = … — a named block: any step name can be defined below, as rounds is here. Its further lines are indented under it.
• # starts a note for the reader. If the chair ever needs settings, they go in a frontmatter block at the top, like any member file.
• [v0.3] [a, b] — at the same time: editor -> [reviewer_1, reviewer_2] -> editor sends the paper to both reviewers at once, and the editor waits for both reports. Neither sees the other's work in progress.
• done ends the flow — or, inside a named block, just that block; stop ends the whole run immediately.

Now the debate runs advocate → critic → advocate → critic → advocate → critic → judge. To see it as a picture, teamrunner explain --diagram opens the flow as a flowchart in your browser.

Step 6 — add a fact checker and a decision

Suppose we want each speech fact-checked before it stands. Write a fact checker with a declared output::

<!-- fact_checker.md -->
---
description: Checks the factual claims in the latest speech
model: openai/gpt-6-luna
output:
  verdict: [pass, fail]
  reasons: ""
---
You are a neutral fact checker in a formal debate. Examine the latest
speech for checkable factual claims and verify them by searching the
web (ask the web toolkeeper). Search only for claims you cannot confirm
from what you know; most speeches need one or two searches. Minor
exaggeration is fine; invented statistics, misattributed quotes and
false claims are not.

Set verdict to fail only if a claim is materially false, and list each
false claim with a one-line correction in reasons. Otherwise set it to
pass.

…and wire its decisions into the chair's script:

# chair.md
# Three rounds. Each speech is fact-checked; a failed speech gets two
# more tries, then the debate moves on. Then the judge decides.

rounds(*3) -> judge

rounds = advocate_turn -> critic_turn

advocate_turn = advocate -> fact_checker
                fact_checker --fail--> advocate
                fact_checker --pass,*3--> done

critic_turn = critic -> fact_checker
              fact_checker --fail--> critic
              fact_checker --pass,*3--> done

--pass,*3--> done reads "on pass, or the third time, go to done": the comma means "or", and *3 means "the third time". Here done ends only advocate_turn (or critic_turn), not the debate: that speaker's turn is over and the debate moves on to the next speaker. A speech that fails three times still stands — but everything said is heard by everyone, so the judge has heard the fact checker fail it, and the judge's prompt says to weigh that. Want a different ending? fact_checker --*3--> stop ends the whole run instead (examples/deb8flow does this); a forfeit that hands the debate to the other side needs a whiteboard item, which comes in 0.3.

An agent whose declared options have labelled outgoing arrows is a decider, and its labelled arrows are a decision. That's a role, not a different file format — any agent can be one. done inside a named block ends that block, so a passed speech hands over to the next speaker.

Two rules make this safe:

Branch labels are checked against declared options. --puss--> is a validation error, not a 2 a.m. surprise. Every option needs exactly one arrow.

Loops are legal only through a decider, and only with a limit. An arrow back to an earlier step needs a counted way out: some arrow from the same decider must carry *N. On the Nth time the decider's choice still counts if it leaves the loop (a pass on the third try is a pass); only a choice to go round again is overridden. When running out should go somewhere else, give the count its own arrow: fact_checker --*3--> stop. There is no goto, no labels-and-jumps: a flow reads top to bottom and always ends.

And note the split that makes components reusable: the fact checker's .md owns how to decide; the flow owns what each decision means here. The same fact checker checks advocate and critic, because neither is written into its file.

Step 7 — give the team the web, through a toolkeeper

The fact checker's prompt says "ask the web toolkeeper", so the team needs one. Its searches go through Brave Search, so add a Brave Search API key to the key file as BRAVE_API_KEY. Searches cost half a cent each at Brave's list price, and a run's cost counts them: the totals show models and tools separately. Agents never hold tools themselves. A tool is reached through a toolkeeper: a member with no model and no prompt, which holds the tool, checks who is asking, and enforces the limits — like the attendant at a factory's tool crib, who checks who may take which tool and how many, and logs it.

<!-- web.md -->
---
description: The web toolkeeper — searches the web, on different terms for different members
kind: toolkeeper
tools:
  web_search_brave: {max_results: 5, calls_per_turn: 2}   # the base terms: the strictest
access:
  fact_checker: {calls_per_turn: 3}                 # a few claims a speech may need checking
  judge: standard                                   # the base terms
---
I search the web. Send me a search query and I return the top five
results with their sources.

Read the file top to bottom:

• tools: names the tool and its base terms — five results, two searches a turn.
• access: lists who may use it, one line per member. The fact checker gets a better card (three searches a turn); the judge gets the standard terms. Anyone not listed is refused — here the debaters, who argue from what they know. One toolkeeper, different terms for different members — the way a library gives professors and students different cards. (A toolkeeper open to everyone says others: standard beside the table.)
• The body is the sign on the door. It is what the members who may use the toolkeeper are told about it. They are never shown the limits; they simply get "limit reached" if they go past them. TeamRunner enforces the terms, not the agent's good behaviour.

Using the toolkeeper is a side trip, not a step. Nothing in the flow mentions web. During its own turn the fact checker may search as often as it needs — or not at all — up to its limit, and each result comes back to it alone. The choice to search, the toolkeeper's yes or no, and the search itself are all logged. (If you write no limit, a member gets five calls a turn, so a side trip always ends.)

When a search must happen, put the toolkeeper in the flow. Say the judge must check the most disputed claims before ruling. Make it a step:

rounds(*3) -> judge:research -> web -> judge:verdict

A toolkeeper step means the member just before it must end its turn with a request to the toolkeeper (that turn's answer is the request, so it gives no typed outputs); the reply goes back to that member, who reads it at its next step. The labels research and verdict tell the judge which part of its job it is on — its prompt says what to do at each (see examples/fact_checked/judge.md). Optional use stays out of the flow; required use is a step. Both are visible. Nothing in the flow line marks web as a toolkeeper — its own file says kind: toolkeeper — but teamrunner explain --diagram draws toolkeeper steps in their own shape, so you can tell them from members' turns at a glance.

teamrunner explain . prints, in plain English, who may use which toolkeeper and on what terms.

TeamRunner 0.1 has four built-in tools: web_search_brave (Brave Search), read_page (a link in, the page's text out), read_file (a file the run was given with --file, its text out) and run_code (Python in a sandbox: no internet unless the toolkeeper allows it, and files written only in the run's own work folder). The sandbox is Python running in WebAssembly, and it installs with TeamRunner — nothing else to set up; it can use packages built for it (numpy, pandas, scipy, sympy and many more), but not every package. examples/fact_checked gives the fact checker a second toolkeeper, sandbox.md, so it can check the sums in a claim by running them. A toolkeeper holds one tool in 0.1, so its limits always mean one thing; a member who needs two tools asks two toolkeepers, as the fact checker does here. examples/reference_check uses read_page and read_file: it checks every reference in a paper.

Step 8 — private information and shared records [v0.3]

The debate doesn't need this, but the moment you study organizations you will. whiteboard.yaml is the team's whiteboard: declared, typed items, each with who may read and write it. Everything anyone says is posted there too, open to all unless a read_by line says otherwise.

# whiteboard.yaml

# Bare list = a choice, starting at the first option.
# Read by everyone, written by the chair — those are the defaults.
stage: [opening, rebuttal, counter, closing, done]

# A bare value = a starting value, read and written by the defaults.
rounds_left: 3

# Spell it out only when you change who reads or writes it.
score:
  starts_as: 0
  written_by: [judge]

# Information asymmetry IS a read permission.
buyer_ceiling:
  starts_as: 850
  read_by: [buyer]

Every field is declared up front — no ad-hoc keys appearing mid-run. Every write is harness-mediated and logged as a state_write event: an agent listed in written_by returns the new value as a declared output field; nothing pokes state directly. Agents reference readable fields in their prompts as {{whiteboard.buyer_ceiling}}, and the validator checks that permission before you spend a cent.

This one file is doing a surprising amount of work. Private valuations, confidential negotiating briefs, payoffs and incentives, task inputs, an artifact handed from one department to the next — all of them are just whiteboard items with the right permissions.

Tutorial part 2: your first experiment [v0.2]

Here's the question that motivated the whole project: does it matter which model plays the advocate?

Don't touch the debate. Make a folder next to it with one file:

[v0.2] Experiments are built after the package is right. The file below is the agreed shape: it lives outside the package (experiments/advocate_model/experiment.yaml) and names it with base:.

# experiments/advocate_model/experiment.yaml
name: advocate-model-ab
base: ../../debate
task: tasks/remote_work

factors:
  advocate_model:
    - label: gpt
      patch: {advocate.model: "openai/gpt-6-luna"}
    - label: claude
      patch: {advocate.model: "anthropic/claude-sonnet-4-5"}

replications: 20        # 2 conditions × 20 = 40 runs
seed: 42

limits:
  max_concurrent: 4
  max_cost_usd: 40.00

teamrunner experiment .

What just became possible:

Factors are patches, so the base spec stays clean. advocate.model means "the model field of the component advocate". Patch paths address any field of any component, including the prompt body (the field is called prompt), and any value can be file:<path> to swap in a whole file. patch: {} is the unmodified base spec — the control condition, written honestly as a level rather than smuggled in as "the normal run."

Anything in a file is a factor. That's not a slogan, it's the mechanism. A few real examples from our worked recasts:

# A prompt/strategy A/B — same agent, two personalities
- {label: tough, patch: {buyer.prompt: file:variants/buyer_tough.md}}

# A treatment: one member is secretly self-interested
- {label: saboteur, patch: {council.prompt: file:variants/saboteur.md}}

# The communication structure ITSELF as the factor:
# same reviewers, same paper — only who can read whose review changes
- label: public
  patch:
    whiteboard.said.reviewer_1.read_by: everyone
    whiteboard.said.reviewer_2.read_by: everyone
    whiteboard.said.reviewer_3.read_by: everyone

That last one is the founding claim of the product. Studying whether peer reviewers herd when they can see each other's reviews is, in TeamRunner, a two-level factor on who may read what each reviewer says.

replications has a minimum of 2. Writing replications: 1 requires an explicit allow_single_run: true. LLMs are stochastic; single runs are anecdotes; we made the anecdote the awkward thing to write.

One master seed fixes everything the harness controls — condition assignment, ordering, per-run seed derivation. It does not, and cannot, make the model deterministic. See Why we built it.

limits is a real budget ceiling. max_cost_usd stops the experiment; max_concurrent keeps you inside your rate limits.

Then get the data out:

teamrunner runs                     # browse what you've run
teamrunner export --experiment advocate-model-ab --format csv

Long format, one row per observation, straight into R, Stata, or pandas. Your analysis never lives inside our dashboard — that's a promise, not an oversight.

The declarative metrics layer (ground truth, LLM-as-judge with a separately pinned judge model, cost and time) is specified but its file format is still open. Today's metrics: references in the examples are placeholders.

What lands in a run folder

Every run is logged in a runs/ folder at the top of the package you ran — even when you run just one subteam ([v0.3]). The folder is git-ignored, so sharing the package never shares its history.

Today a run folder holds three things: run.json (when, which team, the message you sent, a fingerprint of the package, cost), events.jsonl (below) and output.md, in a folder named for when and what you ran (runs/2026-10-12_143201_debate/) — plus inputs/, a copy of any file the run was given, and work/, where run_code keeps what it writes. [v0.2] The fuller record that follows arrives with experiments, which keep their runs in the experiment's own folder:

runs/01JZXG7T3M9QKWP5RVB2C8HD6N/
├── manifest.json     ← the receipt
├── events.jsonl      ← the camera
├── raw/              ← every request and response payload, by hash
└── resolved/         ← the spec, with every default and patch expanded

manifest.json records the condition and its patches, the master and derived seeds, timestamps, resolved model versions (what openai/gpt-6-luna actually was that day), your Python and package versions, cost and token totals, and a component inventory linking every instance path to its permanent ID and content hash.

events.jsonl is one JSON object per line, append-only, in TeamRunner's own schema:

A field never changes its meaning between versions — later versions only add — so an analysis script you write today keeps working.

The to field is worth a pause. [v0.3] The whiteboard's read_by lines for what is said declare the permitted topology; the log captures the realized one. The gap between them is itself a research finding — you gave the team an org chart, and now you can measure the one it actually used.

Query it with DuckDB directly over the files today; [v0.2] export to CSV or Parquet when you want to leave.

resolved/ [v0.2] is the quiet favourite. It is a mirror of your spec tree with every default and every patch expanded to its explicit value — full frontmatter, explicit flow, explicit permissions. diff -r debate/ runs/<id>/resolved/ shows you exactly what the defaults filled in. And:

teamrunner run runs/01JZXG7T3M9QKWP5RVB2C8HD6N/resolved

replays that exact spec. Every run folder is a self-contained replication package. Defaults are for authors; records get the fully-resolved truth.

What your runs keep, and where your words go

A run folder keeps everything, in plain text, on your machine: every prompt, everything every member said, every page a toolkeeper read, a copy of any file you gave the run, and whatever run_code wrote. Open any of it in a text editor. That is what makes a run checkable, and it also means a run folder holds whatever you put into the run, so share one only when you would share all of it.

Your keys are never in it. run.json records only where each key came from ("OPENAI_API_KEY": "key_file"), never the key. Keys live in ~/.teamrunner/.env, which only you can read, outside every team.

Sharing a team through git does not share its runs: teamrunner new puts runs/ in the team's .gitignore. If you zip or copy the folder yourself, leave runs/ out.

What you send a hosted model goes to its provider — OpenAI, Anthropic, Google — under that provider's terms, as with any app that uses them. A local model (through Ollama) keeps everything on your machine. A web_search_brave sends its query to Brave; read_page fetches the page from its site. Search results are the one thing a run does not keep: the member reads them, the log keeps the query and how many came back, because Brave's terms allow storing results only on plans with storage rights. If yours has them, web_search_brave: {keep_results: true} keeps them. Brave's terms also forbid using its results to evaluate or benchmark AI models; check them before a study.

TeamRunner itself sends nothing anywhere. No telemetry, no usage statistics. The only other things it fetches are Pyodide, the first time run_code is used, and the packages your code imports.

Going further: the worked examples

Six examples ship with TeamRunner, in examples/, simplest first — each adds something the one before it doesn't have. Copy one by name to start from it: teamrunner new my_team --from orchestrators_lite. Most are published multi-agent papers or projects, recast as TeamRunner files.

Coming with 0.3, recast on paper and waiting for the features they need: Deb8flow (subteams — each side a debater plus its own fact checker), NegotiationGym (private valuations, a loop either side can end), LLM-Deliberation (four parties, confidential briefs, one secretly given a hidden agenda), AgentReview (three reviewers at once, sealed reviews — do reviewers herd when reviews are public?), ChatDev (a three-level company, each department on its own whiteboard), and the full Invisible Orchestrators study (a hidden orchestrator whose words reach the team only as unsigned "policy updates").

The recasting campaign is the point, not decoration: each one is run before the corresponding code exists, so the spec breaks on paper where breaking is cheap. Two of the open questions in our spec today exist because agent_review and chat_dev refused to be expressed cleanly.

What TeamRunner is not

Not another agent framework — yet. TeamRunner is for asking questions about teams of agents and, with [v0.2]'s replications, getting statistically defensible answers. Its constraints (declared outputs, bounded loops, no free-text routing) are what make the answers trustworthy, and they are also what an organization wants from a team it runs for real: every decision logged, every loop bounded. A package that won its experiment is meant to be shipped — [v0.2] export hands you a standalone copy — and running packages in production, logged and improved over time, is where we intend to go. If you need a general-purpose agent framework today, use one.

Not a programming language. Conditions are exactly <field> is <value> and is not. There is no arithmetic, no boolean algebra, no expressions. Every time this has been tempting, the answer has been the same: the judgment belongs in an agent's prompt, and the wiring belongs in the flow. That line is load-bearing, and we intend to hold it.

Not an open-world simulation. Structured teams with defined roles, protocols, and rules for who sees what — not emergent town-of-agents worlds. We picked a boundary and we'd rather fail cleanly at it than blur.

Not a data roach motel. JSONL on your disk and DuckDB over your own files today; [v0.2] Parquet and long-format CSV export. The dashboard, when it comes, is a convenience, never the only door.

Status and roadmap

Preview. What the roadmap calls 0.1 is out as a soft launch, numbered 0.0.x: every command in the tutorial not marked [v0.2] or [v0.3] works. It may still change in ways that break earlier teams and runs until 0.1.0 is announced. What changed in each version is in the changelog.

Human-in-the-loop nodes (kind: human — the runner pauses and asks a person) are for mixed human–agent studies.

License: GPL-3.0-or-later, like R. Your teams, runs and results are yours, under any licence you choose; the shipped examples, the files teamrunner new writes and the authoring guide are MIT-0. Pull requests are not open yet — see CONTRIBUTING.md.

Questions, bugs and ideas

All of them are welcome, and every one is read. Open an issue on GitHub: there is a short form for a bug and one for a question or idea. No GitHub account? Write to hello@teamrunner.ai. Either way, please never include an API key, and remember that a run folder holds everything said in the run.

A security problem? Please don't open an issue: write to security@teamrunner.ai, as SECURITY.md explains.

Appendix A: file reference at a glance

Inside a package, these names are reserved. Every one of them is optional except team.yaml. Each file answers one question: team.yaml what is this team? (its charter), chair.md or manager.md how is it run?, whiteboard.yaml who sees what?, and the member files who does the work? A folder grows one file per feature you use.

[v0.2] Outside the package, in a folder of its own: experiment.yaml (names the package with base:; factors as patches, replications, seed, limits) with the study's tasks/, metrics/ (format still open), and variants/. An experiment's runs land in the experiment's own folder.

References use bare names within a folder (judge) and paths across scopes (../fact_checker, analytics/critic); a model profile keeps its .yaml. Links to shared files outside the package are fine while you work — validate and run warn about each one loudly, and teamrunner gather copies them in before you hand the package over. YAML is canonical; JSON is accepted. Quote your strings — no is a word, not a boolean.