publishable

Every parameter in one file. Every run reproducible from one command.

You have an experiment to run. Here's the whole arc with publishable:

1. Design the run. One config file holds every parameter, the conditions you're comparing, and the repeats you'll average over. publishable generates it fully populated — you edit rather than author.
2. Run it. publishable run config.yaml. No flags. Conditions and repeats expand on their own, and each one gets its own place in the output tree.
3. Read the results. Estimates, confidence intervals over your units, and effect sizes against your baseline, already computed and sitting next to the run that produced them.
4. Publish it. Hand a collaborator, a reviewer, or your future self one file. publishable reproduce rebuilds exactly what ran — same commit, same locked environment, the input checked against a recorded manifest — and stops at the two things only a person can supply: your copy of the data, and your credentials.

Nothing about what ran ends up in a shell history, so nothing has to be reconstructed later.

Try it

Requires Python 3.11+ and uv. Nothing else to configure:

uv tool install publishable        # or: pipx install publishable, brew install seouri/tap/publishable
publishable demo

demo builds a complete worked example — synthetic data, a three-step pipeline, a parameter sweep — and then walks you through running it, one command at a time. It opens with the part you can't easily make yourself:

Created publishable-demo/
  240 synthetic units      ~/publishable-demo-data/input/
  template                 templates/correlation.py
  experiment               src/correlation_pilot/
  config                   configs/correlation-pilot/config.yaml

Your data sits outside the repo, where real data belongs. Everything from
here is the CLI you'd use on an experiment of your own.

Next:  a look at the config that describes this run
       [Enter] to continue · q to stop here

Six stops, the middle three with the same beat: the real command, its real output, then what it meant. After the config comes validate:

Next:  publishable validate configs/correlation-pilot/config.yaml
       [Enter] to run it · q to stop here

  ✓ config valid · configs/correlation-pilot/config.yaml

validate read your config and your data. It created nothing and reached
nothing off this machine — the 240 units it resolved came from the
index.csv outside the repo, and `input_dir` being outside is enforced.

Nothing you press changes what runs — every stop is proceed-or-quit, and the config written at the first stop is the one executed at the fifth. q prints the remaining commands and publishable demo picks up where you left off. Two stops later, past dry-run:

Next:  publishable run configs/correlation-pilot/config.yaml
       [Enter] to run it · q to stop here

  warning W-ENV-UNLOCKED       environment
          no uv.lock found at ~/publishable-demo; the environment is not
          pinned, and `reproduce` will not be able to restore it
1 problem (0 errors, 1 warning)
run.yaml → ~/publishable-demo-data/results/run_2026-08-07T09-14-03Z_2f5c8d0/run.yaml

W-ENV-UNLOCKED fired because this project has no uv.lock: its pyproject
depends on `publishable`, which cannot resolve until the package is
published, so there is nothing to pin yet. Nothing was suppressed.

run printed no table — its whole output is that warning and the path to the
record. Everything below is `demo` reading the record back:

  condition             r       95% CI            vs baseline (paired, 95% CI)
  00_baseline           0.697   [0.630, 0.757]    —
  01_method=spearman    0.666   [0.582, 0.739]    -0.031  [-0.068, -0.002]
  02_method=kendall     0.482   [0.413, 0.550]    -0.215  [-0.240, -0.190]

  intervals over 228 of 240 units (12 failed) · seed spread std 0.003 of recorded `pred`

  `pred` and `truth` are recorded columns, so each publishes its own
  metric and joins the correction family beside `r` — six members, and
  two of them nobody reads. A template that derived twenty diagnostics
  would correct every interval in the run for numbers nobody reads.

Those are the numbers demo computes on your machine, from the 240 rows it generated — run itself prints no table, no banner and no progress bar, so everything under the record's path is demo saying what the record means. The sweep is 3 conditions × 5 repeats = 15 repeat-scoped executions and 19 in all: the plan also runs step01_load_cohort once for the whole sweep and step02_fit_model once per condition, and 19 is the figure dry-run prints. Absolute paths are elided to ~ above, and the run.yaml path is illustrative either way — a run ID carries the timestamp and the code hash of your run.

That run.yaml is the point, and the last stop opens it and leaves you there. It carries the results and everything needed to regenerate them — so on any other machine:

publishable reproduce <path-to-run.yaml>

clones the exact commit, restores the locked environment, writes the config back out, and prints what's left to fill in. Neither data nor credentials travel, so the last step is yours — and run then verifies the input against the recorded manifest before spending anything.

v0.x — the design is settled, interfaces may still shift before 1.0. Issues and design feedback are very welcome.

Is this for you?

A good fit if you run experiments with parameters you sweep, repeats you average over, and results that end up in a paper — especially with data that can't live in your git repo. What a run executes is your pipeline over that data; measurements taken outside the pipeline arrive as input and are pinned by hash, not produced.

Probably not if you want a pipeline scheduler, a live dashboard, or something to retrofit onto existing scripts. publishable is greenfield only.

Designs it speaks natively: within- and between-subjects (with recorded randomization), factorial over parameters or over crossed arms, ablation, dose-response, train-test holdout, repeated cross-validation, bootstrap, permutation, technical-vs-biological replication, clustered units, matched case-control. The statistics follow the design — bootstrap resamples get percentile intervals, technical replicates never enter n, and a multi-condition sweep won't report uncorrected comparisons without warning you. Modelling beyond summary statistics is yours: see Experimental designs for what's supported, what needs an override, and the errors core refuses to let you make.

These overlap, and several compose fine with publishable. The distinguishing bet: the deliverable is a paper, so the tool should produce something a reviewer can check, not just something you can query.

How it fits together

Five words carry the whole model:

Statistics aggregate within a condition and compare across conditions. Getting that backwards is the most common way a reproducible pipeline still produces a wrong number, which is why the two are named separately rather than both being called "runs."

Where things live

Your repo holds code and configs — never data:

my-study/
├── src/cohort_pilot/          # your pipeline, one file per step   → code_hash
│   ├── experiment.py          #   declares step order
│   └── steps/
│       ├── step01_load_cohort.py
│       ├── step02_fit_model.py
│       ├── step03_analyze.py
│       └── step04_compare_methods.py
├── templates/                 # this project's own templates, if any  → code_hash
├── configs/cohort-pilot/
│   └── config.yaml            # every parameter                    → parameters_hash
├── pyproject.toml + uv.lock   # the environment                    → locked
└── .env                       # credentials, never committed

Your data lives outside it, and so does everything a run produces:

~/results/cohort-pilot/
└── run_2026-08-07T09-14-03Z_8e21ab3/
    ├── run.yaml                        ← results + all three hashes. This is the deliverable.
    ├── conditions/
    │   ├── 00_baseline/                ← one folder per condition, self-describing
    │   │   ├── seed17/                 ← one folder per repeat
    │   │   │   └── step03_analyze/scores.parquet
    │   │   └── seed42/…
    │   └── 01_method=spearman/…
    └── summary/                        ← steps that compare across conditions

The directory structure is the experiment structure, so finding an artifact never requires reading code. Nothing here is ever overwritten: a second run creates a new run_<id>/ beside this one.

And then

run.yaml  ──►  publishable reproduce   ──►  a checkout anyone can run
          └─►  publishable study add   ──►  a bundle beside your manuscript

Full vocabulary: Ontology.

What you actually write

One config, generated fully populated by publishable generate experiment (or its shorter alias publishable init), so you edit rather than author from scratch:

parameters:
  analysis:
    method: pearson              # choices: pearson | spearman | kendall
    min_samples: 30              # integer >= 2

sweep:
  baseline: {analysis.method: pearson}
  grid:
    analysis.method: [spearman, kendall]

replication:
  repeats:
    - {kind: seed, n: 5}         # seed | batch | fold — what a re-execution varies

And steps that never mention sweeps — each condition is resolved before your code runs:

class Step(BaseStep):
    scope = "repeat"                      # run once per repeat, per condition

    def run(self, cfg, io):
        # io.units — your patients, samples, trials: whatever you're measuring
        # cfg.parameters — already resolved to THIS condition's values
        result = analyze(io.units, method=cfg.parameters.analysis.method)

        for unit, pred in zip(io.units, result.per_unit):
            io.record(unit.key, {"pred": pred, "truth": unit.label})   # the per-unit table —
                                                                       #   what every interval is over
        io.write("scores.parquet", result.rows)  # rows in, rows back out — this condition +
                                                 #   repeat's own folder
        return {"r": result.r}               # recorded per repeat, without an interval

Write it once for a single condition; adding a sweep later changes nothing here.

Statistics come back computed, next to the hashes that make them checkable:

results:
  conditions:
    - label: method=spearman
      aggregated:
        step03_analyze: {r: {value: 0.607, basis: units, n: {completed: 228},
                             ci95: [0.517, 0.683], repeat_spread: {std: 0.014}}}
      vs_baseline:
        step03_analyze: {r: {delta: 0.026, paired: true, ci95: [-0.007, 0.059]}}
provenance:
  code_hash: sha256:8e21…            # your src/** + templates/**, from a clean tree
  parameters_hash: sha256:1a2b…      # this exact parameter set
  input_manifest_hash: sha256:3d8a…  # the data it actually read

The r carrying an interval is the derived one: template generic's aggregate recomputes it from the per-unit table the step recorded, and being recomputable on a resampled table is what earns it a ci95. The r the step returned sits beside it under per_repeat, without one. That is the whole rule — an interval is over units, never over executions.

Start your own

# 1. Scaffold a repo. Runs `git init`; writes README, LICENSE, CITATION.cff.
publishable new my-study && cd my-study

# 2. Create an experiment. Data paths must live outside the repo.
publishable generate experiment cohort-pilot \
  --template generic \
  --input-dir ~/data/cohort-2026 \
  --output-dir ~/results/cohort-pilot

# 3. Fill in the two fields `init` leaves for you — metadata.description and
#    metadata.authors — and put your data where the config points.
#    `validate` names anything still missing, and costs nothing.
publishable validate configs/cohort-pilot/config.yaml

# 4. Run it. The scaffold's starter step records one number per unit, so this
#    produces a real estimate with an interval — replace it with your own.
git add -A && git commit -m "Scaffold cohort-pilot"
publishable run configs/cohort-pilot/config.yaml

# 5. Now make it yours: add steps, implement them, commit.
publishable generate step cohort-pilot analyze
git add src/ && git commit -m "Implement cohort analysis"
publishable validate configs/cohort-pilot/config.yaml
publishable dry-run  configs/cohort-pilot/config.yaml
publishable run      configs/cohort-pilot/config.yaml

# 6. When you publish, collect the runs you're reporting.
publishable study new ~/papers/triage/study --title "Cohort triage pilot"
publishable study add ~/papers/triage/study ~/results/cohort-pilot/latest/run.yaml --as main

Step 4 works before you've written any code — the starter step records a number, so a first run produces a real estimate with an interval — so you can see the whole loop before committing to it.

The loop you'll actually live in

Once the code is committed, changing an experiment means editing one file. No commit, no flags, no bookkeeping:

# edit configs/cohort-pilot/config.yaml — say, min_samples: 30 → 50
publishable run configs/cohort-pilot/config.yaml

publishable diff ~/results/cohort-pilot/run_A/run.yaml \
                 ~/results/cohort-pilot/run_B/run.yaml

A  run record  run_A  completed
B  run record  run_B  completed
code_hash          identical    sha256:8e21…
input_manifest     identical    sha256:3d8a…
uv.lock            identical    sha256:6b1f…
parameters_hash    DIFFERS
  parameters.analysis.min_samples   30 → 50

That's the payoff of hashing code and parameters separately: you get to prove only one thing changed, which is the claim a comparison rests on.

What you get

• One file, no flags. No operation command takes an argument other than a path; creation commands take what's needed to bring something into existence, and they're the only exception. A selector flag would live in a shell history that nobody archives.
• Code and parameters hashed separately. code_hash covers your code trees only, so "same code, different parameters" is a provable claim — even across commits weeks apart.
• Code, environment, and data all pinned — and the apparatus too, when you measure through one. A content hash over your code trees, with the commit recorded so a reproduction can fetch them; uv.lock for the environment; a content manifest for the input data, which is the one most tools leave open. And when measurement goes through an apparatus core can't install — a hosted model deployment, an instrument — a plugin probe records its revision per condition and a change fails the run. uv.lock pins the client; that record pins the server.
• Artifacts are append-only and atomic. Nothing is ever overwritten or deleted, and a crash mid-write leaves nothing behind rather than a half-file that blocks the retry.
• Code and data never share a repo. Data paths are structurally forbidden inside the git repo — code is shareable, governed data isn't, and they need different protocols.
• Intervals over units, not over executions. n counts the things your claim generalizes over. Repeats are reported as pipeline stability, separately and labelled, because an interval across five seeds narrows as you add seeds and says nothing about your cohort. Where core can't compute an interval honestly, it reports the estimate and omits the interval.
• Statistics that match your design. Declaring how units are allocated and how repeats are structured determines the analysis: paired or unpaired, t-based or percentile, clustered or not. A t-interval over bootstrap resamples is wrong, and core won't compute one — nor will it count technical replicates as n.
• Pre-registration for free. The config is written and hashed before the run, so declared hypotheses can be checked against results — and after-the-fact additions don't match the hash.
• Stated limits. Core documents what it does not promise — bit-identical reruns against external services, verification of your Python, or scientific validity.

Commands

Creation commands take a name and what's needed to create it. Everything else takes paths only.

Full details: CLI reference — which also carries the Status column saying which of these the current build executes.

Extending it

Core knows nothing about LLMs, cohorts, instruments, or solvers. Domain knowledge lives in plugins, and a plugin doesn't need a PR here:

publishable generate experiment triage-pilot \
  --plugin someuser/publishable-llm \
  --template my_diagnostic \
  --input-dir ~/data/xray-2026 --output-dir ~/results/triage-pilot

--plugin owner/repo is uv add git+… and nothing more — so the plugin lands in your lockfile and gets captured in provenance like any other dependency. Write your own with publishable plugin new, which scaffolds an installable package with its entry points already wired.

A plugin ships machinery; the experiment type stays yours. Its unit resolvers, apparatus probes, writers and readers all dispatch straight from an install. A template it registers does not: core resolves an installed template's name from package metadata and never loads the class behind it, so a config naming one is refused (E-TEMPLATE-INSTALLED-UNSUPPORTED). That is what this project ships rather than a stage it passes through. So --template names a core template or one of your own — publishable generate template my_diagnostic writes it into templates/, where code_hash covers it, which is where the meaning of your parameters belongs anyway.

The reference LLM plugin is publishable-llm.

Documentation

• Experimental designs — how to express each design, and the mistakes core prevents
• Reference — config schema, CLI, io API, templates, sweeps, artifact layout
• Design principles — why the rules are what they are; read before proposing a change
• Plugin guide — building and sharing domain machinery
• Writing a plugin — non-normative: the tutorial, from why you need one to a run that used it
• The development record — non-normative: how each slice was designed, measured and reviewed, kept because a tool arguing the record should be publishable owes its own

Contributing

Most new experiment types should be a template in your own templates/, and most new domain machinery a plugin — neither is a PR here, which is what generate template and --plugin owner/repo are for. Upstream contributions are for core mechanisms: the config envelope, provenance capture, the artifact model, the validation engine, the CLI.

A good test of whether something belongs in core: would it be identical for a wet-lab assay, a simulation sweep, and an LLM benchmark? If not, it's a plugin.

Design disagreements are welcome as issues. If a rule seems arbitrary, design-principles.md probably explains it — and if it doesn't, that's a documentation bug worth filing.

License

MIT — see LICENSE.