homeostat-live

Take part in the Homeostat Live benchmark from the command line. Each week, predict the lab results of a simulated process plant from its sensor data.

homeostat_live register alice                # once: your token is kept in ~/.config/homeostat-live
homeostat_live benchmarks                    # the benchmarks, the open one first
homeostat_live benchmark                     # download the open benchmark and every earlier week
homeostat_live submit 2026-10-09 isidis-01 homeostat-live-data/isidis-01/benchmarks/2026-10-09/predictions.csv --note "v1"
homeostat_live history                       # your submissions and scores
homeostat_live leaderboard                   # the ranking of the last frozen week, and over weeks
homeostat_live predictions --out p.csv      # what the reference models and the best participants predicted, next to the lab results
homeostat_live score my.csv --week 2026-10-09 --plant isidis-01   # backtest on a released week

homeostat_live predictions writes one CSV with a row per lab sample of the released weeks (the last four by default): the time, the week, the tag, the grade then and the half width of its specification (the unit of the normalised error), the lab result, and a column per model, by default the reference models and the best participants of the last scored week (--models a,b for others, up to six; --from and --to for other weeks, named by the Friday that closes them). --mine my.csv adds your own predictions (the columns tag, sampled_at and value) as one more column. Read it with pandas to see where a model errs and where the others do. A participant's predictions are public once their benchmark is scored, as the lab results are.

A benchmark opens every Friday at 17:00 UTC with the week that has just closed. It asks for every lab result of that week, and it freezes a week later, when those lab results are released and every submission is scored. Your last submission before the freeze counts. A prediction for a sample may use only data up to the time the sample was taken: sensor data up to then, and the lab results of earlier, released weeks.

homeostat_live benchmark writes a folder with a README that explains its layout. Every command takes --json, which prints the API's answer as it is, for scripts and AI assistants. --server (or HOMEOSTAT_LIVE_SERVER) points the command at another site.

The command needs Python 3.10 or later and nothing beyond the standard library; building models with it (below) needs NumPy and pandas.

Building a model, with an AI coding agent or without

pip install 'homeostat-live[models]'
homeostat_live init my-model                 # AGENTS.md, model.py and the plant's data
cd my-model
homeostat_live backtest model.py             # score it on the last 4 released weeks, next to the reference models
homeostat_live backtest model.py --all       # on every released week; --week <date> for one week with every metric
homeostat_live predict model.py              # predictions for the open benchmark, checked, ready to submit
homeostat_live check <predictions.csv>       # what the API would refuse, and what looks wrong
homeostat_live data                          # bring the data up to date (backtest and predict do it themselves)

init writes a project that coding agents such as Claude Code, Codex or Cursor can work in: AGENTS.md explains the task, the files, the model interface, the commands, the rules and the scoring (CLAUDE.md points to it), and model.py is a working model to improve: a copy of the reference model ref-linear.

A model is a class with up to four methods; the harness (homeostat_live.harness) makes one per lab analysis and per predicted week:

class Model:
    def __init__(self, plant, tag): ...   # the plant's metadata, and the lab analysis to predict
    def features(self, sensors): ...      # optional: sensor rows (a column per tag, indexed by time) -> features
    def fit(self, X, y): ...              # the features at the sample times of the released results, and the results
    def predict(self, X): ...             # one sample's row of features -> one value

The harness keeps the causal rule by construction: fit gets only the lab results released before the predicted week, predict is called once per sample with the features at its time, and the features must be causal: at a few sample times, the harness computes them again without the data after the sample and stops with an error that names the columns if they change (a centred window, shift(-1), bfill, a mean of the whole frame). A backtest predicts each released week as the benchmark would have and scores it with the official metrics, next to the reference models, which run through the same code on the leaderboard:

• ref-last-lab (naive): the last released result of the same grade.
• ref-linear (linear): a ridge regression on the sensor values at the sample time, predicting the distance from the grade's target in half tolerances around each grade's level, trained on the last eight released weeks. It ignores the process' dynamics on purpose.

Scoring

Every submission is scored when its benchmark freezes, with the code in homeostat_live.metrics; homeostat_live score runs the same code on any released week, so a model can be backtested with exactly the metrics it is ranked by.

• Normalised error: |prediction − lab| divided by half the width of the range of the grade being made when the sample was taken, so that both lab analyses and every grade count alike; each sample counts at most 5. The ranking is its mean over every target of the week (lower is better).
• Accuracy, per analysis: mean absolute error, root mean square error, bias, median, 90th-percentile and worst absolute error, in the analysis' units; the shares within a quarter and within half of the tolerance.
• Grade changes: a change's window runs from the start of its ramp to 12 hours after its end. Inside the windows: the normalised error and its ratio to steady state; the time until the lab and the prediction reach the new grade's range and stay there, and the difference between the two; the worst error of each change.
• Off spec: off spec being the positive class, precision, recall, F1 and balanced accuracy, per analysis and for the product (off spec when either analysis is), overall and inside the windows.
• Dynamics: the share of the lab's moves between consecutive samples that the prediction follows in direction, and the correlation of their changes.
• Skill: 1 − MAE / the reference model's MAE, against ref-last-lab and ref-linear (above).
• Per grade: the normalised error and each analysis' error, grade by grade.

The leaderboard ranks every participant (from the first scored week on; a user who breaks the rules is removed) and the reference models by a frozen week, by the last four weeks and by all weeks.