homeostat-live

Take part in the Homeostat Live benchmark from the command line. Each week, predict the lab results of a simulated process plant from its sensor data.

homeostat_live register alice                # once: your token is kept in ~/.config/homeostat-live
homeostat_live benchmarks                    # the benchmarks, the open one first
homeostat_live benchmark                     # download the open benchmark and every earlier week
homeostat_live submit 2026-10-09 isidis-01 homeostat-live-data/isidis-01/benchmarks/2026-10-09/predictions.csv --note "v1"
homeostat_live history                       # your submissions and scores
homeostat_live leaderboard                   # the ranking of the last frozen week, and over weeks
homeostat_live score my.csv --week 2026-10-09 --plant isidis-01   # backtest on a released week

A benchmark opens every Friday at 17:00 UTC with the week that has just closed. It asks for every lab result of that week, and it freezes a week later, when those lab results are released and every submission is scored. Your last submission before the freeze counts. A prediction for a sample may use only data up to the time the sample was taken: sensor data up to then, and the lab results of earlier, released weeks.

homeostat_live benchmark writes a folder with a README that explains its layout. Every command takes --json, which prints the API's answer as it is, for scripts and AI assistants. --server (or HOMEOSTAT_LIVE_SERVER) points the command at another site.

The command needs Python 3.10 or later and nothing beyond the standard library.

Scoring

Every submission is scored when its benchmark freezes, with the code in homeostat_live.metrics; homeostat_live score runs the same code on any released week, so a model can be backtested with exactly the metrics it is ranked by.

• Normalised error: |prediction − lab| divided by half the width of the range of the grade being made when the sample was taken, so that both lab analyses and every grade count alike; each sample counts at most 5. The ranking is its mean over every target of the week (lower is better).
• Accuracy, per analysis: mean absolute error, root mean square error, bias, median, 90th-percentile and worst absolute error, in the analysis' units; the shares within a quarter and within half of the tolerance.
• Grade changes: a change's window runs from the start of its ramp to 12 hours after its end. Inside the windows: the normalised error and its ratio to steady state; the time until the lab and the prediction reach the new grade's range and stay there, and the difference between the two; the worst error of each change.
• Off spec: off spec being the positive class, precision, recall, F1 and balanced accuracy, per analysis and for the product (off spec when either analysis is), overall and inside the windows.
• Dynamics: the share of the lab's moves between consecutive samples that the prediction follows in direction, and the correlation of their changes.
• Skill: 1 − MAE / the reference model's MAE, against ref-last-lab (the last released result of the same grade) and ref-soft-sensor (a ridge regression on the sensor data, trained on the released weeks).
• Per grade: the normalised error and each analysis' error, grade by grade.

The leaderboard ranks reviewed users and the reference models by a frozen week, by the last four weeks and by all weeks.