onebudgetspec

Register measurable budgets in YAML and gate on them. Each budget names a command that performs the measurement, a unit, a direction and a threshold; onebudgetspec check runs every selected command once and reports the actual value, the budget and the headroom.

It ships as a Rust SDK (onebudgetspec-core) and a command line (onebudgetspec), with Python and TypeScript SDKs released beside them at the same version (see SDKs).

Principles

• Measure once. Each selected budget's command runs exactly once per check. There is no retry, repeat or sampling.
• Report actual, budget and headroom. Every result carries all three, so even a result within budget shows how close it came.
• Record host conditions beside each result. The load average, CPU count, available memory and any conditions the file declares are recorded with every measurement, so a person can judge an extreme circumstance from the result itself.
• Over budget is a defect, never re-sampled. An over-budget result fails the check on any host. It is never re-run until it passes.
• No baselines. A budget is the threshold written in the file, not a comparison with an earlier run.

Install

pip install onebudgetspec-cli         # or: uvx --from onebudgetspec-cli onebudgetspec
npm install --save-dev @onebudgetspec/cli
cargo install onebudgetspec

Each installs a native binary for Linux, macOS and Windows, on x86_64 and aarch64.

The budgets file

A file named budgets.yaml:

schema_version: 1
conditions:            # optional; recorded beside every result measured from this file
  - name: dispatches   # ^[a-z][a-z0-9_]*$, unique, never load1, cpus or mem_available_mib
    command: ["bash", "scripts/count-dispatches.sh"]  # run once per file per check; trimmed stdout recorded
budgets:
  - id: gate-time      # ^[a-z][a-z0-9-]*$, unique across every file one discovery finds
    description: "Wall clock of the full gate"  # optional
    labels: [ci]       # optional free tags, for selecting a subset
    measure: reported  # elapsed | reported
    command: ["bash", "scripts/budget-gate-time.sh"]  # argv, no shell, run from this file's directory
    unit: seconds      # a free label; `elapsed` requires seconds
    direction: max     # max: within when actual <= threshold; min: within when actual >= threshold
    threshold: 1800    # finite and non-negative
    timeout_seconds: 3600  # optional; past it the measurement is an error

Unknown keys are refused at every level, naming the key and the file.

A command is an argv run with no shell, from the directory holding its file; a program named by a relative path with more than one component is found from that directory too. On Linux and macOS an executable script may be named directly, by its #! line. Windows reads no #! line, and a bare name there resolves only to <name>.exe, so a script measured on Windows names its interpreter in the command, as in ["bash", "scripts/budget-gate-time.sh"], ["node", "measure.js"] or ["python", "measure.py"]; that form runs on every platform. A timeout_seconds ends everything the command started: its process group on Linux and macOS, its Job Object on Windows. A failed command is described by its exit status, which on Windows is the exit code (in hexadecimal, such as 0xC0000005, when it is a crash's NTSTATUS); on Linux and macOS a command ended by a signal is described by that signal. load1 is unknown (null) on Windows, which has no load average, and mem_available_mib is unknown on macOS.

elapsed times the command's wall clock; a non-zero exit is an error. reported sets ONEBUDGETSPEC_RESULT to the path of a fresh, empty file, and the command writes one JSON object there and exits 0:

{"value": 1395.0, "detail": "optional text", "conditions": {"runner": "large"}}

detail and conditions are optional. Returned conditions are recorded in that result's host conditions only; a returned name equal to a declared condition, load1, cpus or mem_available_mib makes the measurement an error rather than replacing a sampled value. A command's own stdout and stderr go to stderr, never into a report, with one exception: when a budget's command exits non-zero, is ended by a signal or its timeout, or exits 0 with a result that cannot be read, its result's error states that (for an unreadable result, why it could not be read and then the exit status), followed by ; its stderr: and the tail of what the command wrote there. That tail is decoded lossily, each line trimmed, blank lines dropped, the rest joined by | and other control characters made spaces, and it is bounded to at most the last 1000 characters, after a leading … when anything was cut. A command that wrote nothing but whitespace there gets no such suffix; a command that succeeds is reported without its stderr. A failing condition command is still recorded as unknown, and the same reason appears on the line onebudgetspec prints to stderr for it.

The command line

onebudgetspec check [PATH]... [--id ID]... [--label LABEL]... [--exclude-label LABEL]... [--recursive]
onebudgetspec validate [PATH]... [--recursive]
onebudgetspec list [PATH]... [--id ID]... [--label LABEL]... [--exclude-label LABEL]... [--recursive]
onebudgetspec schema

With no PATH, ./budgets.yaml is read. --recursive searches a directory for files named budgets.yaml, honouring .gitignore. Every verb takes --output text|json (--json for short). schema prints the JSON Schema bundle for the budgets file and both reports.

A text result reads:

budget gate-time: actual 1395 seconds, budget 1800 seconds, headroom 405 seconds (22.5%) — within; host: load=2.1/8 mem_available=5120MiB dispatches=3

The npm launcher adds three of its own, each with the reason on stderr: 64 on a platform no carrier is published for, 69 when the carrier package is missing or its binary cannot run, and 70 when the binary was ended by a signal.

SDKs

Three SDKs, released at the binary's version. The Rust SDK is the engine itself; the Python and TypeScript SDKs run the onebudgetspec binary once per call, parse the JSON it prints, and return the reports as types generated from onebudgetspec schema, so they never drift from it.

In Python and TypeScript, exit statuses 0, 1 and 3 return the report, since the verdicts are in it; status 2 (an invalid invocation or budgets file) raises OnebudgetspecError (Python) or rejects with it (TypeScript), carrying the CLI's own message and exit_code/exitCode 2.

from onebudgetspec_sdk import check

report = check(labels=["api"], exclude_labels=["slow"], cwd="services")
for result in report.results:
    print(result.id, result.verdict, result.actual, result.headroom, result.host.conditions)

import { check } from "@onebudgetspec/sdk";

const report = await check({ labels: ["api"], excludeLabels: ["slow"], cwd: "services" });
for (const result of report.results) {
  console.log(result.id, result.verdict, result.actual, result.headroom, result.host.conditions);
}

Nesting budgets files

Each project registers its own budgets beside its code, and only the budgets a change can affect run; the budgets that must always be checked go in the root file. With Nx as the worked example, a repository lays them out as:

budgets.yaml               # always checked
services/api/budgets.yaml  # checked when services/api is affected
services/api/project.json

The project's own file:

schema_version: 1
budgets:
  - id: api-cold-start
    description: "Seconds from process start to the first healthy response"
    measure: elapsed
    command: ["bash", "scripts/start-and-probe.sh"]
    unit: seconds
    direction: max
    threshold: 2
    timeout_seconds: 30
  - id: api-requests-per-second
    measure: reported
    command: ["bash", "scripts/load-test.sh"]
    unit: requests/s
    direction: min
    threshold: 500

checked by that project's budgets target in services/api/project.json:

{
  "targets": {
    "budgets": {
      "command": "onebudgetspec check services/api/budgets.yaml",
      "inputs": ["{projectRoot}/**/*"]
    }
  }
}

and run for the affected projects only:

nx affected -t budgets

The root file holds what every change must stay within:

schema_version: 1
conditions:
  - name: runner
    command: ["node", "-p", "process.arch"]
budgets:
  - id: gate-time
    description: "Wall clock of the full gate"
    measure: elapsed
    command: ["just", "check"]
    unit: seconds
    direction: max
    threshold: 1800
    timeout_seconds: 3600

checked on every change with onebudgetspec check budgets.yaml. To check every file at once, onebudgetspec check --recursive . discovers both.

Licence

MIT