heat-governor

Keeps machines running heavy AI work without thermal resets, while losing as little throughput as possible. One daemon per host senses the thermal zones about once a second, tells clients when to hold new work, duty-cycles and CPU-caps the containers that opt in, and records each run so the next one is tuned better.

It was built after a DGX Spark hard-reset twice in one day under long-context LLM decode: the hottest ACPI zone reached ~95 C while the GPU sat at ~80 C, and the zone moved up to 5 C in two seconds. See docs/HANDOFF-2026-10-23.md.

heat-governor is also an AgentCulture mesh agent with an agent-first CLI. The runtime package has no third-party dependencies.

Quickstart

uv tool install heat-governor     # or, from a checkout: uv sync && uv run heat-governor ...
heat-governor capabilities        # what this host lets us read and change
heat-governor sense               # one reading of every zone
heat-governor run --once          # one dry-run tick: what it would do right now
heat-governor run                 # the daemon, signalling and recording only
heat-governor run --apply         # ...and governing the containers that opted in

Install it as a user service:

heat-governor unit -- --apply > $XDG_CONFIG_HOME/systemd/user/heat-governor.service
systemctl --user daemon-reload && systemctl --user enable --now heat-governor
loginctl enable-linger "$USER"

How it governs

flowchart LR
  S["sense, about 1 s<br/>board zones + GPU temp"] --> C["controller<br/>effective temp, level"]
  C --> A["admit / hold<br/>new work"]
  C --> D["duty-cycle<br/>batch containers"]
  C --> K["CPU cap<br/>opted-in containers"]
  C --> U["CPU quota<br/>user units: levels or adaptive"]
  C --> G["GPU ceiling<br/>min(operator, band)"]
  A & D & K & U & G --> R["ledger first, then act<br/>records per tick"]

The operator's ceiling is law

flowchart LR
  B["policy band<br/>for the level"] --> M{"min"}
  O["operator ceiling<br/>config gpu_ceiling_mhz /<br/>gpu-clock hold or lock --apply /<br/>observed on a Jetson"] --> M
  M --> W["written ceiling"]
  B -.->|band above operator| L["log: gpu: would raise the ceiling to X<br/>-- operator ceiling Y MHz holds"]

Contract: docs/contracts/gpu-clock.md.

Adaptive user-unit quota

Opt-in ([[units]], e.g. app.slice, where interactive sessions and their test runs live). Every change is systemctl --user set-property --runtime, so a reboot drops it.

flowchart LR
  T["interval trend<br/>least-squares slope<br/>+ its standard error"] --> E{"error vs<br/>noise band"}
  E -->|above, unit busy| TI["tighten:<br/>error / learned gain"]
  E -->|inside| LW{"long window<br/>4 intervals"}
  LW -->|creep beyond its noise| TI
  LW -->|no| H["hold"]
  E -->|below, or unit idle, or cap not binding| RE["release one step"]
  TI --> LE["next interval:<br/>learn gain if the fall beats the noise"]
  LE --> S[("adaptive.json<br/>per platform + unit")]

Contract: docs/contracts/user-units.md. Evidence: docs/evidence/2026-10-24-adaptive-units-live.md.

Measured: GPU clock ceilings (2026-10-24)

DGX Spark

xychart-beta
  title "Long-prompt read speed at 128k context (tok/s)"
  x-axis "GPU ceiling (MHz)" [1800, 2100, 2200, 2300, "2436 default"]
  y-axis "tok/s" 900 --> 1250
  bar [970, 1081, 1122, 1152, 1188]

xychart-beta
  title "GPU power while reading long prompts (W)"
  x-axis "GPU ceiling (MHz)" [1800, 2100, 2200, 2300, "2436 default"]
  y-axis "W" 0 --> 90
  bar [37.2, 50.0, 57.2, 63.0, 82.8]

xychart-beta
  title "Board zone peak while reading long prompts (C); line = reset level"
  x-axis "GPU ceiling (MHz)" [1800, 2100, 2200, 2300, "2436 default"]
  y-axis "C" 60 --> 100
  bar [73.4, 80.7, 86.3, 89.9, 95.0]
  line [94.5, 94.5, 94.5, 94.5, 94.5]

xychart-beta
  title "Decode, LiveCodeBench, 2 streams (tok/s)"
  x-axis "GPU ceiling (MHz)" [1400, 1800, 2200, "2436 default"]
  y-axis "tok/s" 0 --> 45
  bar [34.9, 36.5, 38.1, 38.2]

Jetson AGX Orin (decider)

xychart-beta
  title "Decider requests/s vs GPU ceiling"
  x-axis "GPU ceiling (MHz)" [714, 918, 1122, "1300 full"]
  y-axis "req/s" 0 --> 1
  bar [0.511, 0.647, 0.780, 0.886]

xychart-beta
  title "Orin board peak (C); line = critical"
  x-axis "GPU ceiling (MHz)" [714, 918, 1122, "1300 full"]
  y-axis "C" 60 --> 100
  bar [75.3, 80.2, 83.6, 94.1]
  line [94, 94, 94, 94]

Jetson AGX Thor (worker)

xychart-beta
  title "Worker decode tok/s (bar) and GPU rail W (line)"
  x-axis "GPU ceiling (MHz)" [900, 1100, 1300, "1575 full"]
  y-axis "tok/s or W" 0 --> 110
  bar [83.2, 93.6, 95.0, 98.4]
  line [24.3, 26.9, 28.1, 29.0]

Evidence: docs/evidence/2026-10-24-gpu-clock-ceiling-spark.md, docs/evidence/2026-10-24-gpu-clock-ceiling-jetsons.md.

Configure and tune

Everything above is a setting in one TOML file that a running daemon re-reads within about 2 s. A bad edit is rejected and the last good settings stay.

heat-governor config init --write     # $XDG_CONFIG_HOME/heat-governor/config.toml from the built-in profile
$EDITOR $XDG_CONFIG_HOME/heat-governor/config.toml
heat-governor config check            # every problem at once; unknown keys are errors
heat-governor config show             # what is in force
heat-governor gpu-clock helper        # the root helper + sudoers line for GPU clocks

For other tools

• Label your containers, or read the signal before starting new work: heat-governor wait --timeout 1800 && next-job, or in Python heat_governor.client.wait() / client.Gate(4).slot().
• The JSON contract: docs/contracts/state.md.
• The guide for lobes, sparkrun, mjolnir and scripts: docs/integration.md.

CLI

Every command supports --json. Results go to stdout and errors/diagnostics to stderr. Exit codes: 0 success, 1 user error, 2 environment error, 3 wait timed out while held.

Development

uv sync
uv run pytest -n auto
uv run teken cli doctor . --strict    # the agent-first rubric gate CI runs

Agent harnesses

Four harnesses can work in this repo. Each reads exactly one root prompt file, and there is deliberately no AGENTS.md acting as a shared base:

culture.yaml declares backend: claude, so CLAUDE.md is the mesh resident's prompt. That declaration does not stop you running any of the four harnesses interactively. See docs/harness-selection.md and docs/automation-contract.md. scripts/harness-smoke.py checks that all four configs are correct.

The guildmaster skill kit is vendored under .claude/skills/, cited verbatim. Its provenance is recorded in docs/skill-sources.md.

Contributing

• Every PR bumps the version (version-bump skill). CI's version-check job blocks the merge otherwise.
• CI runs pytest with coverage and SonarCloud, black, isort, flake8, bandit, markdownlint, the secret scan, the agent-first rubric gate, and the per-harness smoke check.
• A merge to main that touches pyproject.toml or heat_governor/ publishes to PyPI via Trusted Publishing. Such PRs publish a dev build to TestPyPI.

The gate section

culture.yaml carries a top-level gate section that declares how to set up and test this repo, so tooling such as a PR-fixer needs no per-repo knowledge:

gate:
  setup:
  - [uv, sync]
  test:
  - [uv, run, pytest, -n, auto]

setup and test are each a list of commands. Each command is an argv list (never a shell string), run in order from the repo root. Culture's loader ignores unknown top-level keys, so the section is safe alongside agents.

License

Apache 2.0 — see LICENSE.