ci-fleet

Fleet management for self-hosted GitHub Actions runners: allocation policy, supervised launch, and execution-side concurrency control.

Extracted from charlie-work so that runner development and orchestrator development stop colliding. The package is ci_fleet; the repository is ci_runners (directory name ≠ package name is the local convention here).

Status

Mid-migration. See PLAN.md for the phased extraction plan and ci-fleet-design.md for the target design. HANDOFF.md carries the current position: what is done, what is blocked on what, and the constraints that are not obvious from the plan.

As of 2026-08-04: suite at 823 passing (measured at dd9257d, the commit that landed the retirement below along with its companion test updates -- collection is clean and uv run --active pytest -q --tb=short is green; 869 was a d891906 reading one commit earlier, before that landing). The extraction itself is written and the section 6.3 cutover gate is implemented and self-verifying; the live repoint of charlie-work's consumers merged as charlie-work#869 (2026-08-01). The legacy planner is retired. Phase 6 (the flip) executed 2026-08-04, and on evidence from a deliberately induced park/restore cycle plus explicit owner instruction, the legacy planner itself was retired early -- 2026-08-04/05, about a month ahead of the originally planned 2026-09-04 retention floor (see HANDOFF.md section 4.11). ci_fleet.planner.plan is now the only planner; there is no actuating_planner value left to flip, and the field rejects anything but "new". Shadow mode ended with the legacy planner -- see "Shadow mode" below for what that cost. Phase 3 is not done (HANDOFF.md section 4.3). Phase 7's coverage-porting precondition was discharged 2026-08-03 (HANDOFF.md section 4.1), but Phase 7 itself -- deleting charlie-work's own dormant copies of the legacy modules -- has not landed as of this writing.

During the migration charlie_work imports this package through ci_fleet.charlie_work_adapter, which holds the legacy symbol names and signatures stable so consumer import lines change only in module path.

Layout

src/ci_fleet/
    _vendor/                 copies of specific charlie_work functions/classes
    charlie_work_adapter.py  legacy import surface (boundary only)
tests/
    fixtures/                frozen pre-move baselines

Invariant: nothing under ci_fleet may import charlie_work. Vendoring exists so the dependency arrow points one way.

Development

uv sync --extra dev
git config core.hooksPath .githooks   # once per clone -- see below
uv run --active pytest -q --tb=short
uvx ruff check .
uvx ruff format

That core.hooksPath line arms .githooks/pre-push, which runs the three commands above against your branch before a push leaves the machine. It is required once per clone: core.hooksPath is local config, so cloning this repo does not arm it, and an unarmed clone pushes with nothing checked.

That matters more here than in a repo that gates its own main, because charlie-work's CI installs ci_fleet from this repository's main as an editable path dependency tracking the branch rather than a pinned SHA. An unlinted push here surfaces as a red required check on somebody else's unrelated PR. Issue #1 has the full reasoning and why real CI is not yet the answer; issue #35 is why the hook was inert in worktrees until recently.

Developed and linted against Python 3.12+. The package itself declares requires-python = ">=3.11" — the oldest interpreter the suite is verified to pass on, and the floor charlie-work needs. See the note in pyproject.toml.

Shadow mode -- retired 2026-08-04

Through Phase 2, the rewritten planner (ci_fleet.planner.plan) ran alongside the legacy one (ci_fleet.runner_allocation.plan_allocation) on every allocation pass, and its output was compared and journaled -- never applied. That arrangement is gone. plan_allocation and its policy core (allocate_slots) were deleted along with the two modules that ran and recorded the comparison (ci_fleet/shadow_pass.py, the pass wiring, and ci_fleet/shadow.py, the ShadowRunner purity guard). The conditional that used to choose which planner actuated was removed entirely rather than collapsed to its surviving branch -- see the comment at the old flip point in runner_allocation_pass.py for why a defaulted-but-dead branch was judged riskier than no branch at all. RunnerAllocationConfig.actuating_planner now defaults to "new" and its __post_init__ rejects anything else, with an explicit retirement message for anyone who types "legacy": that value isn't silently accepted-and-ignored, because someone setting it is attempting a rollback, and a config file that disagrees with the running fleet should fail loudly, not quietly do nothing.

This is not pure cleanup -- retiring the legacy planner ended the shadow comparison, and that is a real cost worth recording rather than a side effect. The legacy planner was the shadow's counterparty; once it was gone there was nothing left for the new planner's output to be compared against. The diff journal (shadow-planner-diff.jsonl, under the fleet directory -- %LOCALAPPDATA%\charlie-work\ on Windows) stops growing: HANDOFF.md section 4.11 records 1051 records at the retirement, and the file settles once the running supervisor respawns onto code that no longer calls the comparison -- per-consumer respawn latency (see "Config propagation" or the note on actuating_planner in config.py) means a few more passes can land after the code changes on disk and before the supervisor picks them up, so a live read may still show a handful more than the retirement figure until the next respawn. Either way, any shadow-status-style tooling built against this file loses its input once it stops. The instrument died with the thing it measured. That is expected, not a regression to chase -- the comparison's job was to validate the cutover, and the cutover is done.

ci_fleet/shadow_gate.py and the frozen journal are both deliberately kept, even though nothing appends to the journal any more. The journal is the evidence artifact the cutover decision rests on, and the gate can still be re-run against it to re-derive that verdict from data rather than from anyone's summary of the data:

uv run --active python -m ci_fleet.shadow_gate <path-to-shadow-planner-diff.jsonl>

PLAN.md section 6.3 defines what the gate checks: a trailing agreement streak, per-decision-class coverage, and every disagreement adjudicated in writing. "Coverage" is not five-for-five against live passes -- of the five decision classes, only demotion is still required live; the other four are credited as discharged by evidence or adjudicated by replay (see shadow_gate.py's DISCHARGED_BY_EVIDENCE and ADJUDICATED_BY_REPLAY), which is how the gate can still be satisfied against a journal that stopped growing. Two config fields on runner_allocation -- shadow_planner and shadow_timeout_s -- also still exist and are, likewise deliberately, now inert: nothing reads either one, a stale shadow_planner: true left in a config file enables nothing and is accepted without complaint (there is no rollback it could be attempting), and both are scheduled for removal once the live config.yaml drops the lines.

The legacy planner's own local copies inside charlie-work (runner_allocation.py, runner_allocation_pass.py) are a separate matter -- unimported there already, and their removal is PLAN.md Phase 7, gated on its own coverage precondition rather than on this retirement. Do not conflate the two: this section is about ci_fleet's own plan_allocation, which no longer exists anywhere.

Measurement layer (Phase 4)

Three primitives from §5.3 of PLAN.md, plus the classifier that reads them.

Three named durations (ci_fleet.durations) — queue_wait, execution, and wall, recorded side by side and never added together. None is derived from another, and the type is what enforces that: the fields take an Interval, which carries two timestamps and the clock that produced them, so there is no constructor that accepts a bare number. The design anticipates three clocks (GitHub, the runner, this fleet), which do not reconcile exactly — and the residual, reconciliation_gap_s, is time the job spent somewhere nobody accounts for. Today's producer uses only GitHub's clock, so single_clock is True and that residual is structurally zero rather than informative. That is what single_clock is for: it marks the degenerate case so a zero gap is read as a tautology instead of as the clocks agreeing. The fleet-clock wall is the upgrade path and is deliberately not taken yet — see ci_fleet.job_observation.

Heartbeats (ci_fleet.heartbeat) — a beat carries a progress counter, not just a timestamp, because a reporter thread happily emitting timestamps over a deadlocked worker is the exact failure being hunted. A counter that goes backwards means broken data, not a stall, so it reports UnknownProgress permanently rather than becoming a kill verdict.

A fixed-op-count canary (ci_fleet.canary) — constant work, so elapsed time measures only host contention. The baseline is the fastest run ever observed on this host, persisted and self-calibrating, rather than a hardcoded constant that would be wrong on any other machine. Each pass reports the minimum of five repetitions (~2 ms each); the amplification factor is elapsed / fastest_ever. Baselines are keyed on workload shape, so changing the op count discards them instead of producing a meaningless ratio.

The 2×2 (ci_fleet.health) — starved vs. wedged, never a duration threshold:

The heartbeat axis decides the response; the canary axis explains it. A starved process still ticks, because a heartbeat costs almost no CPU — so a frozen counter means wedged however loaded the host is. Either axis can come back unknown, and that is a fifth state carrying no response: a job seconds old, with one beat and nothing to compare it against, must not be killed for being new.

A producer (ci_fleet.job_observation) — observe_job maps one GitHub job dict to a Heartbeat and a JobDurations. It is a pure function over the dicts measure_repo_demand already fetches on every pass, so it adds zero API calls; a separate poller would have roughly doubled request volume against a 5000/hour limit and sampled a different instant than the demand measurement, making any disagreement between the two unresolvable.

An earlier version of this section claimed the heartbeat gap was structural — that it needed the jobs themselves to report progress, hence workflow-YAML changes in the consumer repos. That was wrong, and measuring it is what showed it. GET /repos/{owner}/{repo}/actions/runs/{id}/jobs populates each job's steps[] incrementally while the job runs: a live job here read status=in_progress total=7 done=4 mid-flight. The count of completed steps is a progress counter the fleet can poll with no job cooperation at all. Step numbers are unusable for it — a completed job reported 1,2,3,4,5,9,10,11, so max(number) jumps by four at the end.

Three things the measurement settled, each now a test:

• The placeholder gate is started_at == created_at, not status. A job cancelled while still queued reaches status=completed with the placeholder intact (created == started == 07:04:10Z, completed=07:22:03Z, steps=[]). A status-based gate records that job — which waited 18 minutes and ran nothing — as queue_wait=0 and execution=18m: the D-06 zero-default and a fabricated wedged-looking job. Across 51 completed jobs the correlation had no counterexample: all 20 EQUAL pairs were cancelled with zero steps; all 31 jobs that genuinely ran reported the two different.
• wall is still measured for such a job, because created → completed is a real span however the job ended, and "had a lifetime but no execution" is exactly the shape worth being able to see.
• Poll granularity bounds what Frozen can mean. The pass runs every five minutes and a single step legitimately runs far longer: a live job here held done=4 across 20 consecutive polls (200s), and a real Tests job ran 14m25s over 7 steps. Two equal counters is therefore normal, not wedged, so the current Frozen rule would fire on healthy jobs. The stall tolerance must be measured before anything actuates on it — deliberately not added as a default-off parameter, since its only correct value comes from data this producer is what generates. See the open item in PLAN.md §9.

Wired into the pass, and recorded rather than classified. measure_repo_demand returns its observations on a second return channel, deliberately not folded into RepoDemand — that value is planner and shadow input, and widening it would put a measurement into a comparison that is supposed to be about allocation decisions. The pass batches them into one fleet_job_observations event per pass, because the jobs in a pass share an observed_at and that grouping is what makes consecutive beats comparable.

The 2×2 still does not run on live jobs, and that is the stopping point rather than an omission: the Frozen rule needs the stall tolerance below, and logging verdicts already known to be wrong would poison the very data that sets it. A default-off min_stall_s parameter was considered and rejected — it would ship the broken behaviour as the default while looking addressed.

Note the observation filter is not the demand filter, and the two must not be merged: a job that finished while its run is still in flight contributes no demand, yet it is the only place all three durations exist at once.

Independently of all that: one number would have hidden the whole story. On the same run, Lint spent 36m09s queued and 35s executing.

timeout-minutes semantics remain an open question by design (§5.3): recording all three durations separately is correct under either reading, so it blocks nothing, and the plan is explicit that it be measured rather than resolved from a search result.

Suite concurrency chokepoint (Phase 5, Stage A)

Stage A measures. It does not enforce. There is no lease, no counter, and no blocking call anywhere in ci_fleet.pytest_plugin — only a latch. Phase 5's acceptance criterion 3 wants the coverage share established before anything clamps, and that ordering is not politeness: once a lease is narrowing some runs and not others, "what fraction did we clamp" can no longer be answered by the thing doing the clamping.

The chokepoint is a packaging claim, not a code claim. A shim that never reaches a venv's site-packages covers nothing, however correct its body. ci_fleet.sitecustomize_install is where that claim is cashed.

The measurement inverted the design. sitecustomize cannot identify a pytest run from sys.argv at site-import time: under python -m pytest the module name is not in argv yet (it reads ["-m", "--version"]), and an IDE runner calling pytest.main() presents its own script path. Two of the four invocation modes are invisible to an argv predicate. The pytest11 entry point has no such problem, because pytest loads it however pytest itself was started. So the entry point is the chokepoint and the shim's job is the opposite of what it looks like — it exists to catch the runs the plugin missed.

Numerator and denominator fail independently, which is the only reason the share means anything (R-20):

A run that bypasses the plugin — -p no:..., PYTEST_DISABLE_PLUGIN_AUTOLOAD — still trips the exit probe and lands in the log as a bypass. An instrument counting only its own hooks would report "100% of observed runs were observed."

The "fails when" column is deliberately not exhaustive about rows, only about mechanisms. A venv carrying both the packaged .pth and an older hand-installed shim fires the probe twice, and the second write is dropped by a latch in record_process_exit. That is the invariant working rather than a failure, so it is not a row in the table — but anyone computing a share from these numbers should know the denominator counts interpreters, not probe registrations.

The independence is narrower than that table suggests, and the correction is measured. A third case used to be listed here — "or a venv without the package" — as something that would still log a bypass. It does not. A scratch venv carrying the real shim with no importable ci_fleet ran pytest and produced zero rows; the same venv with ci_fleet on PYTHONPATH produced one row with plugin_saw_it=False. The shim's from ci_fleet.suite_coverage import ... sits inside its except BaseException guard, so with no package there is nothing to import and nothing to write. Both halves share one dependency — the package must be importable — so what they actually distinguish is the plugin failing to engage inside an instrumented venv, which is the bypass that happens in practice. An uninstrumented venv is invisible, not a bypass.

Both halves now ship with the package, and that is a recent change. They used to install by different mechanisms: the plugin arrived with a dev extra (it is a pytest11 entry point in this package's metadata, so pip/uv wire it up), while the probe was written into one specific venv's site-packages by an explicit install() call that nothing invoked automatically. The consequence was measured and was the opposite of intuitive — a venv with the package and no probe logged nothing at all, not a bypass, because record_process_exit had exactly one production caller and that caller was the shim. Adding ci_fleet to a repo's dev extras therefore made its runs pass through the chokepoint without making them visible.

The probe is now shipped as a .pth at the root of site-packages (the mechanism coverage.py uses for subprocess support), so installing the package delivers both halves. This was chosen over installing the shim per environment for one reason: coverage that depends on someone remembering a manual step decays, and across ~87 churning worktrees it decays to nothing, whereas coverage that derives from installed state is inherited by every new environment on the next sync.

One row per process is enforced at the write, and that took two attempts. A venv can carry both the packaged .pth and an older hand-installed shim, and then both probes fire. The first fix latched registration through a sentinel on sys; it passed in a scratch venv and then failed in this repo's own, which carries a shim written before the sentinel existed — an old shim registers unconditionally and cannot be taught to check something that postdates it. Two rows landed 0.5 ms apart under one pid. Since shims already on disk elsewhere can never be retroactively changed, the invariant is enforced where the row is written instead: at most one row per interpreter, however many probes fire. A duplicate is not a harmless repeat — it inflates sessions with a run that never happened, which is a false denominator in the one number this instrument exists to publish.

The instrument is validated. The share is not yet established — and those are different claims. A scratch harness started 5 pytest processes across three invocation modes (python -m pytest, the console script, an IDE-style pytest.main()) and the log came back 4 covered, 1 bypass; the bypass was a pytest --version run, which returns from pytest_cmdline_main before a session is ever configured. That is a real result about the instrument: it distinguishes covered from bypassed, and the bypass is self-explaining because each row carries its full argv — classifying early-exit invocations with a hardcoded flag list would rot the first time pytest adds one.

It is not a coverage share, and must not be quoted as one. Every row in it was a process this repo's own harness launched, within eight seconds, in a scratch fleet dir, with the modes and the count chosen by the person measuring. A ratio over a self-selected sample describes the sample. The real log (%LOCALAPPDATA%\charlie-work\suite-coverage.jsonl) held 2 sessions, both covered when this was first written and grows with ordinary use; as of 2026-07-31 it holds 23 rows across 21 distinct pids. They are still the same pytest -q --tb=short invocation from this one repo, which is a denominator far too narrow to gate enforcement on — the count going up does not widen it.

Two of those rows are known-bad and are left in place rather than edited out: they are the double-count described above, written before the one-row-per-process invariant was enforced. Deleting rows from a measurement log to make a number look right is the failure this whole section is about, so the contamination is recorded instead — any share computed over this log before it is rotated is over a denominator inflated by two phantom sessions (~9%).

No data is not zero. CoverageShare.share returns None for an empty log, never 0.0. An empty log is a claim about the instrument, not about the chokepoint, and collapsing it would report total ineffectiveness on the strength of no evidence. Uninstrumented venvs contribute no rows at all, so the number is "coverage across instrumented environments" and nothing wider.

Cost and blast radius. The shim imports nothing at startup — it registers one atexit closure and returns, deferring the ci_fleet import to exit and only in processes where pytest ran. Measured at 416 µs cumulative / 384 µs self per interpreter start (python -X importtime). It swallows BaseException everywhere (R-19, availability, fails open): an instrument that turns a passing suite into a failing one at shutdown would be blamed on the suite. install() refuses to overwrite a sitecustomize.py it did not write, since only one can load per path entry and clobbering a foreign one silently disables whatever it was doing.

The plugin's blast radius is its import, not its hook. That half was measured only after the shim's, and it was worse: pytest_configure carried except BaseException and a docstring promising it fails open, but pytest imports an entry-point module before any of its hooks exist, so a module-level from .suite_coverage import … could raise a plugin-load error that no try in the file can reach — in every venv the package is installed into, not just this one, and the consumer would reasonably blame their own suite. The import is now deferred into the guarded hook body, leaving the module's startup surface as typing alone: importing ci_fleet.pytest_plugin in a fresh interpreter leaves ci_fleet.suite_coverage absent from sys.modules, present only once the hook runs. Two tests pin it — one observing sys.modules from a subprocess (rather than reading the source, since a transitive import three modules away would pass a text check and still break the venv), and a positive control that the hook really does fail open when that deferred import is broken. The general form is worth stating, because Stage B will add imports here: "this function cannot raise" is a claim about a function, and an entry point's real risk surface is its import.

Status: criterion 4 closed; 1, 2, 3, and 5 open — and 3 turns out to depend on 5. Criterion 4 asks that every invocation mode reach the chokepoint, and a harness that deliberately exercises each mode is the right way to answer it. Criterion 3 asks for the coverage share, which the same harness cannot answer for exactly the reason above: the share only becomes meaningful once the denominator contains runs nobody chose. That needs criterion 5 — the plugin added to the dev extras of each participating repo individually, worktree venvs inheriting only after a sync — so 3 is blocked on 5, and local_suite_lock is not retired until that list covers every repo the incumbent guarded. Criteria 1 and 2 are Stage B: atomic reservation (not observe-then-narrow), clamp-not-block (never return 0, never block past a deadline), and the honest bound max(B, R × min_units).

Per the phase spec, and worth stating plainly: this does not fix the CI variance. It bounds CI's contribution to it — and Stage A does not yet even do that, because it does not enforce.

Visibility

This repository is private and must stay that way while PLAN.md and ci-fleet-design.md are tracked. Between them they document:

• the host's runner topology and the repositories each runner is registered to
• a Windows service name and local filesystem paths under the owner's profile
• which paths are and are not on Defender's ExclusionPath
• a latent runner-exposure issue, with the condition that would trigger it

Every one of those is operationally load-bearing for this work and a map of the build host to anyone else. Redacting them would degrade the working documents; keeping the repo private costs nothing. Treat the list above as the checklist to clear before any visibility change — not as an example of the kind of thing to look for.

A prose note is a prompt-strength control. The architectural version is a workflow that fails when github.event.repository.private == false; worth adding if this repo ever grows CI of its own.