Fleet Watch

Local process governance for AI workloads on a single machine. It tracks ports, GPU memory, and repo leases so two jobs do not collide, and it can list GitHub repositories the local gh CLI already sees.

Install

pipx install fleet-watch

fleet --version

From source: pipx install ~/path/to/fleet-watch/

Example

fleet guard --json --port 8899 --repo ~/projects/my-app

If "allowed" is false, do not bind the port or write the repo.

What it refuses

fleet guard --json never silently allows on error: a missing registry, a locked database, an unreadable input, a negative GPU value, or a non-integer port all return "allowed": false. fleet sitrep talks to GitHub only through local gh; it does not clone, store tokens, or invent commit SHAs.

The Problem

You're running MLX, Ollama, vLLM, Candle/Cake, experiment runners, and AI coding agents on the same machine. They don't know about each other. Port 8899 gets stolen by a canary model. A 7B model quietly allocates 11 GB of Metal buffers on an 8 GB machine, swapping to SSD and running 65x slower than expected. Two Codex sessions write to the same repo. Health endpoints say "ok" while GPU memory is exhausted.

Fleet Watch prevents these collisions by maintaining a shared registry of what's running, what resources are claimed, and what's available — including pre-flight working set estimation that catches memory overcommit before it becomes a six-hour debug session.

How It Works

Fleet Watch auto-discovers running AI processes (MLX servers, Ollama, vLLM, Candle/Cake, etc.) by scanning lsof and ps. It registers them in a local SQLite database with their port claims, GPU memory estimates, and repo/session ownership. Any tool — human or AI — can call fleet guard --json before taking resource actions.

You don't register anything manually. Run fleet discover or let the launchd agent do it every 60 seconds.

GPU memory guard estimates total working set (weights + KV cache + activations + framework overhead) and compares it against physical RAM. A Candle-based 7B model with Q4_K_M quantization is estimated at about 7 GB under Fleet Watch's default estimator assumptions.

Quick Start

# See what's running
fleet status

# Auto-discover and register all AI processes
fleet discover

# Pre-flight: will this model fit?
fleet guard --gpu 4096 --framework candle --model "qwen2.5-7B-Q4_K_M" --json

# Check port and cooperative repo availability
fleet guard --port 8899 --repo ~/projects/my-app --json

# Check same-repo write intent before editing concrete paths
fleet guard --repo ~/projects/my-app --write-scope tools/playwright --json

# System health: memory pressure, sessions, GPU memory watch
fleet health

# See the audit trail
fleet history

# Generate state report
fleet report

Example: 7B model on 8 GB Apple Silicon

$ fleet guard --gpu 4096 --framework candle --model "qwen2.5-7B-Q4_K_M"
DENY
GPU 4096MB: working set 7049MB exceeds physical RAM (8192MB) minus reserve (6144MB available)
  Breakdown: weights 3337MB + kv_cache 1792MB + activations 64MB x 2.0x (candle)
  Physical RAM available after reserve: 6144MB
  Suggestion: Use q2_k quantization (~1668 MB weights, ~5380 MB working set)
GPU budget available: 113664MB (0MB allocated)

The same model on a 128 GB machine:

$ fleet guard --gpu 4096 --framework candle --model "qwen2.5-7B-Q4_K_M"
ALLOW
GPU 4096MB: available (113664MB free)
  Working set: 7049MB (weights 3337 + kv 1792 + act 64) x 2.0x
  Physical RAM available after reserve: 114688MB
GPU budget available: 113664MB (0MB allocated)

Always-On Mode (macOS)

fleet install-launchd

Fleet Watch will auto-discover processes and monitor GPU memory pressure every 60 seconds.

AI Session Integration

Add this to your AI tool's system prompt or config:

Before binding a port, starting a model server, or writing to a repo: run fleet guard --json with the relevant --port, --repo, --write-scope, --gpu, --framework, and --model flags. If "allowed": false, do not proceed. A plain --repo check is cooperative and reports active sessions; use --write-scope for concrete edit intent and --exclusive-repo-lock only for operations that require sole ownership. Use ~/.fleet-watch/state.json only as a fallback artifact when the CLI is unavailable.

JSON Contract

fleet guard --json

Top-level keys:

• allowed — boolean allow/deny decision
• request — what the caller asked to use
• checks — per-resource decision objects
• state — current machine summary

request contains:

• port — requested port or null
• repo_dir — absolute repo path or null
• write_scopes — absolute paths this command may edit
• exclusive_repo_lock — whether the caller requested sole repo ownership
• gpu_mb — requested GPU claim or null
• framework — inference framework hint or null
• model — model name/path hint or null

checks.port contains:

• allowed, reason, holder, suggested_ports

checks.repo contains:

• allowed, reason, holder
• holders — advisory cooperative session holders for allowed same-repo work
• overlap_paths — paths that caused a write-scope denial
• stale_holders — dead/stale session leases cleaned during the check
• safe_mode — suggested operating mode such as cooperative-write
• unblock_command when denied by a process or session holder Fleet can name directly. A runnable command only when one truthfully exists — fleet release --pid <pid> for a process holder, fleet session close --session-id <id> for a session whose owner is provably dead. For a LIVE session holder it is remedy TEXT instead, naming who may close the lease (its owner pid, a descendant, or the terminal that spawned it) and the cooperative alternative, because fleet session close is authorized to the lease's own lineage and would refuse anyone else

checks.gpu contains:

• allowed — boolean
• reason — human-readable or "working_set_exceeds_physical_ram"
• detail — expanded explanation when denied
• requested_mb — requested GPU claim
• available_mb — currently available budget
• suggested_max_mb — maximum claim that fits
• working_set — (present when framework/model provided) object with:
  • weights_mb, kv_cache_mb, activations_mb — component breakdown
  • overhead_multiplier — framework-specific pool overhead (e.g. 2.0x for Candle)
  • total_mb — estimated total working set
  • framework, model_size, quantization — detected parameters
  • physical_ram_mb, available_after_reserve_mb — machine context
  • fits — boolean: does the working set fit in available RAM?
  • grounded — boolean: were framework and model size detected from real input?
  • source — "explicit", "command", "fallback_default", or "insufficient_input"
  • suggestion — (when fits is false) actionable alternative

state contains:

• process_count, occupied_ports, safe_ports, locked_repos
• gpu_budget — total_mb, reserve_mb, allocated_mb, available_mb
• external_resources

fleet health --json

• memory — RAM snapshot: total_mb, pressure_pct, pageouts, swapins, etc.
• sessions — discovered CLI sessions with RSS, CPU, classification
• idle — workload processes at near-zero CPU
• gpu_memory_monitor — runtime pressure data: pageout_rate, gpu_process_footprints, alerts

~/.fleet-watch/state.json

Top-level keys:

• agent_interface, generated_utc
• processes, external_resources, process_count
• gpu_budget, ports_claimed, preferred_ports, safe_ports, repos_locked
• session_leases, process_classifications, stale_processes, recent_events
• conflicts_prevented_24h, system_memory, sessions, idle_processes
• gpu_memory_monitor — latest discovery-cycle snapshot of pageout rate, per-process footprints, and active alerts

Commands

Core

Command	What It Does
fleet status	Show active processes, GPU budget, claimed ports
fleet status --json	Machine-readable process and budget state
fleet guard --json	Canonical pre-flight contract for agents
fleet guard --gpu MB --framework FW --model MODEL	Working set estimation + allow/deny
fleet guard --repo PATH --write-scope RELPATH --json	Cooperative same-repo write-scope check
fleet check --port N --repo PATH --gpu MB	Honest availability probe (exit 0/1)
fleet discover	Scan and register running AI processes
fleet report	Write STATE_REPORT.md + state.json
fleet sitrep --json	Read-only GitHub fleet sitrep via gh (no clone, no tokens, no invented SHA)

Observability

Command	What It Does
fleet health	RAM pressure, sessions, idle processes, GPU memory watch
fleet health --json	Machine-readable health and GPU monitor snapshot
fleet changelog	Rolling state changelog
fleet changelog --json	Raw changelog entries
fleet history	Hash-chained event audit trail
fleet stale	List heartbeat-stale processes with evidence
fleet reconcile	Non-destructive ownership diagnosis
fleet reconcile --json	Machine-readable ownership diagnosis
fleet census	What boots and runs on this machine, and what's stale
fleet census --json	Full census record (fleet-census/v1)
fleet census --emit-launchd-plist	Print the staged daily-census launchd job; installs nothing
fleet boot-map	Census record → boot graph + local interactive 3D page

fleet census

A deterministic, read-only sweep of every persistence and runtime surface: ~/Library/LaunchAgents cross-referenced against launchctl list and launchctl print-disabled (both-direction orphan check), /Library daemons and agents, live processes clustered by what they actually run, TCP listeners attributed to their owning process, crontab, login items, brew services, and Fleet Watch's own registry.

Every item carries a status, a verdict (keep / investigate / close / remove), the exact evidence behind it, and the deterministic rule that fired. Records land at ~/.governance/receipts/fleet-census/, and each run diffs itself against the previous one — new, disappeared and verdict-changed boot entries are the signal.

Fleet Watch never kills anything here: close_command is advisory text for the user, and the recurring launchd job is staged (contrib/launchd/io.fleet-watch.census.plist), never installed.

Record contract: docs/fleet-census-receipt-contract-v1.md.

fleet sitrep

A read-only GitHub fleet sitrep. One GraphQL query through the local gh binary lists repositories owned by the authenticated viewer (or --owner) and copies the default-branch object id GitHub returned.

# Viewer-owned repos, most recently pushed first
fleet sitrep --json

# A specific user or org GitHub still lets this `gh` login see
fleet sitrep --owner cjchanh --limit 30 --json

Rules the command is tested against:

• No clone. sitrep may only invoke gh api graphql. There is no git clone / gh repo clone path.
• No tokens in this repo. The code does not read GITHUB_TOKEN / GH_TOKEN, does not run gh auth token, and refuses -t / --token. Auth stays in the user's gh credential store. If gh is missing or the query fails, sitrep prints REFUSAL and exits 1 — it does not invent a fleet from local checkouts.
• No invented SHA. sha is a 40- or 64-char hex object id from GitHub's defaultBranchRef.target.oid. Empty repos, missing oids, and abbreviated values become sha: null plus sha_absent_reason. Short SHAs are never kept or padded.

Records (when not --no-receipt) land at ~/.governance/receipts/fleet-github-sitrep/ as a dated JSON file plus latest.json. Schema: fleet-github-sitrep/v1. At most --limit repos (default 30, max 100); truncated: true means GitHub said another page exists.

fleet sitrep is the only Fleet Watch subcommand that talks to GitHub. fleet guard, discover, census, and health stay local.

Session Lifecycle

Command	What It Does
fleet session start	Open or refresh a session lease
fleet session heartbeat	Refresh session lease heartbeat
fleet session ensure	Idempotent session management with retry
fleet session close	Close a session lease (owner, its descendants, or its ancestor shell only; anyone may reap a provably-dead owner). Also clears the Claude single-writer twin state file. No --force.
fleet share-repo PATH	Close active session leases for a ~/Documents/* editorial handoff

These five commands answer as soon as the registry write is committed. The observability report is refreshed AFTER the acknowledgement, under a wall-clock budget (FLEET_REPORT_BUDGET_S, default 2s) and coalesced across concurrent sessions (FLEET_REPORT_MIN_INTERVAL_S, default 10s; 0 disables). A caller that nonetheless times out must READ THE REGISTRY BACK — fleet session list --json — before concluding the claim failed: a timeout is evidence about the courier, not about the lease. See session start's docstring for the exact match conditions and for what a retry does to fencing_epoch. fleet discover remains the unbounded publisher of STATE_REPORT.md / state.json, so a skipped refresh here delays the report, never loses it.

A COOPERATIVE lease whose owner is alive but whose heartbeat is older than 180s lapses for arbitration: it stops blocking and is reported under stale_holders with reason: "cooperative_lease_idle_lapsed". It is not closed — the owner's next heartbeat revives it. EXCLUSIVE leases do not lapse.

Process Management

Command	What It Does
fleet register	Manually register a process
fleet heartbeat --pid N	Refresh heartbeat for a registered process
fleet release --pid N	Release all claims for a PID
fleet reap	Dry-run: show orphan-confirmed processes
fleet reap --confirm	Kill and release orphan-confirmed processes
fleet reap-sessions	Kill detached hot sessions (dry-run by default)
fleet runaway	Detect runaway high-CPU processes
fleet runaway --kill	Kill flagged runaway processes
fleet preempt	Take a port from a lower-priority holder
fleet clean	Remove entries for dead PIDs
fleet install-launchd	Install/update a launchd agent
fleet watch	Continuous discovery loop (foreground)

Thunder (Remote GPU)

Command	What It Does
fleet thunder sync	Ingest Thunder instances into Fleet Watch
fleet thunder claim	Attach ownership to a Thunder instance
fleet thunder heartbeat	Refresh Thunder resource heartbeat
fleet thunder release	Remove a Thunder instance

Boot Map

One question: what boots on this machine, and where does it go?

fleet boot-map turns a fleet-census/v1 receipt into a node/link graph and one self-contained HTML page you rotate, zoom, and click through locally.

# Default: the latest census receipt -> ~/.governance/graph/boot-map/
fleet boot-map

# Any receipt, any output directory
fleet boot-map --receipt ./census.json --out /tmp/boot-map
open /tmp/boot-map/index.html

What lands in the graph. Launchd jobs, the scripts and binaries they launch, the repos and system areas those live in, the ports they bind, live processes, and the domains that group them. Verdict and status ride along as node attributes, so colour is the census verdict — keep, investigate, close, remove. Filled nodes were rated by the census; hollow nodes inherit the worst verdict of what reaches them (a repo reads red when something red lives in it), and say so in the detail panel. A red ring means a port bound to all interfaces.

Outputs (graph.json, index.html, receipt.json) are regenerable, so they live outside the repo. Same receipt in, byte-identical graph and page out — no wall-clock, no randomness, no layout drift.

Local and sovereign. Zero network at build time and at view time: no CDN, no fonts, no images, no XHR, no libraries. The page ships a default-src 'none'; connect-src 'none' CSP, and the test suite asserts the artifact contains no http(s):// literal at all.

Fail-closed. Exit 3 and a REFUSAL: line when the receipt is absent, unreadable, unparseable, structurally invalid, or degenerate. There is no bundled-fixture fallback: rendering a stand-in as if it were this machine is a fake-green, which is worse than no map. Zero domains or zero items is a refusal, not a clean bill of health. Items missing a status or verdict are never dropped — they are kept, coerced toward investigate (a human should look), and flagged in warnings; off-enum values survive verbatim with a *_valid: false marker so the graph cannot launder bad input into good-looking input.

The page refuses too: an empty or corrupted data island paints a REFUSAL panel, never a clean-looking empty canvas. That behaviour is covered, not assumed — scripts/verify_boot_map_render.js executes the emitted JS against a DOM stub and counts canvas draw calls, so a page that loads clean and paints nothing fails:

node scripts/verify_boot_map_render.js ~/.governance/graph/boot-map/index.html --expect render

GPU Working Set Estimation

Fleet Watch estimates total GPU working set per framework:

Framework	Overhead Multiplier	Why
Candle/Cake	2.0x	Retains intermediate buffers until command buffer completion
vLLM	1.4x	Paged attention overhead
MLX	1.3x	Aggressive buffer reuse
Ollama/llama.cpp	1.1x	Tight memory management

Working set = weights + (KV cache + activations) x overhead multiplier

Override multipliers via ~/.fleet-watch/config.json:

{
  "gpu_estimator": {
    "framework_overhead": {
      "candle": 2.5
    }
  }
}

Configuration

Fleet Watch writes a default config on first run at ~/.fleet-watch/config.json.

{
  "gpu_total_mb": 131072,
  "gpu_reserve_mb": 16384,
  "preferred_ports": [8000, 8001, 8080, 8100, 8888, 8899, 11434],
  "patterns": [
    {
      "name_template": "My Server",
      "process_match": "my_server.*serve",
      "workstream": "my-project",
      "priority": 3,
      "gpu_mb_default": 4096
    }
  ],
  "session_patterns": [
    {"name": "Claude Code", "kind": "claude-code", "process_match": "/claude\\b.*--"},
    {"name": "Codex", "kind": "codex", "process_match": "/codex\\b"}
  ],
  "idle_patterns": ["reranker", "socat.*TCP-LISTEN", "mlx_lm.*server"],
  "idle_cpu_threshold": 1.0,
  "pressure_thresholds": {"elevated": 70, "critical": 85}
}

Event Audit Trail

Every registration, release, conflict, and cleanup is logged with a SHA-256 hash chain. Verify integrity:

from fleet_watch import events, registry
conn = registry.connect()
valid, count = events.verify_chain(conn)
print(f"Chain valid: {valid}, events: {count}")

Ownership Model

Fleet Watch uses session leases to track who owns what. Process classification requires three independent signals before marking a process as safe to reap:

Repo session leases are cooperative by default. fleet guard --repo PATH --json reports active sessions but does not deny solely because another session is in the same repo. Add --write-scope RELPATH when a command may edit files; Fleet denies only when another cooperative session has an overlapping declared scope. Use --exclusive-repo-lock for destructive git operations, whole-repo rewrites, or other work that truly requires sole repo ownership. An ACTIVE exclusive lease also denies when the requested resolved path equals, contains, or is inside the holder's stored repo_dir; cooperative leases stay exact-path.

--gpu rejects negative values. With --json, that is {"allowed": false} rather than Click usage text.

1. Heartbeat expired — not seen by discovery in >180 seconds
2. Session lease missing or closed — no active owner
3. Parent chain detached — parent PID is dead or PID 1

All three must be true for orphan_confirmed. Use fleet reconcile to inspect, fleet reap --confirm to act.

States: live > disconnected > stale_candidate > orphan_confirmed > exited

Design Principles

1. Advisory, not mandatory. If Fleet Watch crashes, all processes continue normally.
2. One honest verb per action. guard decides, check probes, discover observes.
3. Single machine. No distributed consensus. SQLite is sufficient.
4. Observe first. Default to alerting, not killing.

Platform Support

Platform	Auto-discovery	GPU estimation	Manual registration
macOS (Apple Silicon)	Full	Full	Full
macOS (Intel)	Full	Partial	Full
Linux	Process matching + listener probes	Full	Full

Linux memory-pressure admission uses PSI (/proc/pressure/memory, kernel 4.20+ with PSI enabled) when present: some avg10 below 10% is normal, 10% to below 40% is warning, and 40% or above is critical; when the kernel has no PSI it falls back to /proc/meminfo MemAvailable. Without readable PSI or MemAvailable, the enforcing guard fails closed (refuses with an explanation) rather than admitting blind; the initial audit window remains advisory.

Auto-discovery uses ps for process matching. On Linux, listener-to-PID mapping uses ss when available, then falls back to netstat and lsof. Use fleet register for explicit registration on any platform.

Limitations

• Discovery is heuristic and pattern-based. Unknown workloads are invisible until registered.
• GPU working set estimation uses architecture tables and framework multipliers, not kernel-level Metal accounting.
• Auto-discovery is heuristic and uses ps plus platform listener probes such as ss, netstat, or lsof. Use fleet register when explicit claims are more reliable than discovery.
• Fleet Watch is advisory for human use. For AI agent sessions, a PreToolUse hook can make it fail-closed.
• Single-machine by design. No distributed coordination.
• fleet sitrep requires an authenticated gh on PATH. It reports at most --limit owned repositories GitHub returned for that login. It is not a clone, not an unauthenticated public scrape, and not a list of local working trees.

License

MIT