reliquary-fleet 1.1.0

Private-by-default operations dashboard for Reliquary miner fleets

Reliquary Fleet

Private-by-default operations dashboard for Reliquary miners. It combines live miner health, validator truth, sealed-window rewards, exact reward EMA, and chain/metagraph state in one responsive page at http://127.0.0.1:9091.

Reliquary Fleet uses read-only HTTP and archive probes, plus SSH only for miner hosts or optional validator diagnostics you explicitly configure. It does not require a wallet seed, coldkey, hotkey private key, hosted account, or telemetry connection.

Dashboard preview uses deterministic demo data.

What you get

• Fleet summary — live health, exact provisioned model/checkpoint parity, throughput, OOM-free streak, current window + state, per-environment math/code fill, and validator pool/proof admission rate.
• Submit-disabled accelerator labs — separate RTX/B200 host, GPU, source/checkpoint and aggregate offline-evidence or immutable selector-artifact telemetry. Lab rows have no wallet identity and are excluded from every live-miner, selection, accounting and readiness total.
• EMA leaderboard — replays the validator's exact EMA against cached R2 windows. Star any hotkey to mark it as "yours" and have it tinted + included in the "your share" totals.
• Live event tail — merged journal events from every miner box, color-coded per box.
• Validator-side events — direct docker logs reliquary-trainer tail over SSH. Worker-confirmed pool accept/reject plus seal/fail lifecycle events.
• Validator rundown — deployment fingerprint (image SHA + uptime), error counters (Window iteration failed / submission worker / window timeout / randomness retries), throughput, reject reason breakdown. Timeframe pills: 10m / 30m / 60m / all.
• Auction-v2 liveness — event-loop and endpoint latency percentiles, per-environment admission queue/workers/proof activity, prepare/commit/total latency, immutable seal-drain snapshots, and archive enqueue continuity.
• Chain / Baseline / Network RTT — stake + emission per UID, throughput drift vs 6h baseline, latency from each box to the validator.
• Last N windows — slot share per recent window + reject summary, with internal scroll.

Install

# Recommended: isolated application install.
pipx install reliquary-fleet

# Or install beside Reliquary in an existing virtual environment.
python -m pip install reliquary-fleet

# Create an owner-only starter config in your OS user config directory.
reliquary-fleet init

Edit the path printed by init, replace every example host and hotkey, then:

reliquary-fleet doctor
reliquary-fleet doctor --connect
reliquary-fleet serve --open

Upgrade without replacing config or cached history:

pipx upgrade reliquary-fleet
# or: python -m pip install --upgrade reliquary-fleet

Source checkout

git clone https://github.com/reliquadotai/reliquary-fleet.git
cd reliquary-fleet

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Configure the dashboard.
cp config.example.yaml config.yaml
$EDITOR config.yaml

# Prefer the validator's cache-enabled public archive origin when available.
export R2_PUBLIC_BASE_URL="https://archives.example.org"

# Operators of a private bucket may instead use scoped read-only S3 access.
export R2_ENDPOINT="https://<account>.r2.cloudflarestorage.com"
export R2_BUCKET="reliquary"
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."

./run.sh
# Open http://127.0.0.1:9091

Prerequisites

• Validator HTTP access to its read-only /state, /health, and per-hotkey /verdicts surfaces.
• SSH access to each configured miner or lab host. Passwordless key auth is required for rows under fleet: or labs:. Validator SSH is optional; when omitted, only the direct log-tail and deployment-fingerprint panels are disabled.
• Read-only lab access for every optional labs: host. A running lab service must expose RELIQUARY_SUBMIT_DISABLED_LAB=1 and RELIQUARY_LANE_START_ALLOWED=0; evidence databases are opened in SQLite read-only mode, while completed artifact-only jobs are read from an immutable checksummed manifest. Only aggregate counters and whitelisted artifact metadata leave the host.
• Sealed-window archive access for exact score and EMA history. A cache-enabled HTTPS custom domain is recommended. Private-bucket operators can instead provide a scoped read-only R2 key; exhaustive bucket listing is disabled unless explicitly opted in.
• Python 3.10-3.13 on Linux or macOS.

Configuration

Run reliquary-fleet init to create the default private config. An explicit --config path wins, followed by RELIQUARY_FLEET_CONFIG, a checkout-local ./config.yaml, and finally the per-user config. The full shape is documented inline in config.example.yaml. The minimum useful configuration is:

validator:
  url: "http://YOUR_VALIDATOR_IP:8080"
  # Optional operator-only diagnostics:
  ssh:
    host: ""
    key:  "~/.ssh/id_ed25519"

fleet:
  - alias:  "your-miner-1"        # ssh alias or user@host
    hotkey: "5Grwva..."           # SS58
    label:  "m1"
    color:  "cyan"

labs:
  - alias: "root@YOUR_OFFLINE_GPU"
    label: "offline-selector-lab"
    color: "yellow"
    selector_artifact_manifest: "/srv/reliquary-selector/artifacts/challenger/manifest.json"
    source_manifest: "/srv/reliquary-miner-pro/state/source-manifest.env"

r2:
  public_base_url: "https://archives.example.org"

For a private operator bucket, leave public_base_url blank and configure endpoint, bucket, access_key_id, and secret_access_key instead.

labs: is deliberately not an alternate fleet syntax. A lab entry rejects a hotkey, never enters the owned-key set, and cannot affect /healthz fleet readiness. The dashboard reports its service and evidence status separately so an idle or failed experiment cannot be mistaken for a stopped live miner—and an active experiment cannot be mistaken for production mining.

A lab may configure evidence_db, selector_artifact_manifest, or both. The selector manifest is opened read-only, must be a non-symlink regular file with no group/world write bits, and must pass its canonical SHA-256 checksum before the dashboard exposes its whitelisted identity, window range, row counts and held-out calibration metrics. shadow_only and rejected remain offline artifact states; neither can affect miner readiness or live accounting. Bounded selector-search reports additionally expose rolling GPU lift, positive value-fold coverage, the final deterministic CPU challenger and explicit activation blockers. A positive challenger remains visibly shadow_only until every blocker is cleared; the dashboard never infers eligibility from lift alone.

For a deliberate same-host multi-lane miner, list every possible service under unit + allowed_units, then put only the exact simultaneous set under coordinated_units. A single active allowed service remains valid. More than one is valid only when the active set exactly equals coordinated_units; an unexpected, transitional, or differently paired service set still fails closed as multiple_allowed_units_active.

dashboard.refresh_seconds is the idle delay after each completed probe, not the maximum duration of the SSH and validator work itself. /healthz uses the separate dashboard.health_poll_stale_seconds budget (60 seconds by default) so normal bounded probes do not flap readiness; set it just above the slowest expected complete probe cycle so a stuck poller still fails closed.

Shared validator and chain reads have separate lower-bounded cadences under upstream:. The defaults are intentionally safe for distributed installs: state every 15 seconds, health every 30 seconds, incremental verdicts every 60 seconds, at most eight watched verdict hotkeys, RTT every 120 seconds, and chain/metagraph every five minutes. Do not lower these by patching the package; configuration validation rejects unsafe values.

When a logical box can switch between the legacy reference service and the named Math/Code services, keep the legacy unit/env_file and explicitly allow each replacement with its lane-specific state path:

  - alias: "ubuntu@your-h100"
    hotkey: "5..."
    label: "h100-reserve1"
    color: "cyan"
    unit: "reliquary-miner-pro.service"
    env_file: "/srv/reliquary-miner-pro/state/miner-pro.env"
    allowed_units:
      - unit: "[e-mail]"
        env_file: "/srv/reliquary-miner-pro/state/miner-pro-math-reserve1.env"
      - unit: "[e-mail]"
        env_file: "/srv/reliquary-miner-pro/state/miner-pro-code-reserve1.env"

Exactly one of a row's primary/replacement units may be active. Separately configured rows on the same SSH host may run concurrently (for example Code shards code-reserve1 and code-reserve2): each row collects only its own service, while their union becomes the host-wide allowlist. Any active reliquary-miner-pro@*.service outside that union remains a fail-closed unit selection error. This preserves lane-specific environment, frontier, and quarantine checks without treating an intended sibling shard as unexpected.

For one fixed named lane, configure that service directly as unit, provide its exact env_file, and omit allowed_units. Remove obsolete sibling rows: every configured row is a required live fleet member, not a historical lane.

Code auction readiness

Reference opencodeinstruct lanes have an additional fail-closed readiness contract. The dashboard compares the selected service's live /proc values with its root-only EnvironmentFile and requires:

• the miner and source-pinned grader services to be persistently enabled;
• exact Code pre-screening plus RELIQUARY_CODE_AUCTION_POLICY=deadline_aware;
• a non-empty current systemd InvocationID and a code_auction_readiness startup attestation found only inside that invocation's journal;
• the root-owned grader socket, versioned bundle source, and successful grader canary counters to match the validator's public source revision;
• full private/public source revisions with PROVISIONED_OK=1, with the public revision equal to a fresh, error-free status=ok validator health sample; retained last-good health is never used; and
• a healthy v1 Code outcome ledger containing the validator's exact current model repository, checkpoint revision, checkpoint number, public source revision, runtime profile hash, and Code environment tuple.

The SQLite probe uses mode=ro and PRAGMA query_only=ON; it never creates or updates the ledger, scans the attempts table, or exports prompt, completion, wallet, or attempt identifiers. Ledger v1's explicit partitions registry records this six-field identity at startup and dynamic checkpoint changes, so an exact current empty partition is ready before its first natural attempt. The dashboard never guesses currentness from attempt timestamps. Math and legacy lanes do not inherit these Code-only readiness gates.

Newer schema-v1 ledgers may also contain the additive generation_outcomes table. When present, the dashboard shows only aggregate counts for the exact current six-field partition and active Code lane: natural-EOS completions, local token-limit terminations, and safe-deadline cancellations. The displayed natural-EOS rate is complete / (complete + local_token_limit); safe deadlines are right-censored by window position and never count as failures in that denominator. Ledgers created before this optional table remain ready and are shown as legacy telemetry. No prompt, completion, attempt key, reward, or wallet data is read or exported.

The grader service name, socket, bundle link, metrics port, and expected socket mode have non-secret defaults under code_readiness: in config.example.yaml. The startup attestation is invocation-scoped and must be emitted only after the active binary has installed the pre-screen/policy and verified the configured grader and ledger:

code_auction_readiness {"schema_version":1,"prescreen_enabled":true,"auction_policy":"deadline_aware","ledger_path":"/srv/reliquary-miner-pro/state/code-auction.sqlite3","ledger_available":true,"ledger_partition_registered":true,"ledger_schema_version":1,"grader_socket":"/tmp/reliquary-grader.sock","grader_source_revision":"<40-hex-public-source-revision>","process_pid":1234,"miner_source_revision":"<40-hex-private-source-revision>","public_source_revision":"<40-hex-public-source-revision>","runtime_profile_hash":"<64-hex-profile-hash>"}

Managing the fleet

Two ways to edit:

1. Edit the active config and restart — canonical source of truth. Best for the initial onboarding of a box. reliquary-fleet doctor prints the active path.
2. Live in the dashboard — click ★ on any EMA-leaderboard row to mark that hotkey as "yours." It is persisted in the private application state directory. Starred hotkeys appear in the "our share" totals and are tinted in the leaderboard.

The set of hotkeys treated as "yours" is the union of every hotkey: under fleet:, the persistent star state, and starred_hotkeys: in the config.

Daily operator loop

Use this order when you open the dashboard. It keeps the brain calm and prevents chasing the wrong hotkey.

1. Target key: in Fleet and Chain, confirm the row label and short SS58 prefix match the miner you care about.
2. Chain registration: check Chain - netuid 81 before reading performance. active target, not registered on netuid 81 means the dashboard is targeting the right key, but the key cannot yet be counted as a registered SN81 miner.
3. Live window: check Mission control or Window score for the current window, state, and separate math/code valid/target pressure. If an environment is already full and our pool accepts are zero, this is a timing/slot-race problem.
4. Per-box health: check Fleet for process alive, uptime, the atomic source-manifest model identity, GPU memory/utilization, disk, OOM age, and validator-confirmed pool/30m. The model cell compares the provisioned checkpoint (or explicitly reset clean base) with live validator state, unless PID/start-bound load or generation evidence attests a newer exact runtime checkpoint. pool/30m is /verdicts proof/pool admission, not selection or earnings. A live GPU with pool/30m = 0 means the box is running but not entering the validator pool.
5. Pipeline: check GPU/cache pipeline for ready/inflight/submitted. If these stay zero while GPU memory is loaded, inspect miner logs for local skip/reward filtering.
6. Validator liveness: use the auction-v2 ingress liveness block to separate queue pressure from expensive preparation. Compare Math/Code queue depth, workers, prepare p95/p99, commit-lock p99 and total p95/p99; then verify each seal drain is ok and its queue/workers/reservations snapshot is empty. Archive enqueue gaps and seal timeouts make /healthz degraded only when the validator explicitly reports them; an older schema with no fields remains compatible.
7. Validator proof: use direct /verdicts for proof/pool truth. Operators who configure validator SSH also get Validator-side events and the deployment fingerprint. Pool acceptance still does not imply earnings; use sealed archive rows for final batch selection and rewards_by_hotkey.
8. Recent windows: use Last N windows for the last 24-window result quality. Archive-v2 completed objects remain reward evidence. aborted tombstones retain numeric continuity and show their failure stage/type, but are excluded from slots, rewards, EMA and training interpretation.

When the dashboard and chain disagree, trust chain registration for ownership/stake, direct /verdicts only for proof/pool acceptance, and sealed R2 rows for final selection plus rewards_by_hotkey for reward. Miner-local logs are useful for debugging, but neither they nor /verdicts accepted=true prove earnings.

Scale and request boundaries

The browser talks only to its localhost Fleet process. A visible dashboard tab uses one consolidated snapshot request every five seconds, or 12 local requests per minute; hidden tabs stop polling. The logs view uses conditional requests, pauses while hidden, and returns 304 Not Modified without regenerating HTML when its filtered buffer has not changed.

For H watched hotkeys and B miner boxes, the default conservative shared validator envelope is:

6 + H + 0.5 × (B + 1) requests/minute/install

That is 12.5 requests/minute for four hotkeys on four boxes. The first verdict sync requests one hour; later syncs retain a local rolling cache and request only a two-minute overlap. Every collector starts with randomized delay, uses jittered recurrence, honors validator Retry-After, and exponentially backs off after failure. /healthz and /api/export.json expose the effective per-install request budget.

Archive cold starts fetch eight immutable objects per pass with at most two workers. Missing validator archive indexes serve the last-good cache and never trigger an automatic bucket scan. For broad distribution, publish archives through a Cloudflare-cached custom domain; R2 S3 API requests do not use the edge cache.

Fleet contains no central account or tenant service and sends no runtime data to reliqua.ai. Very large public rollouts still require normal validator capacity planning or a shared read-only relay; the package protections prevent accidental request storms but do not manufacture upstream capacity.

Architecture

fleet.py is the data layer: SSH probes for explicitly configured hosts, pooled validator HTTP reads, incremental verdict caching, bounded immutable archive retrieval, exact reward EMA replay, and optional structured validator log parsing.

fleet_web.py is the presentation layer. Background collectors keep immutable snapshots warm; one failure-isolated /api/dashboard-snapshot response updates all 18 panels while the existing /api/* HTML fragments remain backward compatible.

/api/export.json keeps the validator's full health_raw payload and also adds a stable normalized validator.liveness object. Consumers can read the normalized object across old and new validator builds: reported=false, empty maps and null counters mean “not reported,” never a measured zero. Archive-v2 lifecycle and per-environment seal-drain evidence are exported on each explicit-lifecycle window.

State is module-level singletons under a lock; no local database or Redis is required. Installed runs keep owner-only baseline, window cache, star state, and the duplicate-instance lock in the OS user state directory. Set RELIQUARY_FLEET_STATE_DIR or pass --state-dir to override it. Source checkouts retain the compatible ./state layout.

Operations

• Foreground service: reliquary-fleet serve; stop cleanly with Ctrl-C.
• Open automatically: reliquary-fleet serve --open.
• Custom paths: reliquary-fleet serve --config /path/config.yaml --state-dir /path/state.
• Remote access: prefer ssh -L 9091:127.0.0.1:9091 operator-host. A non-loopback bind is refused unless --allow-remote is supplied and must sit behind authenticated TLS.
• Logs: stdout/stderr are service-ready and do not include R2 credentials or signed URLs.
• Reset state: use the path printed by reliquary-fleet doctor. Removing starred.json clears manual stars; removing window_cache_v2.json.gz progressively rebuilds sealed-window history.

Troubleshooting

Symptom	Likely cause
R2 unavailable on every panel	r2.public_base_url is missing/unreachable, or private R2 credentials lack GetObject on the prefix
archive index unavailable; serving cache	validator /health does not currently expose archive_last_uploaded_window; Fleet deliberately avoids a shared bucket scan
Validator panel shows "unreachable"	validator.url wrong, validator container down, or :8080 blocked by host firewall
Validator event tail says HTTP-only mode	expected when validator.ssh.host is blank; /verdicts and sealed-window score remain active
Box stuck at "warming…"	passwordless SSH to that box not working — test with ssh <alias> 'true'
one environment's valid/target never moves	inspect /health proof reservations/rejects and miner parity/quarantine state; the aggregate /state count can hide a starved environment
port already in use on launch	another instance is still running — lsof -nP -iTCP:9091 -sTCP:LISTEN then kill <pid>

License

MIT. Bundled third-party notices are in THIRD_PARTY_NOTICES.md.