aptriever

The goal

Make apt update and apt full-upgrade finish as fast as possible.

Every decision here answers to that, and any of them can be reopened by a measurement showing apt finishing sooner the other way. Elegance, purity, and even correctness-in-the-abstract lose to wall-clock time at the client — with one exception: apt must never receive wrong bytes, because a package that fails verification costs more than every second this project has ever saved.

The rule cuts both ways, and the second way is easy to forget: work that does not make apt faster does not belong on the request path. Nothing may sit between a client's request and the first byte moving — not a lookup, not a measurement, not a bookkeeping step — unless it demonstrably earns its latency.

The goal holds at both ends of connectivity. At a multihomed site on 3 × 1 Gbit the problem is that no single mirror fills the pipe. At the other end — Engineering for Slow Internet, written from the South Pole: round trips of about 750 ms with jitter of seconds, a couple of kbps to 2 Mbps for the whole station, satellite windows of a few hours a day, and at worst 40 kbps with 10% packet loss and 15-second dropouts every few minutes — the local link is the bottleneck, and racing mirrors against each other only divides it. aptriever aims to be useful there too, which turns its rules around: the fewest bytes and round trips over the WAN beat the fastest mirror; a transfer whose bytes are still flowing is never aborted, however slow; progress already downloaded is never thrown away; and no timeout is fixed at a value a multi-second round trip cannot meet. Both ends are design targets: where the code falls short of the slow one, that is a bug, not an edge case.

The problem

apt install and apt full-upgrade are often slow because the configured mirror is (temporarily or permanently) slow.

The default apt source is http://archive.ubuntu.com/ubuntu. The usual first remedy for slowness is the country mirror, http://CC.archive.ubuntu.com/ubuntu with CC the country code, plus extra sources for packages outside the archive (PPAs and similar). That helps, but:

• CC.archive.ubuntu.com resolves to one mirror. When that mirror isn't the fastest reachable one, downloads are slower than they could be.
• Determining the actually-fastest mirror is not trivial: in practice, ad-hoc probe scripts fetch a test file from hundreds of mirrors, either serially (slow) or in parallel (results skewed by the probes contending with each other).
• The measured winner helps only temporarily — mirrors go down, slow down, fall behind — and re-measuring is manual, so stale winners linger in the config.
• A single mirror can be slower than the client's link; on a fast connection (e.g. 10 Gbit) one mirror often cannot fill the pipe.
• A laptop that changes country needs its CC corrected, or the mirror test repeated.

Putting several mirrors into the sources helps: apt spreads requests across them and fails over when one errors. But apt switches mirrors on errors, not on slowness, and each file downloads from one mirror at a time — a big .deb assigned to the slow mirror still drags the whole run. And a machine that moves between networks now has several mirror entries to re-pick instead of one.

deb mirror://mirrors.ubuntu.com/mirrors.txt (apt's mirror method) automates the selection with a per-country list, but keeps the per-file behaviour: a file assigned to a slow mirror just waits. It also broke do-release-upgrade (TODO: verify whether that is still the case). And bootstrap tools are worse off: debootstrap accepts a single mirror URL and does not understand mirror:// lists.

Finally, when several machines share a network, each one downloads the same kernel update or third-party package (SaltStack repositories, for example) separately, instead of the file arriving once and being shared.

The caching proxy built for that last problem, apt-cacher-ng, looked like the right tool: the maintainer ran one instance and pointed every host on the network at it in apt's config. Then it cached something it should not have and package installs stopped working on every host behind it, until the bad entry was dug out by hand. After the third or fourth time the whole setup was reverted and the instance deleted: worse than nothing. That is the void aptriever fills: the same one-URL-for-every-client shape, built so that a wrong byte can never reach apt. It borrows nothing from apt-cacher-ng beyond leaving its port 3142 free.

aptriever aims to solve all of these.

What it is

A metric-aware, parallel, self-tuning download accelerator that clients use as a plain http mirror: it downloads each file from several mirrors at once, adds connections when that increases speed, learns mirror speeds from live traffic, follows mirror redirects itself (so plain-http-only clients — debootstrap runs before ca-certificates are installed — still benefit from https-only mirrors), and caches immutable files (packages, by-hash/ metadata) on disk so a file downloads once and serves the whole network. One plain http URL is also all that single-mirror tools like debootstrap can take — aptriever fans it out for them. It will not beat a full local mirror; it aims to beat everything short of one.

Racing and event logging key on the endpoint, the (mirror, IP address, egress address) triple; throughput, bench and connection pacing key on the destination behind it — the (hostname, IP address, egress address) triple, with the port when the URL spells one — shared across archives, so a host in both the Ubuntu and the Debian pool is measured for throughput once per egress address and holds one connection at a time over each, while two hostnames sharing an address stay distinct. TTFB stays per endpoint, since an https endpoint pays a handshake an http one does not. Both identities carry the egress address a dial is bound to: at a site whose router routes each delegated IPv6 prefix to the provider that delegated it, the instance's own address picks the provider, so aptriever reads its global, stable, preferred IPv6 addresses on its default-route interface fresh at every seating, dials each IPv6 address once per egress address it holds, or once unbound when it holds none, and scores, benches and paces each pair on its own; a definitive failure benches the server over every egress address, since the server lied whichever way it was reached. IPv4 dials stay unbound: the router behind NAT owns that choice. Nothing configures it: which addresses exist is read from the system, which wins is measured. A mirror's IPv4 and IPv6 addresses dial simultaneously with no head start, first byte decides, as do one address's endpoints over two egress addresses — replacing aiohttp's Happy Eyeballs connect stagger — and DNS lookup time never enters the measured latency, since requests dial an already-resolved address directly.

Status

Working: origin mode, apt against the Ubuntu and Debian archives, racing across mirrors, and an on-disk cache of immutable paths with cross-mirror dedup. Not yet: self-tuning connection counts, proxy mode, apk. Vocabulary lives in CONTEXT.md, decisions in docs/adr/, the roadmap in the issue tracker.

LLM use

aptriever is written with LLM assistance, and the method is in the repository rather than in a claim: each non-trivial change starts as a spec and then an implementation plan under docs/plans/, the plan is executed test-first, and the maintainer reviews it before it merges. Constants carry their reasoning (ADR 0003) and measurements name where and when they were taken, so a reader can check the numbers instead of trusting them. Commits before 2026-07-29 also carry a per-commit Co-Authored-By trailer; this section replaces it.

Running it

Released on PyPI, so uvx aptriever runs the latest release with no checkout; uv tool install aptriever keeps the command around. From a checkout:

uv run aptriever        # zero config: loopback, an ubuntu pool seeded with archive.ubuntu.com and a debian pool seeded with deb.debian.org, each grown by its mirror lists, cache capped at 5% of the disk
uv run aptriever serve --config aptriever.toml   # or with a config; `cp aptriever.example.toml aptriever.toml` to start one
uv run aptriever check aptriever.toml            # validate a config without starting anything: exit 0, or 2 with the reason

Turn DNS negative caching back on

On Ubuntu, do this first — it is the single largest configuration win available to an aptriever host:

printf '[Resolve]\nCache=yes\n' | sudo tee /etc/systemd/resolved.conf.d/x-cache-negative.conf
sudo systemctl reload systemd-resolved

Ubuntu ships /usr/lib/systemd/resolved.conf.d/cache-no-negative.conf setting Cache=no-negative, which disables caching of negative DNS answers (LP #1668771) to accommodate split-horizon DNS. The cost lands on everyone else: a mirror with no AAAA record — a large share of any Ubuntu mirror list — re-queries upstream for that missing record on every lookup, forever, because "no such record" is exactly what never gets cached.

Measured on one laptop resolving through DNS-over-TLS (2026-07-21): mirrors.aliyun.com cost 33.7 ms per lookup before, 0.146 ms after — 200× — while a host resolving through a nearby router paid only ~1 ms, since the penalty is one upstream round trip and scales with resolver distance. Every name that has AAAA records was already fast, and the fix changes nothing for them.

Only apply it if you do not depend on split-horizon DNS, which is what the default exists for.

The cache lives under cache_dir (default /var/lib/aptriever/cache); set cache_dir = "" to disable it. Sizing is dynamic against a live statvfs: unconfigured, a localhost-only instance keeps the cache within 5% of the filesystem (cache_max_percent), while an externally reachable one fills the disk to an 80% high-water and then evicts oldest-first — even when other data caused the fill (disk_high_water_percent); cache_max_bytes remains as an absolute override. aptriever clean deletes every completed entry (they re-download on demand), and aptriever purge <archive> <path> removes just one, addressed by the archive and path fields of its events — refusing, with exit code 2, both a rejected ../absolute path segment and a target under the in-flight partial/ spool, which is never a cached entry, and exiting 2 the same way when it cannot stat or remove the entry it was given. aptriever clean exits 2 when it skipped anything it could not read or remove, a missing cache directory included, so a count of zero means the cache was empty and nothing else. A fill that runs the disk out of space fails that download rather than falling back.

Without any [archives.*] two pools are synthesized — ubuntu, seeded with archive.ubuntu.com, and debian, seeded with deb.debian.org — and each grows from its own mirror lists, lazily: for ubuntu Launchpad's archive-mirror feed (https://launchpad.net/ubuntu/+archivemirrors-rss, every registered mirror whatever the egress's GeoIP mapping — the membership source) and Canonical's GeoIP-selected list (http://mirrors.ubuntu.com/mirrors.txt, a cheap locality prior when the egress is mappable); for debian the mirror masterlist (https://mirror-master.debian.org/status/Mirrors.masterlist, every registered Debian archive mirror with its country — no security mirror list exists, so debian-security needs the embedded spelling, in the second stanza below). An idle daemon sends nothing to mirrors or list sources; each list refreshes at most daily and only when a download request finds its on-disk copy (launchpad-mirrors.txt, geo-mirrors.txt and debian-mirrors.txt under $STATE_DIRECTORY or ~/.cache/aptriever/) past its TTL; a failed refresh retries hourly with a list in hand and every minute while a pool is still its seed alone; hostname aliases of one physical mirror are collapsed by DNS canonical name. Arrivals are resolved as they are merged, so the next download's fan-out races them beside the ranked seats: a new mirror is used the moment it measures well, never after a timer. Configuring any [archives.*] section disables all of this — an explicit pool is never augmented. A public mirror the lists miss — your own unregistered one, say — joins its pool through an [additional_mirrors] table keyed by archive (ubuntu = ["http://mirror.example/ubuntu"], debian = [...]): membership only, so it is measured, seated, swept, demoted and benched like every listed mirror and, like them, refused any internal address (a mirror in private space belongs under [archives.*], and a non-empty table beside [archives.*] is a config error); its endpoints take the next download's exploration slots ahead of everything else in the exploration order, so it is measured by the first request after it appears rather than by the next sweep. Every outbound request — list refreshes, location lookups, probe sweeps and downloads alike — identifies itself as aptriever/<version> (+https://gitlab.com/paulto/aptriever) so a mirror operator can attribute the traffic and reach someone about it; a client's own User-Agent is never forwarded upstream, only recorded in the event stream.

On AWS the ubuntu pool also takes the EC2 mirror: the Ubuntu archive mirror Canonical names for the instance's region, http://<region>.ec2.archive.ubuntu.com/ubuntu — the one cloud-init writes into apt's sources on EC2 — which no list carries. The daemon finds it the way cloud-init does, with a few deliberate differences (src/aptriever/ec2.py names the cloud-init commit it follows, every rule it borrows and every divergence with its reason): /sys/class/dmi/id/sys_vendor starting with Amazon EC2, or /sys/hypervisor/uuid starting with ec2 in either byte order, letter case ignored, says the instance is on AWS — anywhere else nothing more happens — and the instance metadata service (IMDSv2, 169.254.169.254 and [fd00:ec2::254] raced) names the region. The mirror joins like an additional mirror, measured by the first fan-out after it joins, unless its name does not resolve or its CNAME chain ends at archive.ubuntu.com or ports.ubuntu.com. Some regions' names are CNAMEs to another region's mirror, which is kept — checked 2026-09-11: ap-east-2, cn-north-1 and cn-northwest-1 get ap-east-1's (Hong Kong), ap-southeast-7 (Thailand) gets ap-southeast-3's (Jakarta), ca-west-1 gets ca-central-1's, mx-central-1 gets us-east-2's — and a Local Zone gets the mirror of the region the metadata service names for it, its parent region (for the Bangkok Local Zone, Singapore). The lookup starts with the daemon and retries after a second, doubling to a minute, until it has a definite answer — an instance can boot before its interface has an address, and a metadata service blocked by mistake can be unblocked later — which makes it the one lookup an idle daemon repeats: it goes to the instance's own metadata service and to the resolver for the mirror's name, never to a mirror or a list source. A definite answer holds until the next restart, which is when a mirror Canonical adds to a region later joins. Configuring [archives.*] disables it with the rest of the zero-config pool.

Each archive's pool is also probe-swept lazily: request-triggered and due again a week after the last one — at once when a mirror joins a live pool (a mirror list's refresh, the EC2 mirror's detection) or when consecutive elections measure every known mirror endpoint far above its usual latency, the sign of a network change; a daily floor holds back only addresses, so an address that arrived unmeasured and has not failed (a rotating DNS answer) brings the next sweep forward no sooner than a day after the last — every free mirror endpoint answers one conditional release download that measures TTFB and staleness (its body is far under the throughput sample floor, so throughput is left to real transfers), seeds the next election's conditional requests, and marks hosts whose "release file" is an error page or parked domain as failures; a sweep probes every dialable endpoint — each IPv6 address over every egress address the instance holds, IPv4 unbound — at a probe cap that scales with the egress addresses those probes dial from, so probes and bytes double at a two-provider site while the sweep's wall time does not. The sweep walks the pool in echelons rather than in the feed's order: the daemon's egress country first, then that country's continent, then everything else, each echelon ordered by previous measurements and the feed's declared bandwidth over a shuffle — which takes the wait for a first local measurement from a median 21.2 s to 0.2 s across the feed's 84 mirror-holding countries. The egress country is read at the start of each sweep from Cloudflare's trace endpoint, with geoip.ubuntu.com queried in parallel as an independent second opinion and both answers recorded; the winner is cached in location.txt beside the mirror lists, and when neither answers and no cache exists the pool sweeps as one shuffled echelon, exactly as it did before. None of this is configurable and none of it needs to be — the echelon only decides when a mirror is measured, never whether it is chosen, and an unknown country simply degrades (ADR-0005). Scores persist across restarts in scores.json beside the geo cache (TTFB and staleness per archive endpoint, throughput per destination) — written a minute after any activity, after every sweep, and at shutdown, never while idle — and aptriever mirrors prints the resulting ranking (TTFB, throughput, how far each endpoint's release date trails the newest copy of the same release file, the egress address it dials from) straight from that file, no running daemon needed. An entry whose mirror is not in the pool at load rides along unchanged until the 30-day retirement, so a restart that races the mirror-list fetch cannot erase history it never applied. Election seating uses the same staleness measurement: an endpoint recently observed serving a stale release copy is moved behind the rest of the candidates rather than excluded outright, and a background pass after each settled election re-probes just those endpoints so a recovered one is seated normally again on the next election instead of waiting out a timer. systemctl kill --signal=SIGUSR1 x-aptriever asks for a whole-pool sweep on the next request, scores replaced rather than blended as after a network change — for labs and tests; normal operation never needs it.

On archives with a configured keyring (the zero-config archives verify automatically whenever gpgv is installed: each takes the system's keyring from /usr/share/keyrings/ — ubuntu-archive-keyring.gpg, debian-archive-keyring.gpg — when it is readable and otherwise the copy aptriever ships (aptriever/keyrings/ inside the installed package), refreshed from the distribution packages by tools/refresh-keyrings.sh, so a Debian host, an Ubuntu host and an Alpine container all verify both archives), aptriever verifies every InRelease body with gpgv before trusting it: a mirror serving a forged, tampered, or foreign-signed release file fails verification and can never be elected or enter release memory — though a forger cut before its own verification finishes still leaves a TTFB sample, like any cut racer — and shows up in sweep summaries under its own bad signature count. On the same archives, a genuine signed body served at another suite's path (resolute-security's InRelease answered for dists/resolute/) fails its racer under a suite error kind instead of winning the election — apt would reject the swap anyway ("repository changed its Codename"), but the mirror now loses the race before the body is served, and no such body enters release memory. A release naming neither Suite: nor Codename:, a flat repository, or an archive without a keyring is unaffected. apt's own Signed-By verification remains the end-to-end gate; aptriever's check protects mirror selection, release memory, and the cache from a forged body. A by-hash fill is checked the same way before it is trusted: the body is hashed and compared against the digest its own path names once it has fully arrived — a mismatch in a latency race fails that one endpoint and the race goes on without it, a mismatch on the single stream fails the download outright (close-delimited) or, with a known Content-Length, has already reached the client while only the entry fails to land, and a mismatch in a body reassembled from several endpoints (throughput race) fails the fill outright, so the entry is never renamed into the cache and never served from disk. On any immutable path, an empty body where the object cannot be empty fails the endpoint at its headers, or, close-delimited, at the stream's end where there are no headers to judge it by. Configure it per archive with keyring = "/usr/share/keyrings/ubuntu-archive-keyring.gpg" under [archives.*].

Alongside signature verification, aptriever decides for every address it is about to dial whether that address may be dialed at all — the outbound address policy (#89) — and trust comes from a mirror's provenance, never from the address itself. A mirror configured under [archives.*] may name any address but link-local, so a company mirror in private space or a LAN instance to chain to works; a mirror from the zero-config pool's own lists may name no internal address, so neither a hostile listed host nor its DNS can steer aptriever onto the operator's networks; and a client-named on-demand origin may name an internal address only when the listener is loopback-only, because such a client could already reach the private network itself and the proxy adds no reach — off a listener reachable beyond loopback it may not. Link-local (169.254.0.0/16, fe80::/10, where the IPv4 cloud metadata address 169.254.169.254 lives) is refused whatever the provenance; a metadata endpoint in another class — AWS's opt-in IPv6 endpoint fd00:ec2::254 is unique-local, Alibaba's 100.100.100.200 is shared address space — is refused for untrusted provenance by its class and dialable through a configured mirror as the operator's own trust, so the policy stays a list of classes rather than of addresses. The check runs where the address is dialed, so a mirror's DNS answer, a redirect target and a literal-IP origin meet one rule: a refused on-demand literal origin is answered 403, and a mirror or redirect target whose every address is refused fails like a dead host. The mirror-list sources, the location lookups and the EC2 mirror's metadata lookup are operator-constant URLs fetched outside this policy. One caveat: a loopback-only listener behind a reverse proxy or a port forward is misclassified as private, so it will trust an internal on-demand origin it should refuse — no key turns that off today (ADR-0005).

Point apt at it with an /etc/apt/sources.list.d/aptriever.sources (deb822) entry:

Types: deb
URIs: http://localhost:3143/ubuntu
Suites: noble noble-updates noble-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

A Debian client needs two stanzas: the archive through the pool, and the security suite, which has no mirror list, through its embedded origin:

Types: deb
URIs: http://localhost:3143/deb.debian.org/debian
Suites: trixie trixie-updates
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: http://localhost:3143/http://security.debian.org/debian-security
Suites: trixie-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Sources that embed the origin (URIs: http://localhost:3143/http://deb.debian.org/debian) need no change: an embedded spelling of a pool member races the pool.

do-release-upgrade rejects the /ubuntu spelling: its mirror check recognizes a proxied URI only when it ends with a known mirror's host and path. Every pool mirror's host+path therefore also works as a path prefix — the release-upgrade-safe spelling is:

URIs: http://localhost:3143/archive.ubuntu.com/ubuntu

The aliased URI races the same pool, shares the same cache tree as /ubuntu, and survives do-release-upgrade's in-place rewrite with no per-host configuration.

Canonical index requests (dists/…/Packages.xz and friends) are answered with a redirect to the by-hash/SHA256/… object named by the InRelease last served for that suite, whenever the elected release advertises Acquire-By-Hash: yes: apt can then never receive index bytes from a different generation than the InRelease it holds — the hard File has unexpected size error class during mirror-sync windows — and index files are raced and cached like any immutable object instead of pinned to one mirror. Paths the served list does not name (Release.gpg, a suite with no election yet, archives without by-hash) keep the reference-pinned fetch.

Any other deb repository — including https-only ones no ordinary proxy can cache — needs no configuration at all: embed its URL in the path.

URIs: http://localhost:3143/https://packages.broadcom.com/artifactory/saltproject-deb

aptriever fetches from the embedded origin (TLS included), caches its pool/ and by-hash/ content, and keys everything by hostname, so every client of the same repository shares one cache tree. An embedded URL that names one of a configured archive's own pool mirrors routes to that archive's racing pool instead of a single origin.

An embedded origin is served only under a repository root aptriever has seen prove itself, so an instance reachable from anywhere is no general-purpose forward proxy. A root proves itself the way apt already asks it to: a clearsigned InRelease, or a Release.gpg beside a Release that parses — fetched through this same instance, which a client's own apt update does. What is learned is the repository root (the request path minus its dists/<suite>/… tail), never the whole host, and it is forgotten 30 days after its last release fetch. Release files themselves are always fetched, so apt update is never refused; anything else under an unproved root is answered 404 with a body saying so and naming the fix. This is a check of shape, not of validity — aptriever holds no keyring for third-party repositories — and it reduces abuse rather than establishing a trust boundary.

Every unit of work appends a wide event to events_path (default aptriever-events.jsonl): one kind: "request" event per client HTTP request — served, rejected, failed, or disconnected — carrying client address, user agent, method, status, cache disposition (hit, tail, miss, bypass, or "" when no cache took part), and a reason on failures; a request the client abandoned before any answer carries status 0; one kind: "fill" event per cache fill with the per-endpoint network breakdown, including an empty error kind when a zero-length body answers for an object that cannot be empty, a hash error kind when a by-hash body does not match the digest in its own path, and an address error kind when the outbound address policy refuses the address an endpoint was about to dial or every address a redirect target offered, and a reason naming the failed cache commit beside completed: true when every byte arrived but the entry never landed; one kind: "sweep" event per probe sweep, whose failures break its failed count out by error kind and whose probes give one row per probe that ran — mirror, address, egress address, outcome (ok, not_modified, invalid, bad_signature, verifier_error, failed), the probe's own ttfb, and a failure's kind and detail — so which endpoints failed and how is read off the event rather than inferred from a total; a skipped seat and an unresolved mirror made no request and take no row. One kind: "recheck" event per laggard recheck pass lists in endpoints a row per endpoint it re-probed — mirror, address, outcome, the release date and lag_seconds behind the watermark it measured against, and a failure's kind and detail. empty, signature and suite bench the endpoint at once instead of after three strikes, and so does hash when one endpoint delivered the body (latency race, single stream) — a throughput fill's hash mismatch blames no single endpoint and benches nobody. Every event names the aptriever version that wrote it. The completed field is true only for a clean finish, and racing_gain — aggregate throughput divided by the fastest single participating endpoint — says whether racing helped. A transfer that failed or was abandoned mid-body still records the bytes and the per-endpoint breakdown it had already spent, so what a disconnect cost stays in the stream instead of leaving with the client. A throughput race's duplicates lists every duplicate it ran — which chunk, how many chunks were still unstarted when it began (pending_at_start, zero for an endgame duplicate), what the primary had delivered, who won, the bytes discarded and the wait a win is estimated to have saved — so whether duplication pays is a question for the events, not a guess. A throughput race also lists its transfers — every body-carrying upstream request with its role (anchor, chunk, duplicate), byte range, start, TTFB, end, bytes read and outcome (done, lost, failed, torn_down) on the fill's clock, with an error naming the endpoint, kind and detail behind a failed outcome and null behind the other three — and a timeline of [seconds, bytes received from upstream] at every transfer start and end, so the aggregate rate at each concurrency level is a query rather than a guess; parallelism names the cap the download ran under, and every request and fill event's network names the egress network it was written from (a digest of the public address as the egress lookup saw it, an IPv6 address by its /48; 32 bits of digest over the IPv4 space are enumerable by anyone holding the file), so one machine's fills from different sites compare apart, and every contribution, election sample and transfer names in egress the instance's own address its dial was bound to, "" when it dialed unbound, a duplicate naming both its own and its primary's in duplicate_egress and primary_egress, so a multihomed instance's fills split by provider. An upstream answer a score or a verdict is taken from carries its response headers on its row — all but two corner cases the spec names, the eviction refetch and a diverted racer answering after its event was written (docs/plans/2026-09-17-response-headers.md): a contribution the headers of its endpoint's first answer in the download, a transfer, an election sample (a 304's included), a laggard recheck's row and a sweep's probe row their own, the last two beside the probe's ttfb. They are [name, value] pairs as received — wire order, the name's own letter case, a repeated name as one pair per field, octets mapped through latin-1 — because front ends differ in exactly those (the EC2 mirror answers Server: with squid's X-Cache fields from one front end and server: without them from another), so a query matches names through ascii_downcase. Set-Cookie is never recorded, nor are the headers of a redirect hop or an error status; null means no answer arrived; and an answer whose pairs would pass 16 KiB as the stream writes them, escapes included, is cut there, its last pair carrying an empty name and the count of fields and bytes dropped. Each endpoint's contribution names the rule that seated it (seated_via) and the scores it was seated on (ttfb_score, throughput_score), so a seating decision can be judged against what the seat then delivered, and counts its own failures by kind in errors, so a fill with several failing endpoints attributes each kind to the endpoint that produced it. A race's initial fan-out labels its slots exploit (the top-ranked endpoints the download leans on), reserve (the two exploration slots that sample endpoints carrying no throughput score yet, without which the same few scored endpoints would win every fan-out and the rest of a ~600-mirror pool would never be measured), topup (the remainder of the ranking, so a caller can seat past busy endpoints) and laggard (moved behind the field for having served a stale release copy); outside the fan-out, fallback is a seat taken on the blocking path when nothing was free, pool a chunk worker's draw of the next endpoint by rank, reseat a seat given up to wait for admission and then taken again, substitute a stand-in for an endpoint that failed mid-download, replacement a fresh racer entering a latency race, and reference the pinned mirror a mutable path streams from. The event's substitutions counts the seats drawn because an earlier one failed: every substitute, every replacement, and each further upstream a single-stream walk moved on to. journald carries only daemon-level signal; per-request narration lives at DEBUG. A racer whose task ends in an exception that is neither an upstream failure nor a cancellation nor a verifier failure (the election reports that one itself, once) is aptriever's own defect: one ERROR line says it died of a bug and carries the traceback, naming the archive, the path and the endpoint, while the download goes on as after any racer that delivered nothing — a replacement is drawn. Traffic aiohttp's HTTP parser rejects — an unparseable request line, a malformed or oversized header — is answered 400 before any route exists and leaves no event; aiohttp logs those itself. An events file that cannot be written (full disk, wrong permissions) costs events, never requests: one ERROR names the fault, one INFO names its end, and the daemon keeps serving throughout.

Racing costs bytes. Over 41 days of one laptop's events to 2026-08-27 (5.6 GB delivered on cache fills), mirrors sent 13.5 % more than apt received. Almost all of it is the latency race, which streams a small file from every racer until the first finishes and cuts the rest: 76 % extra on the 994 MB of files it served, because a loser cut at the win has often delivered most of the body. Endgame duplication on chunked large files discarded 5.5 % of what it delivered in its first twelve fills (11.3 MB of 208 MB). Both are the price of finishing sooner on a link that is not the bottleneck; on a slow link they are the bottleneck, which #73 measures before anything changes.

Nothing in the daemon reads the stream back — jq over events_path answers the questions this section raises. Each of these is one query over one instance's events file; the stream is per-instance, so comparing vantage points means comparing their files (#42, #32).

# Racing overhead: percent extra the mirrors sent over what apt received, whole stream. Cache hits cost no mirror bytes, so this falls as the cache warms.
jq -s '([.[]|(.mirrors//[])[].bytes]|add) as $sent | ([.[]|select(.kind=="request").bytes]|add) as $apt | {apt_received_MB:($apt/1e6|round), extra_pct:((($sent/$apt-1)*1000|round)/10)}' events.jsonl

# Did racing help? Median racing_gain per mode; below 1.0 is a race losing to its own best participant (#74).
jq -s '[.[]|select(.racing_gain)]|group_by(.mode)|map({mode:.[0].mode, fills:length, median_gain:(map(.racing_gain)|sort|.[length/2|floor])})' events.jsonl

# Latency-race waste: percent extra over the size of the files fetched, the bytes the losers had already delivered when the win cut them. files_MB is what the fills wrote, not what any client received.
jq -s '[.[]|select(.kind=="fill" and .mode=="latency")] | ([.[].mirrors[].bytes]|add) as $s | ([.[].bytes]|add) as $g | {files_MB:($g/1e6|round), extra_pct:(($s/$g-1)*100|round)}' events.jsonl

# Does duplication pay? Every duplicate's outcome and what the wins cost in discarded bytes, endgame duplicates (pending_at_start 0) and head-of-line ones apart (#33, #74).
jq -s '[.[]|(.duplicates//[])[]] | group_by(.pending_at_start>0) | map({kind:(if .[0].pending_at_start>0 then "head_of_line" else "endgame" end), duplicates:length, discarded_MB:(([.[].bytes_discarded]|add//0)/1e6), outcomes:(group_by(.outcome)|map({(.[0].outcome):length})|add)})' events.jsonl

# Was a seat worth its rank? Bytes and TTFB delivered per seating rule. `reserve` delivering nothing is the exploration slots doing their job; an `exploit`, `pool` or `topup` seat that delivers nothing is the finding (#76, #77).
jq -s '[.[]|(.mirrors//[])[]|select((.seated_via//"")!="")]|group_by(.seated_via)|map({via:.[0].seated_via, seats:length, MB:(([.[].bytes]|add)/1e6), median_ttfb:([.[]|.ttfb//empty]|sort|.[length/2|floor])})|sort_by(-.seats)' events.jsonl

# Does the k-th open transfer add throughput? Aggregate Mbps between consecutive timeline points, grouped by how many transfers were open, median per level (#77). A transfer counts as open from its request until its end, header wait included; intervals under 0.05 s are dropped as timer noise. Run per network: add `select(.network=="…")` after the first select.
jq -s '[.[]|select(.timeline)|. as $e|range(0;($e.timeline|length)-1) as $i|$e.timeline[$i] as [$t0,$b0]|$e.timeline[$i+1] as [$t1,$b1]|select($t1-$t0>0.05)|{k:([$e.transfers[]|select(.started_at<=$t0 and (.finished_at//1e9)>$t0)]|length), mbps:(($b1-$b0)*8/($t1-$t0)/1e6)}]|group_by(.k)|map({open:.[0].k, intervals:length, median_mbps:(map(.mbps)|sort|.[length/2|floor]|round)})' events.jsonl

# Which cache answered? First-byte time per mirror and cache verdict over throughput transfers; names are matched through ascii_downcase because front ends differ in letter case, and a mirror that names no cache groups under "" (#151, #152); -a keeps a hostile value's control characters escaped on the way to the terminal.
jq -s -a '[.[]|(.transfers//[])[]|select(.headers and .ttfb)|{mirror, ttfb, verdict:([.headers[]|select(.[0]|ascii_downcase|test("^(x-cache|x-cache-status|cf-cache-status)$"))|.[1]]|join(" / "))}]|group_by([.mirror,.verdict])|map({mirror:.[0].mirror, verdict:.[0].verdict, transfers:length, median_ttfb_ms:((map(.ttfb)|sort|.[length/2|floor])*1000|round)})' events.jsonl

# Mirror snapshot lag: median hours behind the freshest election participant, worst first.
jq -s '[.[]|(.election//[])[]|select(.skew_seconds!=null)]|group_by(.mirror)|map({mirror:.[0].mirror, samples:length, median_skew_h:((map(.skew_seconds)|sort|.[length/2|floor])/3600), elected:([.[]|select(.elected)]|length)})|sort_by(-.median_skew_h)|.[0:10]' events.jsonl

License

AGPL-3.0-or-later, see LICENSE: a network service, so the network-use clause is the point.