divergulent 0.1

Measure how far a Debian machine has drifted from pure upstream

divergulent

(Working name — I am bad at naming things.)

How divergent from pure upstream is this machine?

divergulent is a tool for answering that question about a Debian system. It looks at the packages you actually have installed and reports, per package and as a whole-machine summary, how far your distribution has drifted from what upstream actually ships.

Why

It is hard, as a user of a Linux distribution, to tell how stale or how divergent the software being packaged for you is compared to pure upstream. That gap is also a supply-chain concern: malicious change does not have to be introduced at the upstream author layer — it can just as easily be introduced at the distribution layer, as a carried patch. Today there is no easy way for a user to ask "how much of my machine is not what upstream released?" and get a reasonable answer.

Two axes of drift

divergulent measures drift along two distinct axes, because they have different causes and different data sources:

1. Staleness — the packaged version is behind pure upstream (version lag). "Am I running something old?"
2. Divergence — the distribution ships code that is not in any upstream release (carried patches). Same version number, distro-only changes grafted on. This is the axis that is almost invisible to users today, and the one most relevant to the supply-chain question.

A large, trusted patch set (e.g. the kernel) is normal — so divergulent aims to provide visibility and ranking, not a verdict.

Approach (early days)

divergulent is written in Python and, for its first swing, runs entirely on the local machine:

• It reads the installed-package set from dpkg.
• It leans on data Debian and the ecosystem already publish rather than crawling upstream version control itself:
  • Repology for the staleness axis.
  • sources.debian.org debian/patches/ plus DEP-3 headers for the divergence axis.
  • (Later) UDD, uscan/DEHS, and Wikidata.
• It treats sources honestly: heuristic or editable signals are surfaced as uncertainty, not presented as fact.

The local-package inventory is sensitive (it fingerprints the host), so the default posture is local-only — nothing leaves the machine.

Documentation

Reader-facing documentation lives in docs/: what the project is and why (docs/what-is-divergulent.md), how the processing pipeline works end to end (docs/workflow.md), and a complete description of the deterministic classification rules — every rule, its precedence, and its measured corpus hit rate (docs/deterministic-rules.md). The planning documents that drove each phase are under docs/plans/.

Usage

List the installed packages mapped to their source packages:

divergulent inventory          # aligned table
divergulent inventory --json   # machine-readable

Report packages that are behind upstream (the staleness axis, via Repology):

divergulent staleness          # packages behind upstream, worst first
divergulent staleness --all    # also show current and unknown
divergulent staleness --json   # machine-readable

Staleness is heuristic: it relies on Repology resolving your Debian source package to an upstream project, and reports unknown (never "behind") when it cannot. Each source is looked up individually and cached locally (~24h), and Repology is queried politely (≤1 request/second).

Report how many patches each package carries (the divergence axis, via sources.debian.org):

divergulent divergence            # packages carrying patches, most first
divergulent divergence --all      # also show clean / native / unknown
divergulent divergence --limit 50 # cap how many source packages are queried
divergulent divergence --workers 4 # fewer concurrent requests (default 8)
divergulent divergence --json     # machine-readable

The whole-machine view reports a patch count per package, using one request per source so a full run stays fast and polite. sources.debian.org has no documented rate limit, so requests run concurrently — --workers (default 8) bounds how many are in flight at once and is the politeness control; --workers 1 is fully serial. For the per-patch DEP-3 classification (forwarded-upstream vs Debian-only vs unknown), either drill into one package with divergulent show <package> (see below), or classify the whole machine with --classify:

divergulent divergence --classify   # Debian-only/forwarded/unknown per package
divergulent score --classify        # ranked, weighting Debian-only patches

--classify fetches each source package's packaging (the .debian.tar.*, not the upstream source) from your configured apt mirror — so it needs deb-src indices enabled (apt-get update after adding them). Without them it prints a notice and falls back to patch counts.

Combine both axes into one ranked, whole-machine answer:

divergulent score                 # ranked drift report + whole-machine summary
divergulent score --all           # include packages with no detected drift
divergulent score --limit 50      # cap how many source packages are queried
divergulent score --json          # machine-readable

score is the heaviest command (it queries both axes for every source package), so it shares one rate-limited HTTP client, reuses the caches the other commands populate, and supports --limit and --workers. Repology stays at ≤1 request/second whatever the worker count, and the sources.debian.org fetches overlap under that wait, so a cold score is bounded by the Repology half. The score only ranks; both axes are always shown. Note that being behind pure upstream is expected on a stable Debian release and is weighted lightly — carried patches are the stronger signal. (Use show for the per-patch Debian-only/forwarded classification of any package.)

The long whole-machine commands (staleness, divergence, score) show live progress on a terminal (and periodic lines in logs); pass --quiet to suppress it. The first run is slower while it builds a ~24h-cached snapshot of upstream versions; later runs reuse it and are near-instant.

If you have a precomputed cache bundle (the gzipped artifact the builder produces — see Status), point any of these commands at it to resolve covered packages instantly from disk instead of querying the network, falling back to live lookups only for what the bundle does not cover:

divergulent score --bundle cache-debian13.json.gz       # both axes from the bundle
divergulent staleness --bundle cache-debian13.json.gz
divergulent divergence --bundle cache-debian13.json.gz

The bundle is read locally (your package list never leaves the machine) and is used only if its schema is recognised and it describes the Debian release you are running; otherwise the command prints a notice and runs fully live. A package present in the bundle but installed at a different version, or absent entirely, falls back to a live lookup — so results never regress, and unknown still means genuinely unresolved.

Rather than pass --bundle every time, download the bundle once and let the commands find it automatically:

divergulent cache pull                       # download + store this release's bundle
divergulent cache pull --cache-url URL       # ... from a specific URL or mirror
divergulent score                            # now uses the stored bundle, no flag needed

cache pull (no arguments) downloads the bundle the project publishes for your release, checks it is recognised, verifies it, and stores it under the cache directory; later runs use it automatically (an explicit --bundle still overrides). The bundle is rebuilt and re-published daily (with a weekly full rebuild) to a stable URL, so cache pull refreshes it on demand; pass --cache-url to use a mirror or a hand-hosted bundle. Divergence from a stored bundle is always used (a fixed version's patches never change); staleness is used only while the bundle is fresh (within a week) — past that, staleness is queried live so newly-behind packages are not missed, while divergence still comes from the bundle.

A downloaded bundle is untrusted, so two independent checks run before it is stored:

• Signature — the bundle is signed in CI with Sigstore. Install the optional verifier (pip install divergulent[verify]) and cache pull checks the signature against the publishing workflow's identity, refusing a bundle that fails. Without the extra the check is skipped with a notice (use --require-signature to make a missing/failed signature fatal).
• Spot-check — always on, needs no extra: a random sample of the bundle's entries is compared against the live origin, and a bundle whose data demonstrably disagrees is refused (a transient live failure never causes a false refusal). Tune with --spot-check N (0 disables).

--insecure skips both. Re-check a stored bundle anytime with divergulent cache verify.

Drill into a single installed package:

divergulent show bash          # per-patch detail with Debian bug links
divergulent show bash --json   # machine-readable

show lists each carried patch with its classification, description, and any bug references the patch declares (Debian references are linked to bugs.debian.org). A patch that declares no bug shows "none declared" — it means the patch does not reference one, not that none exists.

Patch classification (an optional second bundle)

Beyond the DEP-3 forwarded/Debian-only class, there is a richer, curated classification of what each patch actually is — its category (security, feature, bugfix, packaging, documentation, test…), plus security-risk, install-base reach, and reviewability axes. That curation is expensive (deterministic rules, a verified LLM tier, and human review), so like the divergence cache it is done once, centrally and published as a small signed bundle keyed by patch fingerprint. Pull it and show annotates each patch with its category and why:

divergulent cache pull-classification   # download + verify + store this release's bundle
divergulent show bash                    # patches now carry: class, axes, and the deciding rule

The client runs no classifier and no LLM — it hashes the patch body it already fetched and looks the verdict up in the bundle (hashing a diff is not classifying it). Without the bundle, show behaves exactly as before. The bundle is signed and verified the same way as the divergence cache; it grows as review settles more of the residue, so re-pulling simply enriches what you see.

Status

Five commands work against real data: divergulent inventory (installed packages → source packages), divergulent staleness (behind pure upstream, via Repology), divergulent divergence (carried distro-only patches, via sources.debian.org), divergulent score (both axes combined into a ranked, whole-machine drift report), and divergulent show (per-package patch detail with Debian bug references). The plan lives in docs/plans/PLAN-initial.md; see docs/plans/index.md for the plan index, including planned next steps (Debian BTS cross-referencing, and a patch-hygiene assessment).

A published precomputed cache is in progress (docs/plans/PLAN-published-cache.md): the slow half of a cold run (staleness + divergence) is a function of the Debian release, not of your machine, so it can be computed once centrally and downloaded as a small signed bundle. Two pieces exist now: a central builder (divergulent cache build, run in CI) that sweeps the whole archive into a ~0.73 MB gzipped bundle; client consumption — the --bundle PATH flag and cache pull resolve covered packages from a bundle (downloaded and stored locally, used automatically, with a live fallback); trust — the bundle is Sigstore-signed in CI, verified on the client (with the optional verify extra) and always spot-checked against live origins; and publishing — a scheduled CI job builds, signs and publishes the bundle daily to a stable URL, so cache pull just works. Growing the published cache to a Debian 11/12/13/testing/unstable matrix is tracked in the road-to-1.0 plan.

A patch-classification pipeline (curation-side, for whoever builds the published cache — not something end users run) turns the carried-patch residue into an explainable, signed classification, layered by cost: free deterministic rules settle roughly a third of the ~60k distinct carried patches; a claim-blind, adversarially-verified LLM tier triages the substantive residue, prioritised by a security-risk gate; and a Sigstore-signed human tier (a CLI and a local web UI recording byte-identical verdicts against one append-only ledger) tops the precedence. Cheap deterministic axes — reviewability (patch size), reach (install base via Debian popcon), and an external CVE/bug cross-reference against pinned Debian snapshots — keep the expensive tiers pointed at the patches that matter most. The pipeline is documented end to end in docs/workflow.md, and every deterministic rule — what it matches, its precedence, and its measured corpus hit rate — in docs/deterministic-rules.md; the plan history is in docs/plans/PLAN-patch-classification.md.

Development

Tests and linting run through tox:

tox -epy3      # unit tests (stestr + testtools)
tox -eflake8   # style checks on the current change

CI runs the same checks on push and pull requests (.github/workflows/unit-tests.yml). A separate workflow (.github/workflows/sample-output.yml) runs a full divergulent score on a Debian 13 runner and uploads the rendered report as a build artifact, so reviewers can see how the output looks on a real machine (and as a live end-to-end check). A scheduled workflow (.github/workflows/build-cache.yml) builds the whole-archive bundle on a Debian 13 runner (tools/build-cache.sh), signs it (tools/sign-bundle.sh), and publishes it (tools/publish-cache.sh) to the rolling cache prerelease daily — incremental each day, a full rebuild weekly — so divergulent cache pull serves a fresh, signed bundle. Software releases are tag-driven (v*) and publish to PyPI via Sigstore-signed tags and PyPI trusted publishing — see RELEASE-SETUP.md for the one-time configuration.

Planning and pre-push workflow templates live at the repository root: PLAN-TEMPLATE.md and PUSH-TEMPLATE.md.

License

Apache License 2.0. See LICENSE.