Python Consumer (Minimal Reference)

Install (from repo)

cd implementations/python-consumer
uv sync
uv pip install -e .

Requires Python ≥3.9. Runtime dependency: requests.

API

• fetch — fetch and validate manifest; graceful fallback on errors
• diff — compare manifest to cached site_rev / per-URL rev. Crawl changed + new + fallback_urls (rev-decrease anomalies); do not treat anomaly URLs as skippable
• audit — identity-encoding digest check (streams the body with a size cap)
• check_site — fetch + diff + optional sampled audit plan
• verify_live — fetch a manifest and sample live identity-encoded responses for digest verification
• identity_digest — stream-hash a stream=True response body, capped at max_bytes, for custom audit pipelines
• validate_manifest, resolve_url_key, manifest_url — validation and URL helpers
• format_state_header, parse_state_header — strict optional PageDigest-State helpers

resolve_url_key raises ValueError("url-key-transport-normalization") for valid keys that the HTTP stack would rewrite, including dot segments, empty queries, escapes of unreserved characters, and lowercase percent escapes. These keys remain valid protocol keys; callers should fall back to their normal crawl path rather than associate a normalized URL's response with the original key. audit and reconciliation report such targets as inconclusive without fetching them.

CLI

pagedigest verify-live https://example.com --sample-size 25

verify-live exits 2 on digest mismatches so it can act as a deployment gate. Redirects, network errors, and body-size caps are reported as inconclusive.

Example

from pagedigest import check_site

decision = check_site(
    "https://example.com",
    cached_site_rev=12,
    cached_revs={"/": 3, "/about": 1},
    sample_audit_rate=0.01,
)

After a successful manifest check, an integration may make its observed state visible on subsequent page requests:

from pagedigest import format_state_header

headers = {
    "PageDigest-State": format_state_header(
        decision["manifest"]["site_rev"],
        "/.well-known/pagedigest.json",
    )
}

This is a corroborating observation signal, not authentication. See SPEC.md §5.4.

Conformance fixtures: tests/test_vectors.py exercises ../../test-vectors/.

Persistent cache example

uv run python examples/cache_persistence.py https://example.com ./pagedigest-cache.json

The example stores site_rev, per-URL rev, ETag, and Last-Modified between runs and caches fetched page bodies beside the state file. Bodies and state are replaced atomically, page responses are size-capped, redirects are rejected, and manifest state advances only after every required page fetch succeeds. Use --pages to choose a body-cache directory and --max-page-bytes to change the default 10 MiB response cap.

Safe reuse (0.2.0 and later)

Pass only revisions for results you still have in cached_revs. check_site compares those entries even when site_rev is equal. Supply cached_manifest together with its ETag/Last-Modified to enable conditional requests. A 304 still returns a full download/audit plan; never interpret not_modified as completion. The function plans audits; the caller must execute audit_candidates.

The persistent example now stores the validated manifest and content-addressed bodies, validates body integrity before reuse, and runs audits on 200 and 304 cycles. --audit-rate selects the fraction (default 0.01). Mismatch marks the snapshot distrusted and the next run refreshes all described resources; successful verified refresh clears distrust. Inconclusive audits return nonzero for retry. A digest change at the same revision also triggers conservative refresh. Anomalous manifests require ordinary crawl fallback, which this collection-cache example does not implement. Partial coverage needs separate discovery/removal policies.

The example serializes writers using a state lock. Content-addressed orphan bodies from interrupted cycles are harmless and can be garbage-collected offline after checking all live state references. Use one state file per origin. Restoring old consumer metadata without corresponding bodies never authorizes skipping.