Keble Scraper API

The API package owns FastAPI composition, Mongo lifecycle/index persistence, Redis-backed egress budgets and sessions, Celery workers/beat, secure HTTP and Cloudflare fetching, private OSS artifacts, HMAC callbacks, authentication, auditing, and Sentry instrumentation. It imports keble-scraper-contract; the contract never imports this package.

Version 0.2.0 retains the undiscounted Bright pay-as-you-go baseline for pre-dispatch cost reservation so temporary coupon pricing cannot understate production spend and releases typed provider recovery, crash-safe callback and job budgets, bounded history, artifact-namespace ownership, ordered retention, and the guarded first-release reset.

The workspace consumes keble-scraper-contract 0.3.0 as the foundation for the API 0.2.0 recovery release. Attempt/job/callback persistence stores discriminated typed failure snapshots instead of terminal strings, and retry classification uses their finite dispositions. Public HTTP errors use the same envelope; mandatory retention revisions prevent delayed pin delivery from reversing a newer paid-storage policy.

Each job and callback row owns its immutable attempt budget; jobs also own a fixed deadline. Lease acquisition and recovery enforce those stored values, temporary capacity retains its future horizon, and unexpected runtime faults remain leased for expiry recovery rather than becoming a false network failure. Cache pointers cannot outlive referenced artifacts, missing cache metadata invalidates the pointer before refetch, and expired callback leases are recovered or dead-lettered by maintenance.

The WP8 clean-break command lives at scripts/reset_pre_release_state.py and delegates to the package-owned typed maintenance module. DRY_RUN is the default. Local/test-only APPLY deletes referenced binaries before their Mongo metadata, then the fixed Scraper collection allowlist and exact Redis namespace. It requires explicit writer-stop, database, Redis, artifact-owner, snapshot, artifact-inventory, campaign, and immutable-audit evidence; failures propagate with metadata or RUNNING evidence retained for replay. See ../docs/RUNBOOK.md.

HTTP, Cloudflare, and registered callback transports own reusable connection pools closed by ScraperContainer. The shared Data Infra Retry-After parser is used at every response boundary, with durable due times rounded upward to Mongo millisecond precision. Maintenance projects due job ID plus transport policy in one indexed read and publishes with configured bounded concurrency.

Maintenance also owns bounded historical retirement. Terminal Jobs without a callback receive an absolute retention clock immediately; callback Jobs receive one only in the same transaction that marks their outbox event delivered. Pending, leased, and dead-letter outbox rows therefore cannot be purged. One Mongo singleton lease serializes each minute's bounded purge, and the majority transaction deletes each expired Job with all immutable Attempt children before reporting success. Delivered callbacks and audit facts use independent absolute clocks. Defaults are 90 days for Job/Attempt evidence, 30 days for delivered callback replay, 400 days for security audit, and 1,000 rows per collection per minute; every value is a typed SCRAPER_* environment setting.

Sentry is initialized explicitly for the API, Celery parent, every worker child, and beat. Runtime tags distinguish those processes while credentials, URLs, request bodies, and provider payloads stay out of telemetry configuration.

Side effects if changes:

• Shopify must repin and exhaustively map local versus upstream failures;
• frontend error presentation and OpenAPI generation ship with the API release;
• pre-production 0.1.x scraper job rows require the guarded reset/repair step.

Runtime commands:

uv run --package keble-scraper-api keble-scraper-api
uv run --package keble-scraper-api keble-scraper-worker fetch
uv run --package keble-scraper-api keble-scraper-worker render
uv run --package keble-scraper-api keble-scraper-worker proxy-health
uv run --package keble-scraper-api keble-scraper-worker maintenance
uv run --package keble-scraper-api keble-scraper-beat

Service-authenticated consumers may download a verified artifact through the audited signed-URL endpoint and synchronize a downstream retention pin with a required positive retentionRevision. The Mongo transaction increments the artifact reference only on first owner-pin creation, applies only newer policy revisions, ignores stale deliveries, and rejects a divergent replay at the same revision. Both finite archive expiry and indefinite latest retention are valid ordered policies. The operator and service paths share the same public download route but retain separate authorization and audit identities.

The download service reserves a server UUID, places it in the immutable audit event and the signed URL's x-keble-audit-id query, and returns the URL only after the audit append succeeds. OSS GetObject logs can therefore reconcile reads exactly. New OSS objects also carry their canonical SHA-256 as private user metadata for bounded integrity checks.

For a multi-process local canary without OSS, configure both SCRAPER_LOCAL_ARTIFACT_DIRECTORY and SCRAPER_LOCAL_ARTIFACT_BASE_URL, then serve only that isolated directory on localhost. The development adapter keeps content-addressed bytes shared between API and Celery workers; staging and production reject it and continue to require private OSS.

Empty or whitespace-only values for those two optional local-artifact settings are treated as absent. This supports explicit Compose overrides when private OSS replaces a legacy local-artifact env_file configuration; all backend-completeness and environment restrictions still apply.

The optional OSS endpoint, bucket, access-key ID, and access-key secret are also normalized from blank to absent as one typed tuple. Partial or empty production configuration fails startup before any external request or artifact write.

Startup now applies two independent artifact fences before readiness. Mongo persists and compares the complete non-secret backend identity: backend kind, deterministic prefix, OSS endpoint/bucket or resolved local directory, and the configured owner. The binary store also contains one deterministic owner canary inside that prefix. Existing rows without a complete marker, backend/location changes, and a second owner all fail closed before workers accept jobs.

Set SCRAPER_ARTIFACT_NAMESPACE_OWNER in staging and production to a stable, non-secret value unique to the Mongo deployment/database that owns the prefix, for example cn-hangzhou/mongo-primary/keble_scraper. Do not reuse it for a second database. Development/test may derive environment/database locally. The pre-production prefix-only marker is intentionally incompatible and must be cleared or explicitly migrated before this release line starts.

When a local VPN returns fake-IP DNS answers, configure both SCRAPER_DEVELOPMENT_HTTP_PROXY_URL and SCRAPER_DEVELOPMENT_DNS_OVER_HTTPS_URL. The worker validates real DoH A/AAAA answers through the explicit proxy, preserves IP pinning, and applies the proxy only to the verified DIRECT route. The pair is rejected outside development/test.

Every route admission is atomically assigned a future Redis timestamp derived from its requests-per-second policy. A fetch worker asynchronously waits for that timestamp, preserving the requested route and job attempt; internal burst pressure therefore does not become terminal ROUTE_UNAVAILABLE evidence or a paid fallback.

The repository release workflow builds this package's runtime image only from an annotated keble-scraper-api-vX.Y.Z tag whose version matches this package. GHCR version/source tags are discovery aids; production consumes the digest evidence artifact and pins ghcr.io/keble-ai/keble-scraper-api@sha256:....