keble-data-infra-api

Install the composed raw-data API with pip install "keble-data-infra-api>=0.1.0,<1". The published distribution resolves the released Amazon and TikTok providers; editable local sources in the uv workspace are development-only.

The sole FastAPI composition root for raw Amazon and TikTok provider data. Provider packages export framework-free services and manifests; this package owns route generation, exception mapping, runtime resources, and ingestion.

The Stand Out platform consumes this service over HTTP and does not import its contracts or provider packages.

Local provider environment

Build an ignored API-local .env from the existing Keble developer env files. The helper copies only the required named settings and prints key names, never credential values:

python scripts/build_local_env.py
uv run --env-file .env pytest -q tests/live -m live

The live suite uses independent RUN_KEEPA_LIVE, RUN_FASTMOSS_LIVE, and RUN_LIVE_ECHOTIK gates. A gate is written only when its complete credential set exists. Each configured provider runs as a separate canary, so one missing credential cannot hide another provider's result. Provider failures are test failures; missing opt-in credentials are the only valid reason to skip. Live providers honor PROVIDER_HTTP_TRUST_ENV through ApiSettings instead of hard-coding direct network access, matching the production runtime boundary. The pinned TikTok provider includes HTTPX's SOCKS transport, so production SOCKS4/5 proxy settings remain executable rather than failing during client construction. The image also pins the timezone-aware/warning-free keble-db checkpoint used by the local-full persistence proof.

Runtime ownership

keble_data_infra_api.main:create_app constructs the only raw FastAPI app. api.router:create_router builds discovery and provider routes from the registry, and api.error_handlers performs typed exception conversion. The Stand Out adapter resolves the single product-search capability through GET /v1/{channel}/providers and then calls its generated operation. The old /v1/commerce/search route and compatibility DTO snapshot are deleted.

API settings and immutable application values reuse keble-helpers.PydanticModelConfig or the contract-owned ContractModel. Durable aggregates use SchemaBase/MongoObjectBase; do not introduce a local ConfigDict or direct persistence BaseModel policy.

Mongo/Redis settings enable the API-owned provider infrastructure. The local env builder copies only MONGO_DB_URI, REDIS_URI, and named provider credentials; API-owned database/collection/namespace defaults stay isolated. Startup creates cache indexes before provider handlers run and shutdown closes all sync/async Mongo and Redis clients.

Video Enrichment projection is also API-owned and requires Mongo whenever its Result origin is configured. One document stores the source event, owner-token claim, attempts, failure state, and optional projection. Unique event/result indexes plus expiring claims make concurrent consumers safe; Redis entries are acknowledged only after the document atomically reaches completed. The old replica-local JSON state file is not supported.

The same typed gateway centralizes submission and job/result correlation:

POST /internal/video-enrichments:submit
GET  /internal/video-enrichment-jobs/{job_id}
GET  /internal/video-enrichment-jobs/{job_id}/projection

The projection lookup uses the API-owned job/status/completion-time compound index. Consumers never call the independent media service or infer result_id from provider storage. A projection 404 means durable completion ingestion is still pending and is safe for a bounded worker retry.

The submission gateway forwards the expected media SHA-256 as a dedicated integrity field. When an authorized provider transcript exists, the contract binds its normalized content checksum, source reference, and source media checksum before the independent service may skip ASR. The API never treats a provider title or video description as spoken captions.

The producer's Job/Result payload contains additional stage, artifact, usage, frame, warning, and error fields. Data Infra validates the typed stable subset through the contract's source-projection inheritance and ignores only those additive producer fields. All submission and public normalized models retain extra="forbid"; compatibility does not weaken command validation.

Configured provider routes include:

POST /v1/amazon/providers/keepa/products/lookup
POST /v1/amazon/providers/keepa/products/search
POST /v1/amazon/providers/keepa/products/rank
POST /v1/tiktok/providers/fastmoss/products/search
POST /v1/tiktok/providers/echotik/products/rank

Run the isolated all-provider generated-route canary with:

python scripts/build_local_env.py
uv run --env-file .env pytest -q tests/live/test_provider_routes_live.py -m live

Container and observability

The multi-stage image installs exact pushed provider commits and receives the GitHub token only as a BuildKit secret. It runs as UID 10001, exposes /health for liveness and /ready for configured Mongo/Redis readiness, and closes all runtime clients through FastAPI lifespan.

Build the credential-preserving Docker env and start the profile on the existing keble_default network:

uv run --directory keble-data-infra-api python scripts/build_local_env.py
uv run --directory keble-data-infra-api python -m scripts.build_local_docker_env
GH_TOKEN="$(gh auth token)" docker compose \
  --env-file keble-data-infra-api/.env.docker \
  -f deploy/compose.raw-api.yaml --profile raw-api up --build

.env, .env.docker, and the BuildKit secret stay outside Git and image layers. Set SENTRY_DSN, SENTRY_ENVIRONMENT, and SENTRY_TRACES_SAMPLE_RATE in the source env to enable error/performance telemetry. Registry discovery, generated operations, cache outcomes, lease contention, upstream work, batches, and video ingestion emit structural spans; request/response content and credentials are not attached.