Keble Idea

keble-idea owns the provider-neutral contracts and application behavior for public content snapshots, standout ideas, semantic concepts, relationships, immutable generations, durable work ledgers, and concept-first hybrid search.

It deliberately does not import FastAPI, Celery, MongoDB, Redis, Qdrant, Neo4j, raw providers, or keble-data-infra. The host application implements the protocols under keble_idea.application.ports and converts raw source responses into idea-owned schemas at its outer boundary.

The standalone 0.10.0 intelligence-domain contract release expands that ownership from a concept-only search package into the typed meaning layer for Keble's intelligence data plane. It keeps immutable Content, standout Ideas, Concepts, evidence, and lineage while adding scope-local Entities, reversible resolution, product hierarchy, unified relationships, observations, Events, TrendEpisodes, canonical commits, resource/cost dimensions, reprocessing, and a bounded query DSL. It releases only provider-neutral contracts; Data Platform continues to own adapter migration and coordinated integration gates.

The complete zero-context implementation and migration contract is docs/INTELLIGENCE_DOMAIN.md. The machine-readable legacy mapping is docs/compatibility/intelligence_domain_v1.json.

Version 0.10.2 adds InitialRelationshipLineageCandidateV1, a provider-neutral initial-activation projection candidate. After a producer calls KnowledgeRelationshipVersion.validate_activation(...) with loaded authority facts, it may construct this candidate in the same preparation boundary. The candidate requires one exact same-scope, same-commit, same-sequence, and same-generation assertion/ACTIVE-version pair; it then emits only canonical subject-to-object RELATIONSHIP_ASSERTION, assertion-to-Evidence EVIDENCED_BY, and assertion-to-version ACTIVATED_AS edges. Assertion edges retain original endpoints, predicate, and concrete node kinds, never resolved endpoints. This is not a fabricated activation receipt or a re-resolution/ supersession contract: cross-commit evolution is rejected until a separate policy defines it. The released lineage registry and its compatibility digest remain unchanged.

Version 0.10.3 adds the finite EXECUTE_SOURCE_PAGE operation and its sole SOURCE_PAGE_EXECUTION queue job type. Campaign SOURCE_PAGE work now admits only that operation through the existing external-I/O gate; it no longer borrows PUBLISH_INTELLIGENCE semantics. The contract remains provider-neutral: Data Infra owns the authenticated execution boundary and Platform owns durable attempt composition.

Version 0.10.1 adds the provider-neutral LineageProjectionPolicyV1 candidate. It is a finite, fail-closed publication policy rather than a graph-storage schema: exactly ten public node families project from 11 admitted canonical record kinds, while the other 14 record kinds are rejected before publication. It names all eight public relations, deliberately leaves DERIVED_FROM, BOUND_TO_SOURCE, and GENERATED_BY unsupported, and forbids EventKind as generic edge metadata. Event kinds remain typed metadata on an EVENT node with their existing ordered Entity-role contract. RELATIONSHIP_ASSERTION requires an exact current relationship-policy-v2 predicate and concrete Entity/Concept node pair; the lineage registry pins that immutable V2 SHA-256 without changing either existing relationship-policy registry.

The version notes below describe the released 0.5.0–0.10.3 lines. Their physical projection wording is historical; the store-neutral v1 intelligence domain is released here, while its coordinated Platform release replaces the multi-store intelligence path directly.

Version 0.5.0 changes immutable model execution from one physical deployment to one Infisical-catalog logical model. Every model-backed envelope, receipt, and accepted relationship freezes its purpose, ProcessingTier, selection rule, logical model/provider keys, model revision, catalog fingerprint, and generation. Every real provider request separately records a secret-free ProviderAttemptBinding with endpoint-pool revision, endpoint, deployment, region, and billing mode.

Version 0.6.0 aligns durable failure evidence with the universal raw contract: a physical HTTP rejection may truthfully have complete zero billable usage (for example reviewed authentication, permission, payment, invalid-request, or rate-limit rejection). A zero-attempt local gate still cannot carry usage, and USAGE_ONLY evidence remains financially incomplete.

Version 0.6.1 makes physical-attempt budgeting explicit on the immutable failure snapshot. A zero-attempt PLATFORM_RUNTIME observation—such as waiting for an already-paid completion projection—does not consume another provider attempt. Provider/adapter failures and missing evidence remain conservative and consume the budget. The wall-clock deadline still terminates local polling.

Version 0.6.2 completes the REVIEW2 replay boundary. Publication repositories return one canonical hydrated result for both first publication and receipt replay, so a redelivered paid extraction retains every idea, concept, and posting identity needed for downstream projections. Durable upstream failure evidence also keeps the authoritative classification origin and provider mapping revision, while separately metered identity correction remains owned by ExtractionService and is included exactly once in terminal receipt usage.

Version 0.7.0 introduces one evidence-backed semantic corpus for seller and shopping discovery. COMMON and UNCOMMON ideas and facets pass the same quality gates, persist normal concepts and postings, and share one vector/graph projection. SearchCorpusMode.DISTINCTIVE_ONLY is the seller-safe default; ALL_DETERMINED explicitly admits common purchase attributes for shopping search. UNDETERMINED content retains compact assessment, receipt, usage, and lineage facts but produces no searchable semantics. Content summary is derived as COMMON, UNCOMMON, MIXED, or UNDETERMINED only after publication gates.

Version 0.8.0 combines the separate provider-neutral content catalog with the durable restart-recovery contracts in one package candidate. Catalog projection uses PROJECT_CONTENT_SEARCH plus CONTENT_EMBEDDING; recovery extends the same canonical envelope with lease-renewal, expired-attempt planning, and finite sweep reports. It adds no parallel job state machine and remains unreleased until the coordinated Platform candidate and owner relevance gate are complete.

Version 0.9.0 releases the provider-neutral query-routing model-purpose contract. ModelPurpose.QUERY_ROUTING freezes and serializes the logical model selected for a bounded natural-language request without creating an IdeaJobType or widening the closed queue lifecycle mapping. The final Idea source authority for this contract is commit fb90d06; Platform retains route selection, request orchestration, and durable query/session attachment.

The content-catalog contract adds a separate provider-neutral catalog-search surface without changing semantic idea discovery. Both extraction result variants now return up to 48 evidence-grounded human term units inside the existing metered model call. ContentSearchProfileBuilder merges those proposals with typed structured content facts by canonical NFKC/casefold identity, preserves multi-word terms, caps only after merge priority, and builds one deterministic token-measured embedding input for every classification, including UNDETERMINED.

Content-profile publication is receipt-last: a terminal result is complete only when its deterministic profile row also exists, and replay returns that exact profile key. Host adapters implement the Idea-owned projection, ordered Mongo hydration, Redis session, cursor-codec, and runtime-resolution protocols; ContentSearchService composes them into the one new-search/cursor workflow. It pre-generates the search-session identity used for semantic cost lineage, probes one row beyond the configured result window, freezes only unique compact frontier rows for fifteen minutes, and fills pages from ordered active Mongo pairs without rediscovery. Exact admission always uses normalized keyword units; semantic similarity can rank only that admitted set and EXACT_ONLY never invokes query embedding. A generation-owned ContentSearchRankProfile changes match/semantic/freshness weights independently from the pinned CONTENT_EMBEDDING model, vector dimension, and Qdrant resource. Content-search point UUIDs reserve generation bits while retaining canonical content-key order inside a generation, so Platform can refine a newest-timestamp boundary with Qdrant's stable point-ID order instead of accepting an arbitrary subset of tied rows.

VectorGenerationRecord is the provider-neutral, immutable result passed to Platform's search_vectors_v1 conversion. It binds the resolved Content and CONTENT_SEARCH_PROFILE refs, profile-input SHA-256, CONTENT_EMBEDDING snapshot/resource, fixed v1 1,536-dimensional finite values, canonical vector SHA-256, COSINE distance, source core frontier, vector descriptor, and activation group. Its checksum preimage is exactly 1,536 IEEE-754 binary32 values, packed big-endian/network order as four bytes per dimension and concatenated in order. The builder normalizes values through that encoding, so a ClickHouse Float32 readback reconstructs the same digest. A tombstoned result is never semantically eligible. The stable key binds semantic subject/profile/input/model-resource/vector-generation/descriptor authority, not mutable frontier or activation lineage. Platform repositories own the transactional unique (record_key, record_version) policy: an identical digest rerun returns its existing row/version; a changed digest allocates exactly the next contiguous version; a conflict fails and never overwrites history.

INGEST and ENRICH envelopes carry the tier before model selection; model-backed envelopes carry it only inside ModelExecutionSnapshot. Platform must resolve the logical snapshot before admission, select a physical endpoint per attempt, and never substitute another logical model or tier during retry. The removed physical snapshot fields and prior model_id, error_code, and retry_classification contracts have no compatibility aliases.

ModelPurpose.QUERY_ROUTING identifies the logical model selected to translate a bounded natural request into a typed query plan. It can be frozen and serialized through ModelExecutionSnapshot, but it is not an IdeaJobType and does not expand the queue envelope's closed job-to-purpose lifecycle mapping. The host Platform owns request orchestration, route selection, and any durable query/session attachment.

Architecture

provider DTOs / immutable source artifacts
        -> Platform conversion and visibility authorization
        -> keble-idea scoped records + canonical IntelligenceCommit
        -> Platform canonical writer and atomic activation
        -> snapshot-pinned query repository

keble-idea defines meaning, identity, invariants, checksums, and protocols. It does not choose a database, table, index, queue name, provider client, router model, or execution order. Data Platform owns those adapters and composes them around these contracts.

Derived identities and lifecycle transitions are owner-type conversions: IntelligenceRecordIdentity.with_record_key and RelationshipCandidateResult.as_shadow reconstruct and validate the complete schema. Production domain/application code must not use model_copy(update=...), because Pydantic does not validate updated values on that path.

The v1 domain is organized around eight boundaries:

1. IntelligenceScope separates reusable PUBLIC truth from tenant-private truth. Every Entity, Concept, relationship, observation, evidence ref, merge, and query target carries that boundary. Repository maps use ScopedRecordLookupKey, which includes scope kind, scope key, record kind, and record key; a PUBLIC and tenant record cannot alias through a bare key.
2. EntityVersion distinguishes stable Brand, Storefront, Seller, Person, SocialAccount, ContentItem, ProductListing, ProductVariant, and ProductFamily identity. Raw/normalized terms, source bindings, policy/model revisions, and evidence remain append-only.
3. EntityResolutionDecision records create/bind/merge/reject/possible/reverse outcomes. Prefix overlap is candidate evidence only; accepted redirects do not delete the source record.
4. KnowledgeRelationshipAssertion selects an immutable versioned endpoint/evidence matrix. Historical relationship-policy-v1 stays byte-stable; current relationship-policy-v2 adds quality-gated Concept-to-Concept RELATED_TO, BROADER_THAN, ENABLES, and APPLIES_TO. SAME_AS remains absent: identity belongs to resolution, temporal changes belong to Events, and a completed zero-edge stage is a RelationshipEvaluation.
5. IntelligenceCommit groups bounded scoped mutations under canonical JSON, digests, counts, and immutable time. Physical batch serialization and activation mechanics remain adapter-owned.
6. KnowledgeObservation, Event, and TrendEpisode keep scalar facts, temporal occurrences, and noncausal precedence distinct.
7. IntelligenceSchemaRegistry, typed feature rows, and reprocess/rollback contracts make new metrics rebuildable without mutating old truth.
8. IntelligenceQueryPlan is a finite dumb-router output. Commutative groups normalize arbitrary model order; sort priority remains ordered. Standalone standout Ideas sort through the typed idea.confidence field, while TrendEpisodes sort through trend.second_interval_end, the end of their second half-open observation interval. Raw Content filters and sorts through content.observed_at, a DATETIME SearchProfile derived from its immutable observation instant.

Source campaign work is shard-and-page exact. SourcePageWorkIdentity requires both shard_ordinal and page_ordinal alongside its sole work_key and immutable request artifact. Neither ordinal has a compatibility default. A continuation may advance only from a parent identity in the same shard and at the immediately previous page; the outer Platform campaign controller owns that comparison before queue admission.

Automatic Entity merge authority is intentionally narrower than candidate generation. One endpoint-free EntityResolutionQualityStratum can be reused only for later Amazon-to-Shopify fashion ProductFamily pairs whose canonically loaded bindings, active Brand/category versions, market, model, prompt, cleaner, and policy facts exactly match its persisted 500-pair/0.99-lower-bound gate. The gate counts unique directional source/target pairs; multiple Evidence refs for one pair cannot inflate that threshold. Every pair binds the exact stratum, adjudication policy, and human outcome, and the gate recomputes its 95% Wilson precision lower bound from those outcomes. SKU/model-prefix overlap remains uncertain evidence and cannot create or merge an Entity by itself.

Canonical terms and source bindings are owned facts, not reusable name bags. Every CanonicalTermAssertion names its exact Entity. A source binding is unique by (scope kind, scope key, channel, provider namespace, resource kind, provider object id); ingestion service, evidence class, and market are provenance. Provider adapters must encode provider-defined market identity into the object id (for example US:<ASIN>). ProductListing activation additionally fences the exact binding version, channel, provider, resource kind, market, and currency, while ProductVariant remains canonical and has no direct source binding.

Content-to-Entity authority is two-stage and version-exact. An accepted ContentEntityResolutionDecision binds one immutable Content version to one active Entity version; ContentEntityLinkVersion materializes that decision with a semantic role. Relationship span/region evidence names those links, and activation resolves each link, decision, Entity version, Content version, and predicate-specific role. Bare Content evidence cannot authorize an Entity endpoint.

Model relationship activation is corpus-backed. A RelationshipModelQualityGate embeds exact distinct typed endpoint pairs, human outcomes, Evidence membership, stratum checksum, and adjudication policy; it recomputes its Lineage manifest, pair count, and two-sided 95% Wilson precision bound. Repeated evidence for one pair, a swapped symmetric pair, or an invented count/bound cannot unlock activation. Market/category strata are validated from active source bindings and exact category-membership authority, never caller booleans.

Coverage and time are reconstructible facts. CoverageCountProof hashes the exact subject/group/window/cohort count basis; a multi-cohort aggregate requires its own Lineage proof and cannot use one well-covered cohort to hide an under-covered member. Events resolve exact Observation rows, metric registry and derivation revisions, active Entity versions, every ordered subject role, source period end, and the complete Evidence union. Frozen sequence definitions state whether each leg needs any allowed rule or every rule per subject.

An ACTIVE reprocess run owns one complete multi-scope activation receipt. Each scope proof binds its base/final frontier and activation group to the target generation, and hot-search/vector rebuilds also bind the exact serving descriptor artifact. Receipts and rollback previews persist canonical self-digest-free body bytes plus their artifact checksum; rollback targets the full receipt rather than reconstructing state from a partial pointer. Reprocess page queue bindings carry a required immutable execution kind and the exact page-plan or derived-action artifact digest. INTERNAL_COMPUTE and EXTERNAL_IO authorize only their matching admission pair, while later DURABLE_RESULT_COMMIT and CONTROL_FINALIZATION actions retain their bounded no-I/O drain semantics. Provider/model pages therefore use the same owner gate and resumable envelope as internal pages without granting external-I/O authority from caller-supplied labels or to any other origin family.

Terminal visibility is proved from canonical storage, not from receipt-carried hashes. Platform resolves the exact generation-pinned commits, activation groups, assessment, and search profile into ContentPublicationActivationProof; Idea then verifies every-and-only mutation output and commit-set digest. Entity MERGE/REVERSE uses one EntityResolutionActivationReceipt, so its decision, redirect, hierarchy closure, and relationship revisions become visible together. Every variable persisted collection is bounded and guarded by an introspective contract test.

Content Catalog and contextual Idea discovery remain separate application profiles. The former admits evidence-grounded exact terms and may request typed semantic fallback; the latter is Concept-first and retains distinctiveness, postings, relationship expansion, and ranking explanation. The generic query DSL adds Entity/Content/Idea/Concept targets, typed scalar/text/temporal/path clauses, three-hop limits, finite registries, snapshot frontiers, evidence, and explicit unavailable/insufficient-coverage outcomes. Router numeric inputs are compiler-safe signed Int64 or Decimal(38, 12) values; named discriminated aliases preserve complete valueKind/clauseKind mappings in the generated validation schema even when scalar clauses are reused inside relationship paths. Temporal metrics retain this compact decimal shape and add one bounded currency field: it is required exactly when unit=CURRENCY and forbidden for every other unit, so CURRENT and DELTA price filters cannot compare implicit currencies.

Hydrated query results expose logical values rather than ClickHouse row projections. matchedScalars reconstructs the registered ScalarQueryValue union, so text, money, dates, booleans, numbers, and scoped refs retain their typed meaning. matchedTemporals carries the registered metric/statistic, DERIVED_FEATURE/OBSERVATION/EVENT source family, value, unit and optional currency, exact absolute interval, source channel, cohort, canonical denominator, group-scope digest, member count, CoverageResult, Evidence refs, and complete proof refs. relationshipEndpoints binds the same summaries to an Entity/Concept that terminates a returned path. Each detail family is canonical and bounded to 16 summaries; evidence/proof collections are canonical, single-owner, snapshot-authorized, and bounded to 64 refs. Existing result-item readers remain compatible because these additive collections default to empty. Existing top-level evidence, paths, and temporalSequences collections are also duplicate-free and canonical; path-internal node/predicate order remains semantic and is never sorted.

Coverage failure is not an empty successful query. An INSUFFICIENT_COVERAGE page contains canonical coverageFailures, each locating the failed temporal calculation at ROOT or RELATIONSHIP_ENDPOINT and naming the exact affected subjectRef. Root subjects use the finite result-record families; endpoint subjects must be Entity or Concept. The page cannot carry items, a cursor, or an unavailable reason, and it has no second page-level coverage array: coverageFailures[*].metric.coverage is the canonical path. A failure metric must be insufficient, while every root or endpoint matchedTemporal must be sufficient. In contrast, a COMPLETE page with no items and no failures is the valid zero-match result, while UNAVAILABLE keeps its finite infrastructure reason vocabulary.

Contextual Concept expansion is revision- and checksum-pinned by SearchProfile. It admits only the four v2 context predicates, keeps direct TAGGED postings and their COMMON/UNCOMMON distinctiveness unchanged, and bounds depth, per-origin edge fanout, expanded concepts, path confidence, and the final frontier independently. RELATED_TO is symmetric; BROADER_THAN, ENABLES, and APPLIES_TO retain direction in storage while a query may deliberately read reviewed reverse adjacency. Every accepted context edge requires ProfileEvidence and the existing 500-distinct-pair, 95% Wilson lower-bound model-quality gate. See docs/CONTEXTUAL_CONCEPT_RELATIONSHIPS.md.

Every generic query has one deterministic total order. Repositories apply the router-authored logical fields in tuple order, then optional semantic score, then the automatic QUERY_RESULT_TIE_BREAK_V1 suffix: scope kind, scope key, record kind, and record key, all ascending. The router never emits that scoped identity suffix. Cursor continuation must compare the same components in the same order, and a non-null cursor requires at least one item on its COMPLETE page so an empty-page continuation loop cannot form. For raw Content, Platform publishes content.observed_at into the ordinary DATETIME SearchProfile from ContentSnapshotRecord.observed_at; the field is not an adapter timestamp and does not make Content a graph endpoint.

Query identity and qualifying proof are deliberately separate. A ScopeServingFrontier owns one typed IntelligenceScope; an IntelligenceSnapshot binds the viewer, those exact frontiers, the authorization-policy revision, and all projection/vector generations into snapshot_sha256. QueryProofAuthorization then groups exact canonical proof refs by their true owner scope and binds them to that digest. An authorized tenant query may therefore return a PUBLIC result qualified by tenant-owned relationship assertions, observations, Events, derived features, evidence, Lineage, a canonical content-search profile, or ContentEntityLink. The same result cannot replay that proof for a PUBLIC viewer or another tenant. Search vectors remain repairable projections: semantic proof uses the canonical content-search profile ref plus the snapshot-pinned vector generation/ descriptor, never an invented vector-record identity. Every QueryPathSummary also carries the exact qualified_result_ref; this keeps a Content, Idea, or TrendEpisode result bound to its underlying Entity/Concept path without pretending those persisted identities are equal. Immutable Content never becomes a graph endpoint. A raw Content result carries one QueryContentGraphBinding whose exact ContentEntityLink authorizes a stable CONTENT_ITEM Entity anchor at the start of the path through the canonical CAPTURE_OF role.

Digest-critical query value objects are immutable through the full object tree: IntelligenceScope, ScopeServingFrontier, and ScopedRecordRef are frozen as well as their enclosing snapshot and proof receipt. Callers rebuild and revalidate a new value instead of mutating a nested scope, epoch, or key after its digest was accepted.

Unavailable query pages use one finite durable vocabulary: INTEGRITY_UNAVAILABLE, QUERY_EXECUTION_UNAVAILABLE, or SNAPSHOT_STALE. Temporary compiler migration guards are internal failures and must disappear when their contract lands; they never become public reason codes.

An all-owned schema audit rejects unconstrained public strings, including list items, keys, generations, cursor/session identities, and narrative fields.

Development

Python 3.13 and uv are required.

uv sync
uv run pytest -q -m "not live and not slow and not eval and not local_stack and not db_stack and not container"
npx --yes pyright .
uv build

The portable test lane uses only reusable in-memory adapters from keble_idea.testing. Stateful integration adapters and their isolation/cleanup belong to the host Platform. Build verification also installs the wheel into an empty environment and checks every name in keble_idea.__all__.

Contract rules

• Persist only HIGH and MEDIUM ideas. LOW candidate details are discarded.
• Persist evidence-backed COMMON and UNCOMMON semantics through the same publication path; only UNDETERMINED content omits semantic projections.
• Publish one deterministic content-search profile for every active content classification; do not infer missing profiles from historical receipts.
• Admit catalog results only through exact canonical term units. Semantic similarity is an optional post-admission ranking input, never an admission fallback, and exact-only search incurs no query-embedding attempt.
• Keep catalog rank generations independent from content-embedding generations; rank-weight-only changes rebuild frontiers without rotating vector resources.
• Classify ideas and facets independently, then derive the content summary from the surviving common/uncommon counts rather than trusting a model-owned label.
• Default search to DISTINCTIVE_ONLY; require ALL_DETERMINED explicitly for shopping discovery, and pin the mode into the cursor session filter hash.
• Keep one concept vector and graph node across both modes. A common-only concept remains projected while its determined availability is positive.
• Write exactly one minimal receipt for every content/generation terminal state.
• Hydrate the complete canonical publication result on receipt replay; never return an empty derived-key set for already-published work.
• Keep receipts provider-neutral: physical token/image/duration usage remains in ProcessingUsage, while monetary pricing and per-content allocation belong exclusively to the platform-owned financial ledger. Receipt schema version 4 rejects the removed cost_micros and loose model_id fields.
• Keep batch-identity correction in ExtractionService: one corrective call, one ordinal owned by the host, and one merged receipt usage total.
• Preserve the eight released semantic Concept values and the five generation-new CATEGORY/PRODUCT_CONCEPT/STYLE/ATTRIBUTE/PROBLEM values; provider/source metadata is not a Concept.
• Use standalone_concept_error at model boundaries so invalid facet labels receive a typed correction before publication applies the same strict policy.
• Merge only HIGH-confidence PURPOSE concepts automatically and keep it reversible.
• Vectorize concepts only unless a separately approved retrieval eval changes the design.
• Use immutable generations and active pointers; never mutate active indexes in place.
• Persist named, collection-owned integer-version migration checkpoints so outer workers resume by stable record key and never migrate a corpus at startup.
• Preserve original ASR and OCR independently from English-normalized fields; unknown-language OCR belongs in ocr_text_original, never guessed into ocr_text_en from the transcript language.
• Cutover only verified generations through compare-and-swap pointers and preserve the immediately previous verified generation for exact rollback.
• Report minimum-age or live-session-pin deletion guards as GENERATION_RETENTION_BLOCKED, never as an optimistic revision conflict.
• Stamp relationship coverage with the exact embedding, neighborhood, and relationship generations plus bounded neighbor similarities; a valid null/no-relation result still completes coverage. Persisted current/previous neighbor sets let outer adapters apply entry/exit hysteresis without all-pairs comparison.
• Keep relationship_priority_score and relationship_priority_at as bounded, provider-neutral scheduling hints. The platform derives them idempotently from posting quality/time so Stage-B graph budget reaches high-value/recent concepts first; they never change semantic concept identity or Stage-A search readiness.
• Compact alias postings asynchronously in bounded resumable pages; merge and unmerge correctness must never depend on compaction finishing first. Derived rows retain origin_concept_key, so unmerge removes them without deleting the original posting identities.
• Search is bounded, concept-first, count-free, and cursor-session based.
• Cursor signatures bind expiry, frontier revision, and the complete generation/profile identity.
• Operator lifecycle is canonical and typed (schema version 2): content moves ACTIVE -> ARCHIVED -> ACTIVE and enters PURGING terminally; ideas move ACTIVE -> RETIRED -> ACTIVE and enter PURGING terminally. archived_at / retired_at must match the state, transitions go through assert_lifecycle_transition, and public reads exclude non-ACTIVE rows.
• Operator job commands use the canonical status sets: retry targets TERMINAL_JOB_STATUSES, cancel targets CANCELLABLE_JOB_STATUSES (never LEASED work), and purge work routes through IdeaJobType.PURGE.
• Processing tiers are exactly ECONOMY, BALANCED, and QUALITY. They express requested cost/quality position, not provider identity. A retry keeps the frozen tier; changing it requires explicit reprocessing.

Durable provider recovery

Each queue envelope snapshots the complete active ProviderRecoveryProfileSnapshot, its matching attempt_limit, and an absolute deadline. Automatic retry retains that snapshot. A manual retry also preserves it unless the operator explicitly requests the active profile. Profile records are append-only and activation uses a revision-guarded pointer; changing the pointer affects future admissions only.

The initial profile is fixed at eight attempts, a 72-hour deadline, 30-second fallback doubled to a one-hour cap, a 24-hour maximum provider hint, deterministic 0-10% jitter, five unavailable failures in 60 seconds, a five- minute open circuit, one half-open probe, and blocked release batches of 100. Those are defaults for the first persisted profile, not module constants that silently mutate existing jobs.

retry_at_for_failure anchors delay and jitter to the failure's immutable observation time. Re-evaluating the same envelope attempt therefore returns the same not-before timestamp. A provider hint above the submitted 24-hour policy becomes BLOCKED; attempt/deadline exhaustion or permanent failure becomes DEAD_LETTERED; operator action becomes BLOCKED. Only a retryable failure within both budgets becomes RETRY_SCHEDULED.

JobFailureSnapshot.consumes_attempt_budget is the sole classification for whether a leased retry counts as external work. Queue decision and persistence must use it together: a zero-attempt Platform observation is evaluated against the prior physical-attempt count and its lease increment is refunded atomically. This preserves earlier paid attempts while allowing local projection polling until the immutable deadline.

Restart recovery uses the same envelope identity. QueueLeaseRenewal describes an exact owner/revision/attempt lease extension without implying a domain revision, while ExpiredEnvelopeRecoveryPlan classifies an expired lease as deferred, attempt reuse, fresh-attempt advancement, exhaustion, or a concurrent resolution. A full financial-attempt census supplies the stable attempt keys to abandon; a missing attempt is the only automatic-recovery case that may reuse the queue attempt number. QueueSweepReport is the shared bounded operator result and separates recovered work from live-claim deferrals, exhaustion, and concurrent resolutions.

UpstreamFailureEvidence retains bounded status/code/request identity, opaque quota scope, exact capacity, physical-attempt count, usage, and a completeness bit. Empty usage is known zero only when the producer explicitly marks it complete; incomplete empty usage and every USAGE_ONLY event remain billing unknown. Data Infra evidence also retains a provider-neutral classification origin plus the upstream adapter's mapping revision separately from the Platform mapping revision. This lets operators distinguish provider, contract, and Data Infra runtime classification without making Idea depend on a provider package. Payment requires a reviewed billing-code/account-status source; HTTP 402, negative capacity, quota, and rate limiting are never sufficient by themselves.

Side effects if changes:

• Platform Mongo queue/profile/incident adapters and central indexes implement these exact field paths and compare-and-swap semantics;
• Data Platform job and commerce-operation APIs expose the same failure/policy facts through generated clients and translated frontend labels;
• finance records failed physical usage exactly once while pre-provider gate rejection remains zero spend;
• queue decision and persistence must never duplicate or disagree with consumes_attempt_budget;
• Platform must apply each expired-envelope plan in one queue/financial Mongo transaction and must not clone the envelope or create a parallel status;
• EnvelopeProcessResult must continue to contain exactly one result reference or failure snapshot.

Images are a critical asset

Content payloads treat visuals as first-class evidence: ProductContentPayload.image_artifacts carries model-ready product photos, CommerceProfileContentPayload.image_artifacts carries acquired creator or seller portraits, ShortVideoContentPayload.thumbnail_artifacts carries safe ranked-video thumbnails without transient playback URLs, and VideoContentPayload preserves uploaded video bytes plus key frames. IdeaSearchMedia.from_payload projects the representative image/video (with IdeaSearchSourceSummary for type-specific display facts) onto every search item so consuming platforms can render media-rich cards. Artifact references stay checksum-addressed (ArtifactRef) — delivery URLs are always resolved by the serving platform at read time.

Commerce discovery content

Trending commerce ingestion stays provider-neutral after the platform boundary:

profile = CommerceProfileContentPayload(
    profile_type=CommerceProfileType.CREATOR,
    display_name="Rocket Lab Notes",
    image_artifacts=[acquired_profile_image],
)

video = ShortVideoContentPayload(
    title="A compact model-rocket launch guide",
    thumbnail_artifacts=[acquired_thumbnail],
)

Products keep ProductContentPayload; creators and sellers share the typed profile payload with a stable profile_type; channel-wide ranked videos use a separate short-video payload. Provider names, provider response bags, volatile playback URLs, and raw enum labels do not enter these durable contracts.

Language and standout evidence

ContentBody.original_language remains the backward-compatible ISO code. Optional ContentLanguageProvenance explains whether it was source-declared, text/media detected, or unresolved; a missing provenance on und identifies a legacy row. Provider-neutral ContentStandoutEvidence records why a source surfaced content without claiming profitability. Performance observations are identified by metric kind, unit, observed time, and window: equal duplicates collapse to the standout-linked evidence, while conflicting values are rejected.