keble-idea 0.3.0

Provider-neutral idea extraction, concept graph, generation, and search contracts.

Keble Idea

keble-idea owns the provider-neutral contracts and application behavior for public content snapshots, standout ideas, semantic concepts, relationships, immutable generations, durable work ledgers, and concept-first hybrid search.

It deliberately does not import FastAPI, Celery, MongoDB, Redis, Qdrant, Neo4j, raw providers, or keble-data-infra. The host application implements the protocols under keble_idea.application.ports and converts raw source responses into idea-owned schemas at its outer boundary.

Version 0.3.0 adds the canonical persisted job-failure and recovery-policy contract. It reuses only the provider-neutral failure/capacity/usage vocabulary from keble-helpers; it does not import Data Infra wire types or provider exceptions. Platform must convert each outer failure once into an immutable JobFailureSnapshot and may not restore the removed error_code or retry_classification fields.

Architecture

raw provider HTTP responses
        -> platform-owned adapters
        -> keble-idea schemas and services
        -> platform-owned persistence/projection adapters

MongoDB is canonical in the host application. Qdrant stores canonical concept vectors only, Neo4j stores the concept graph only, Redis stores TTL search frontiers, and RabbitMQ delivers identifiers for Mongo-backed queue envelopes. Ideas are retrieved through scalar idea-to-concept postings rather than idea vectors or multikey story-array scans.

Cost-bearing extraction and relationship protocols receive an EnvelopeExecutionIdentity copied from the atomically leased Mongo envelope. The envelope key identifies logical work; attempt_number identifies the physical broker/worker retry. Optional queued_at and leased_at preserve the exact durable queue interval without introducing provider or pricing ownership. Financial adapters include the identity in provider attempt keys so redelivery cannot overwrite or hide a second possible spend.

Development

Python 3.13 and uv are required.

uv sync
uv run pytest -q -m "not live and not slow and not eval and not local_stack and not db_stack and not container"
npx --yes pyright .
uv build

The portable test lane uses only reusable in-memory adapters from keble_idea.testing. Stateful integration adapters belong to the platform and must isolate and clean MongoDB, Redis, Qdrant, Neo4j, RabbitMQ, and artifact state after each test.

Contract rules

• Persist only HIGH and MEDIUM ideas. LOW candidate details are discarded.
• Write exactly one minimal receipt for every content/generation terminal state.
• Keep receipts provider-neutral: physical token/image/duration usage remains in ProcessingUsage, while monetary pricing and per-content allocation belong exclusively to the platform-owned financial ledger. Receipt schema version 2 rejects the removed cost_micros field.
• Use the eight finite semantic facets; provider/source metadata is not a concept.
• Use standalone_concept_error at model boundaries so invalid facet labels receive a typed correction before publication applies the same strict policy.
• Merge only HIGH-confidence PURPOSE concepts automatically and keep it reversible.
• Vectorize concepts only unless a separately approved retrieval eval changes the design.
• Use immutable generations and active pointers; never mutate active indexes in place.
• Persist named, collection-owned integer-version migration checkpoints so outer workers resume by stable record key and never migrate a corpus at startup.
• Preserve original ASR and OCR independently from English-normalized fields; unknown-language OCR belongs in ocr_text_original, never guessed into ocr_text_en from the transcript language.
• Cutover only verified generations through compare-and-swap pointers and preserve the immediately previous verified generation for exact rollback.
• Stamp relationship coverage with the exact embedding, neighborhood, and relationship generations plus bounded neighbor similarities; a valid null/no-relation result still completes coverage. Persisted current/previous neighbor sets let outer adapters apply entry/exit hysteresis without all-pairs comparison.
• Keep relationship_priority_score and relationship_priority_at as bounded, provider-neutral scheduling hints. The platform derives them idempotently from posting quality/time so Stage-B graph budget reaches high-value/recent concepts first; they never change semantic concept identity or Stage-A search readiness.
• Compact alias postings asynchronously in bounded resumable pages; merge and unmerge correctness must never depend on compaction finishing first. Derived rows retain origin_concept_key, so unmerge removes them without deleting the original posting identities.
• Search is bounded, concept-first, count-free, and cursor-session based.
• Cursor signatures bind expiry, frontier revision, and the complete generation/profile identity.
• Operator lifecycle is canonical and typed (schema version 2): content moves ACTIVE -> ARCHIVED -> ACTIVE and enters PURGING terminally; ideas move ACTIVE -> RETIRED -> ACTIVE and enter PURGING terminally. archived_at / retired_at must match the state, transitions go through assert_lifecycle_transition, and public reads exclude non-ACTIVE rows.
• Operator job commands use the canonical status sets: retry targets TERMINAL_JOB_STATUSES, cancel targets CANCELLABLE_JOB_STATUSES (never LEASED work), and purge work routes through IdeaJobType.PURGE.

Durable provider recovery

Each queue envelope snapshots the complete active ProviderRecoveryProfileSnapshot, its matching attempt_limit, and an absolute deadline. Automatic retry retains that snapshot. A manual retry also preserves it unless the operator explicitly requests the active profile. Profile records are append-only and activation uses a revision-guarded pointer; changing the pointer affects future admissions only.

The initial profile is fixed at eight attempts, a 72-hour deadline, 30-second fallback doubled to a one-hour cap, a 24-hour maximum provider hint, deterministic 0-10% jitter, five unavailable failures in 60 seconds, a five- minute open circuit, one half-open probe, and blocked release batches of 100. Those are defaults for the first persisted profile, not module constants that silently mutate existing jobs.

retry_at_for_failure anchors delay and jitter to the failure's immutable observation time. Re-evaluating the same envelope attempt therefore returns the same not-before timestamp. A provider hint above the submitted 24-hour policy becomes BLOCKED; attempt/deadline exhaustion or permanent failure becomes DEAD_LETTERED; operator action becomes BLOCKED. Only a retryable failure within both budgets becomes RETRY_SCHEDULED.

UpstreamFailureEvidence retains bounded status/code/request identity, opaque quota scope, exact capacity, physical-attempt count, usage, and a completeness bit. Missing or USAGE_ONLY evidence on a physical call is billing unknown. Payment requires a reviewed billing-code/account-status source; HTTP 402, negative capacity, quota, and rate limiting are never sufficient by themselves.

Side effects if changes:

• Platform Mongo queue/profile/incident adapters and central indexes implement these exact field paths and compare-and-swap semantics;
• Data Platform job and commerce-operation APIs expose the same failure/policy facts through generated clients and translated frontend labels;
• finance records failed physical usage exactly once while pre-provider gate rejection remains zero spend;
• EnvelopeProcessResult must continue to contain exactly one result reference or failure snapshot.

Images are a critical asset

Content payloads treat visuals as first-class evidence: ProductContentPayload.image_artifacts carries model-ready product photos, CommerceProfileContentPayload.image_artifacts carries acquired creator or seller portraits, ShortVideoContentPayload.thumbnail_artifacts carries safe ranked-video thumbnails without transient playback URLs, and VideoContentPayload preserves uploaded video bytes plus key frames. IdeaSearchMedia.from_payload projects the representative image/video (with IdeaSearchSourceSummary for type-specific display facts) onto every search item so consuming platforms can render media-rich cards. Artifact references stay checksum-addressed (ArtifactRef) — delivery URLs are always resolved by the serving platform at read time.

Commerce discovery content

Trending commerce ingestion stays provider-neutral after the platform boundary:

profile = CommerceProfileContentPayload(
    profile_type=CommerceProfileType.CREATOR,
    display_name="Rocket Lab Notes",
    image_artifacts=[acquired_profile_image],
)

video = ShortVideoContentPayload(
    title="A compact model-rocket launch guide",
    thumbnail_artifacts=[acquired_thumbnail],
)

Products keep ProductContentPayload; creators and sellers share the typed profile payload with a stable profile_type; channel-wide ranked videos use a separate short-video payload. Provider names, provider response bags, volatile playback URLs, and raw enum labels do not enter these durable contracts.