sibyl-core

Core library for Sibyl. Domain models, graph operations, retrieval algorithms, the AI substrate, and tool implementations. Shared foundation for the API server and CLI.

Quick Reference

# Install
uv add sibyl-core

# Development
moon run core:lint        # Ruff check
moon run core:typecheck   # ty
moon run core:test        # Pytest

What's Here

• models/: Domain entities (Task, Project, Epic, Source, reflection, synthesis)
• backends/surreal/: SurrealDB driver, schema, and per-table operations
• retrieval/: Native context-pack retrieval, query planning, refinement, reranking, hybrid retrieval, fusion, dedup
• memory_pipeline/: Canonical memory-pipeline contracts and policies (capture, lifecycle, quality, retrieval)
• projection/: Projection helpers for native memory graph enrichment
• audit/: Audit event helpers shared by Sibyl runtimes
• ai/: Native LLM substrate, model registry, providers, validation
• embeddings/: Embedding provider clients
• services/: Memory loop, reflection, synthesis, autonomy, source adapters, and the EntityManager / RelationshipManager graph managers
• tools/: MCP tool implementations
• tasks/: Workflow engine and dependency resolution
• migrate/: Migration archive merge and rewrite logic
• auth/: JWT primitives and password hashing

Structure

src/sibyl_core/
├── models/
│   ├── entities.py       # Entity, EntityType, base classes
│   ├── tasks.py          # Task, Project, Epic, Milestone
│   ├── sources.py        # Source, Document
│   ├── context.py        # Context-pack models
│   ├── reflection.py     # Reflection candidate models
│   ├── synthesis.py      # Synthesis plan and artifact models
│   └── responses.py      # API response models
├── backends/surreal/     # Driver, schema, table operations
├── retrieval/            # Native context retrieval, fusion (RRF), dedup
│   ├── query_planning.py # Structured query planning for the accurate retrieval lane
│   ├── refinement.py     # Deterministic feedback queries for iterative retrieval
│   ├── reranking.py      # Cross-encoder reranking of query-document pairs
│   ├── hybrid.py         # Hybrid retrieval combining vector search and graph traversal
│   └── operational_evidence.py # Composition of raw and distilled operational evidence
├── memory_pipeline/      # Memory-pipeline contracts: capture, lifecycle, quality, retrieval
├── projection/           # Projection helpers for native memory graph enrichment
├── audit/                # Audit event helpers shared by runtimes
├── ai/
│   ├── registry.py       # Curated LLM/embedding model registry
│   ├── providers.py      # PydanticAI provider model factory
│   ├── clients.py        # Scoped agent caching
│   └── llm/              # Extractor, Generator, config sources
├── embeddings/           # Embedding provider clients
├── services/
│   ├── graph.py          # EntityManager, RelationshipManager
│   ├── graph_client.py   # SurrealGraphClient driver wrapper
│   └── ...               # Memory loop, reflection, synthesis, source adapters
├── tools/                # MCP tool implementations
└── tasks/                # Workflow state machine, dependency resolution

Usage

Models

from sibyl_core.models import (
    Entity,
    EntityType,
    Task,
    TaskStatus,
    Project,
    Epic,
)

task = Task(
    name="Implement OAuth",
    content="Add Google and GitHub OAuth",
    project_id="proj_abc",
    status=TaskStatus.TODO,
)

Graph Client

from sibyl_core.services import get_graph_client
from sibyl_core.services.graph import EntityManager

client = await get_graph_client(group_id=str(org_id))
manager = EntityManager(client, group_id=str(org_id))

# CRUD
await manager.create(entity)
# Retrieval uses search or list_by_type rather than direct ID lookup
results = await manager.search(query="authentication patterns", limit=20)

Write Concurrency

The SurrealDB driver serializes WebSocket operations per client, and org-scoped graph access should use a per-org client (get_graph_client(group_id=...) returns one scoped to the org namespace).

# Native write path, no LLM extraction
await manager.create_direct(entity)

# Compatibility path with LLM-backed extraction
await manager.create(entity)

Task Workflow

from sibyl_core.tasks import TaskManager

manager = TaskManager(entity_manager, relationship_manager)
await manager.create_task_with_knowledge_links(task)
await manager.find_similar_tasks(task)
await manager.estimate_task_effort(task)

AI Substrate

from pydantic import BaseModel

from sibyl_core.ai import Extractor, Generator, LLMSurface


class ExtractedFact(BaseModel):
    name: str
    summary: str


extractor = Extractor(ExtractedFact, surface=LLMSurface.CRAWLER)
fact = await extractor.extract("Extract one fact from this document chunk.")

generator = Generator(surface=LLMSurface.SYNTHESIS)
draft = await generator.generate("Summarize this context pack.", max_tokens=512)

The substrate uses PydanticAI under sibyl_core.ai, with provider API keys passed through provider objects rather than mutating os.environ. Extractor[T] handles structured output and classified LLM errors. Generator handles text generation and streaming. Surface-specific config is resolved through an LLMConfigSource so the API can supply database-backed settings while core stays pure.

Entity Types

Sibyl models 33 entity types so memory stays structured. The registry lives in models/entities.py and covers, among others:

• Work: task, epic, project, milestone, team
• Knowledge: pattern, episode, procedure, rule, guide, template, error_pattern, tool, language, topic
• Memory: decision, plan, idea, claim, artifact, session, note, preference
• People & places: person, place, event
• Sources: source, document, domain, community, knowledge_source, config_file, slash_command

Relationship Types

from sibyl_core.models import RelationshipType

# Knowledge
RelationshipType.APPLIES_TO, REQUIRES, CONFLICTS_WITH, SUPERSEDES

# Task
RelationshipType.BELONGS_TO, DEPENDS_ON, BLOCKS, REFERENCES

Predicates a writing agent can declare

Most relationship types are minted by the system. Five are declarable by the agent doing the write, on the related_to channel of add(), the MCP add and remember tools, and POST /entities. Prefix a target id with the predicate: related_to=["supersedes:ep_0a1b"]. The memory being written is always the subject, so that entry reads "this new memory supersedes ep_0a1b" and mints new -SUPERSEDES-> ep_0a1b, which is the direction graph expansion walks.

A bare id still creates an untyped RELATED_TO edge, and any prefix outside this closed set is read as part of the id rather than rejected. Full semantics, direction rationale, and the rejected candidates live in sibyl_core/models/relations.py.

Configuration

SIBYL_LLM_PROVIDER=anthropic          # anthropic | openai | gemini
SIBYL_LLM_MODEL=claude-haiku-4-5
SIBYL_LLM_TEMPERATURE=0
SIBYL_LLM_MAX_TOKENS=2048

# Per-attempt read timeout. A shared value wins over every surface default, so
# setting this also shortens the memory surface, which waits 600s by default
# because consolidation sends a whole cohort in one non-streaming request.
SIBYL_LLM_TIMEOUT_SECONDS=60
SIBYL_LLM_MEMORY_TIMEOUT_SECONDS=600

# Surface-specific values override shared LLM values.
SIBYL_LLM_CRAWLER_PROVIDER=gemini
SIBYL_LLM_CRAWLER_MODEL=gemini-3-1-flash-lite
SIBYL_LLM_SYNTHESIS_PROVIDER=anthropic
SIBYL_LLM_SYNTHESIS_MODEL=claude-sonnet-4-6

SIBYL_ANTHROPIC_API_KEY=...           # LLM provider key
SIBYL_OPENAI_API_KEY=sk-...           # LLM or embedding provider key
SIBYL_GEMINI_API_KEY=...              # LLM or embedding provider key

SIBYL_EMBEDDING_PROVIDER=openai       # openai | gemini
SIBYL_EMBEDDING_MODEL=text-embedding-3-small
SIBYL_EMBEDDING_DIMENSIONS=1536
SIBYL_GRAPH_EMBEDDING_PROVIDER=openai
SIBYL_GRAPH_EMBEDDING_MODEL=text-embedding-3-small
SIBYL_GRAPH_EMBEDDING_DIMENSIONS=1024

LLM settings are instance-wide. Environment variables win over database settings and mark individual fields as locked.

Gemini keys can also come from GEMINI_API_KEY or GOOGLE_API_KEY. Changing embedding provider, model, or dimensions requires re-embedding existing graph and document vectors before comparing old and new search results.

To add a first-class LLM provider, add a provider factory branch in sibyl_core.ai.providers, add registry entries in sibyl_core.ai.registry, extend LLMProviderName and the API DTOs, and add a live probe to scripts/llm/verify_registry.py.

Key Patterns

Multi-tenancy: Every operation requires org context.

manager = EntityManager(client, group_id=str(org.id))

Node shapes: Native retrieval queries direct Surreal records. Archive compatibility keeps old Episodic/Entity records readable without Graphiti.

SELECT * FROM entity WHERE entity_type = $type;

Creation paths: direct native writes first, LLM-backed extraction when explicitly needed.

await manager.create_direct(entity)  # Native write path, no LLM
await manager.create(entity)  # Compatibility extraction path

Legacy Compatibility

Legacy Graphiti-shaped records remain readable through Sibyl-owned Surreal projection and archive code. The package no longer exposes a Graphiti compatibility extra or installs the Graphiti Core package.

Testing

# With mock LLM (fast, deterministic)
SIBYL_MOCK_LLM=true uv run pytest tests/

# Live model tests (costs money)
uv run pytest tests/live --live-models

# Retrieval benchmark suite
moon run core:bench-retrieval

# Live read-only search benchmark against a running stack
moon run core:bench-live

# Live context-pack smoke benchmark
moon run core:bench-context

core:bench-live probes the real /api/search path with CLI auth. core:bench-context probes /api/context/pack. Both benchmarks are read-only. Saved reports can be compared with uv run python benchmarks/compare_eval_reports.py <baseline.json> <candidate.json>.

Completed Validation Receipt Recovery

Validation writes an encrypted completed-result receipt before committing its result to SurrealDB. Set SIBYL_VALIDATION_RECEIPT_DIR to persistent private storage (default: ~/.sibyl/validation-receipts). Quickstart shares its existing server-state volume between API and worker; production Compose mounts a shared receipt volume. Helm deployments must provision a claim and set backend.validationReceipts.existingClaim; use ReadWriteMany storage when replicas run on different nodes. Keep the same directory available after process or container restart. A different replica without that storage refuses incomplete replay and cannot recover the receipt.

Each receipt is encrypted with a per-execution key stored in the private content ledger. Source purge erases the key in the existing purge transaction. Recovery requires the original canonical request and current authorization, preserves terminal history, and runs the existing source/publication fences. Recovery never calls the model again. Files are removed only after database result retention. Back up the journal together with the content database when pending receipts must survive host loss; a database-only backup cannot recover a pending local receipt.

The readiness probe checks journal write access before dispatch. If the journal fails after a provider returns, database result persistence can still preserve the receipt. Simultaneous loss of both stores, or a crash between provider completion and receipt fsync, leaves the pre-dispatch physical attempt explicitly unknown. The system does not report that interval as zero cost or automatically redispatch.