Sibyl API Server

sibyld is the FastAPI + MCP SDK server behind Sibyl's knowledge graph, agent memory loop, task workflow, search, synthesis, and real-time updates.

Quick Reference

# Install the embedded daemon without the web UI
curl -fsSL https://raw.githubusercontent.com/hyperb1iss/sibyl/main/install.sh | sh -s -- --daemon

# Start server from the monorepo
moon run api:serve        # or: uv run sibyld serve

# Start worker (Redis coordination only)
moon run api:worker       # or: uv run sibyld worker

# Quality checks
moon run api:test         # Run tests
moon run api:lint         # Lint
moon run api:typecheck    # Type check

What's Here

• MCP Server: thirteen tools for search, context packs, exploration, bounded traversal, capture, memory, synthesis, and management
• REST API: 31 routers covering entities, tasks, teams, projects, experience, memory, synthesis, sources, auth, settings, and admin
• Auth System: JWT sessions, GitHub OAuth, OIDC enterprise SSO (see docs/admin/), API keys with scopes, MCP OAuth clients, RBAC, SMTP-backed password reset
• Background Jobs: in-process local runtime or Redis-backed arq workers, including the nightly reflection dream-cycle
• WebSocket: Real-time updates for entities and tasks

Architecture

Sibyl API (port 3334)
├── /api/*              → FastAPI REST endpoints
├── /api/openapi.json   → OpenAPI schema
├── /mcp                → MCP server (streamable-http, 13 tools)
├── /api/ws             → WebSocket for real-time updates
└── Lifespan            → Background jobs + coordination broker

Key Directories

Configuration

Required:

SIBYL_JWT_SECRET=...              # Auth (required in production; dev auto-generates)
SIBYL_ANTHROPIC_API_KEY=...       # Required when LLM provider=anthropic
# SIBYL_OPENAI_API_KEY=sk-...     # Required when LLM provider=openai
# SIBYL_GEMINI_API_KEY=...        # Required when LLM provider=gemini
# LLM provider=bedrock needs no key: AWS credentials plus AWS_REGION

# Embeddings: choose OpenAI, Gemini or Amazon Bedrock (Cohere Embed v4)
SIBYL_EMBEDDING_PROVIDER=openai   # openai | gemini | bedrock
SIBYL_OPENAI_API_KEY=sk-...       # Required when embedding provider=openai
# SIBYL_GEMINI_API_KEY=...        # Required when embedding provider=gemini

Optional:

SIBYL_STORE=surreal                   # the only supported value
SIBYL_COORDINATION_BACKEND=auto       # auto | local | redis
SIBYL_SURREAL_URL=ws://127.0.0.1:8000/rpc
SIBYL_SURREAL_USERNAME=root
SIBYL_SURREAL_PASSWORD=root
SIBYL_REDIS_HOST=127.0.0.1            # only needed for Redis coordination
SIBYL_REDIS_PORT=6381
SIBYL_LLM_PROVIDER=anthropic          # anthropic | bedrock | openai | gemini
SIBYL_LLM_MODEL=claude-haiku-4-5
SIBYL_LLM_CRAWLER_MODEL=claude-haiku-4-5
SIBYL_LLM_SYNTHESIS_MODEL=claude-sonnet-4-6
SIBYL_LLM_TEMPERATURE=0
# A shared timeout wins over every surface default, so the memory surface needs
# its own value; consolidation sends a whole cohort in one request.
SIBYL_LLM_TIMEOUT_SECONDS=60
SIBYL_LLM_MEMORY_TIMEOUT_SECONDS=600
SIBYL_EMBEDDING_MODEL=text-embedding-3-small
SIBYL_EMBEDDING_DIMENSIONS=1536
SIBYL_GRAPH_EMBEDDING_PROVIDER=openai
SIBYL_GRAPH_EMBEDDING_MODEL=text-embedding-3-small
SIBYL_GRAPH_EMBEDDING_DIMENSIONS=1024
# Amazon Bedrock (provider=bedrock) signs with the AWS credential chain
SIBYL_BEDROCK_REGION=us-west-2        # falls back to AWS_REGION
SIBYL_BEDROCK_INFERENCE_SCOPE=us      # us | global | regional

Gemini keys can also be supplied through GEMINI_API_KEY or GOOGLE_API_KEY. Changing embedding provider, model, or dimensions changes vector spaces; re-crawl sources and rebuild graph indexes before mixing old and new search results.

LLM settings are instance-wide. Environment variables win over database settings field by field; env-backed fields return 409 LOCKED_BY_ENV on write. Database settings are managed under:

GET  /api/settings/ai/llm
PUT  /api/settings/ai/llm/{surface}
POST /api/settings/ai/llm/{surface}/test
POST /api/settings/ai/keys/{provider}/test
POST /api/settings/ai/models/{model_alias}/test
GET  /api/settings/ai/registry?kind=llm

Crawler extraction and synthesis generation call sibyl_core.ai rather than provider SDKs directly. Custom model IDs are accepted as database settings with an unverified_model warning; validate them with the model test endpoint before using them in production flows.

CLI Commands

sibyld serve              # Start the HTTP server
sibyld serve -t stdio     # Start a stdio server (MCP subprocess mode)
sibyld worker             # Start the job worker (local mode exits cleanly)
sibyld up                 # Start data services + API
sibyld down               # Stop all services
sibyld db backup          # Back up the graph database
sibyld migrate import ... # Import a migration archive
sibyld generate realistic # Generate sample data

Runtime Modes

For single-machine Surreal development, run sibyld serve or sibyld up with SIBYL_STORE=surreal. The default coordination_backend=auto resolves to local, so background jobs, pending state, locks, pub/sub, and schedules all stay in-process with no Redis requirement.

Redis remains available for distributed or multi-process dev. Set SIBYL_COORDINATION_BACKEND=redis when you want the arq worker model, then run sibyld worker or moon run api:worker separately.

Archive Checks

The authenticated archive API records an immutable check before migration applies data:

Send the check as multipart/form-data with exactly two parts: archive contains the exported archive bytes, and options contains a JSON object. The options use this shape (replace the owner and actor placeholders with the source owner and authenticated destination actor):

{
  "mappings": {
    "source_private_owner_id": "<source-owner>",
    "projects": {},
    "teams": {},
    "quarantine": {
      "memory_scope": "private",
      "scope_key": "<authenticated-actor>"
    }
  },
  "conflict_policy": "additive"
}

Project and team mappings bind source labels to current writable destinations. Eligible private rows belong to the authenticated actor. Protected or retired rows and their dependents are quarantined. Current membership and credential restrictions gate each destination.

An optional Idempotency-Key header binds the request to the actor and organization. Retrying the same request preserves its original plan and credential ceiling. A changed request or originating API key returns 409. Invalid archives return 422; requests exceeding configured budgets return 413. Successful responses contain the run identity, checked status, digests and per-kind counts. The status endpoint accepts any currently valid same-actor session or API key with read access, even after the originating key is revoked. Status reads do not grant permission to replay or apply.

The check stores an inert archive artifact and plan in one native transaction. Applying data and the client migration command are subsequent protocol layers. Resource settings use the SIBYL_ARCHIVE_IMPORT_ prefix; the complete metadata request defaults to 4 MiB for HTTP and 64 MiB for WebSocket or embedded transport. Set SIBYL_ARCHIVE_IMPORT_METADATA_TRANSACTION_BYTES only to a budget supported by the configured native server.

Key Patterns

Multi-tenancy: Every operation requires org context.

manager = EntityManager(client, group_id=str(org.id))

Write concurrency: the SurrealDB driver serializes WebSocket operations per client. Clone graph drivers per organization rather than sharing one driver across org scopes.

SurrealDB access model: The API server, worker, CLI, and schema bootstrap flows use configured SurrealDB system credentials (SIBYL_SURREAL_USERNAME / SIBYL_SURREAL_PASSWORD) so they can run migrations, background jobs, and admin workflows. Route code must keep explicit org, project, and principal predicates because system users sit above table-level permissions.

Auth and content schema migrations also define table permissions for future scoped Surreal record users. Tenant-owned tables accept rows where organization_id (or organizations.uuid) matches either $token.org from an external JWT access method or $auth.organization_id from a Surreal record session. Secret-heavy and global tables, such as API keys, sessions, OAuth tokens, system settings, and telemetry rollups, remain PERMISSIONS NONE for direct scoped DB access.

Request context: Auth middleware injects user and org.

from sibyl.auth.dependencies import get_current_user, get_current_organization

Dependencies

Depends on sibyl-core for models, graph client, AI substrate, and tool implementations.