HeavenBase

HeavenBase is an agent-native polyglot data engine for structured data management. You define one logical model, connect the storage systems you already use, and expose one workspace API to applications and agents.

It is not a database replacement. It is the layer that tells agents what data exists, what it means, where it lives, how to query it, and which tool surface they can use safely.

import heavenbase as hb

Release Status

This source tree is the prepared 0.1.2.0 release candidate. That is a repository-state claim, not a claim that 0.1.2.0 has been published to a package index. It has a working workspace facade, entity DSL, query routing and aggregation, system catalog rows, storage routing, backend handler registry, MCP workspace tools, LLM utilities, prompt storage, config management, interop helpers, a CLI, and the local hb dashboard workbench.

Expect a developer-facing API, useful local defaults, and visible diagnostics. Do not expect a fully stabilized production control plane yet: provider-native pushdown is still being deepened, some operations intentionally fall back to scan-style execution, and row-data snapshot export is still future work.

Why HeavenBase

Agents fail less often because they cannot reason, and more often because they cannot see the data landscape:

• they do not know a silo exists
• they have stale schema or source maps
• they miss domain vocabulary
• they retrieve only a convenient top-k slice
• they cannot relate data, playbooks, prompts, tools, and memory

HeavenBase addresses that by putting data, metadata, domain knowledge, prompts, tools, and agent memory into one typed workspace. Agents can discover Catalog rows for concrete objects, inspect MetaSchema rows for structure, and query typed entities through the same API.

What Ships Today

• HeavenBase workspaces for schemas, backends, routing, data, Catalog, MetaSchema, audit, repair, and MCP exposure.
• Entity classes with hb.field(...), logical types, defaults, compute hooks, query-compute hooks, JSON schema compilation, and stable object_id identity.
• Query surfaces through Python expressions and Mongo-style JSON filters, including strict Json dotted paths and registry-driven aggregation/HAVING, all lowering into QuerySpec.
• Field-level storage placement with strategies such as InlineColumn, JsonField, SideTable, VectorIndex, InvertedIndex, GraphEdge, and ExternalRef.
• Built-in backend families for in-memory, file JSON/pickle, SQL, vector stores, SQL-hosted sparse-GRAM keyword indexing, and search.
• Handler-based query compilation with QueryBuilder.explain() diagnostics for backend, strategy, handler mode, fallback, and unsupported reasons.
• Catalog for object discovery and MetaSchema for workspace structure.
• Workspace MCP profiles for agent and full/admin, plus extension-owned memory and memstate-style project memory tools.
• hb.LLM and hb llm for configured chat, embeddings, image generation, sessions, MCP tool loops, caching, and provider/gateway routing.
• hb dashboard for loopback-local config editing, read-only workspace inspection, and a streamed tool-using assistant with query evidence frames.
• CM_HVNB, ConfigEngine, prompt utilities, SQL helpers, interop helpers, and a small public capability index.

Primary API

The root heavenbase as hb facade is broad for notebooks and developer workflows. Start with this smaller surface for first-run code:

Use support exports such as hb.backends, hb.capabilities, hb.Database, compatibility interop helpers like hb.from_pandas(...), and config helpers when building integrations. Prefer Entity-owned import staging such as hb.Entity.from_pandas(...) in new examples. Use hb.ext for extension authorship, backend/provider registration, handlers, serializers, and profiles, not for first-user onboarding. See docs/resources/reference/public-api.md.

Architecture Philosophy

The short architecture model is: a workspace owns schemas and routing, and everything that varies is a registered piece. HeavenBase is designed for long-term composability more than one-off feature completeness.

For the concise design philosophy summary, see docs/resources/architecture/design-philosophy.md. For the canonical boundary map, read docs/resources/architecture/mental-model.md.

Install

For the latest published package, which may trail this source candidate:

pip install heavenbase
hb setup

To use or develop the prepared source candidate, uv is the recommended path:

git clone https://github.com/Magolor/HeavenBase.git
cd HeavenBase
uv sync --extra dev
uv run hb setup

The project pins the default local uv interpreter through .python-version while keeping package support for Python 3.10 through 3.13.

Optional extras are available when you need more providers:

# SQLAlchemy plus the default PostgreSQL, MySQL, and SQL Server DBAPI drivers
pip install "heavenbase[sql]"

# All maintained optional providers
pip install "heavenbase[full]"
uv sync --extra full

First Workspace

Use the debug preset first. It needs no Docker services and creates local SQLite plus in-memory vector/search backends. Short keyword arrays use SQL-hosted sparse-GRAM indexes by default.

import heavenbase as hb


class Product(hb.Entity):
    name = hb.field(hb.ShortText).desc("Display name")
    body = hb.field(hb.LongText).desc("Searchable description")
    price = hb.field(hb.Float).desc("List price")
    tags = hb.field(hb.Array[hb.ShortText]).default([])
    details = hb.field(hb.Json).default({})


ws = hb.HeavenBase("shop", preset="debug")
ws.register(Product)

desk_id = ws.upsert(
    Product,
    {
        "name": "Oak desk",
        "body": "Writing desk with cable tray",
        "price": 129.0,
        "tags": ["office", "furniture"],
        "details": {"material": "oak", "assembly": {"minutes": 25}},
    },
)

frame = (
    ws.query(Product)
    .where(Product.price < 150)
    .where(Product.tags.array_contains("office"))
    .select("name", "price")
    .execute()
)

print(desk_id)
print(frame.rows())

Every row has exactly one object_id. If you omit it and provide name, HeavenBase derives a deterministic ID from the entity schema and row name.

Query and Explain

Python expressions and JSON filters share the same routing path:

cheap = ws.query(Product).where(Product.price < 100).count()

json_frame = ws.query_json(
    Product,
    {
        "filter": {"body": {"$match": "desk"}, "details.material": "oak"},
        "select": ["name", "price"],
        "limit": 5,
    },
).execute()

summary = (
    ws.query(Product)
    .where(Product.details["material"] == "oak")
    .aggregate(items=Product.name.count(), average_price=Product.price.avg())
    .execute()
)

plan = ws.query(Product).where(Product.body.match("desk")).explain()

print(cheap)
print(json_frame.rows())
print(summary.rows())
print(plan["steps"][0]["handler_mode"])

Ordinary entity projection keeps object_id so agents can display compact rows and still call get, set, delete, query, or explain later with stable identity. Aggregate frames are intentionally identityless: their columns are grouping keys and named reducer outputs.

Agent Discovery

HeavenBase writes system rows automatically:

catalog_rows = ws.query(hb.Catalog).select("target_entity", "target_id", "name").execute().rows()
schema_rows = (
    ws.query(hb.MetaSchema)
    .where(hb.MetaSchema.kind == "field")
    .where(hb.MetaSchema.subject_id == Product.schema().entity_id)
    .select("field", "dtype", "desc")
    .execute()
    .rows()
)

print(catalog_rows)
print(schema_rows)

Use Catalog to find concrete objects. Use MetaSchema to learn entities, fields, storage placement, backends, capabilities, and extensions.

Workspace MCP

Any workspace can become an MCP toolkit:

print(ws.to_mcp_json(name="shop-mcp", profile="agent", transport="http", host="127.0.0.1", port=7001))
ws.serve(name="shop-mcp", profile="agent", transport="http", host="127.0.0.1", port=7001)

The agent profile exposes a compact schema/data surface: define_entity, list_entities, describe_entity, upsert, get, set, count, query, and explain. Use profile="full" for trusted administrative flows. Enable the optional memory extension and use ws.memory.to_mcp(profile="memory") for note-style memory or ws.memory.to_mcp(profile="memstate") for versioned project memory.

CLI

The hb command is a shallow layer over the same APIs:

hb --help
hb setup
hb dashboard
hb ws list
hb ws presets show debug
hb config get heavenbase.workspace.default_preset
hb cfg set heavenbase.llm.default_preset mock
hb llm chat --preset mock --provider mock --gateway mock --no-stream "hello"
hb llm embed "semantic text" --preview
hb prompt create demo.hello --template "Hello, {name}" --tr-key name
hb prompt render demo.hello --args '{"name":"Ada"}'

hb setup initializes global HeavenBase config and the default workspace. Runtime state lives under the HeavenBase config root, not in a new project-local folder.

Backends and Presets

Start with presets:

Use explicit backend maps when deployment placement matters:

ws = hb.HeavenBase(
    "custom-shop",
    backends={
        "main": {"type": "sqlite", "database": ":memory:"},
        "vec": {"type": "inmem"},
    },
)

Built-in backend families include inmem, json, pickle, sqlite, duckdb, postgres, pgvector, mysql, oceanbase, mssql, oracle, trino, starrocks, lance, chroma, milvus, pinecone, and elasticsearch. SQL backends host SparseGramIndex by default for short keyword arrays. Optional providers require their Python drivers and reachable services.

Inspect support from code:

hb.capabilities.backends(hb.Vector, op="near")
ws.capabilities.ops(hb.ShortText, hb.InlineColumn, backend="main")

LLM Utilities

hb.LLM resolves presets, model aliases, providers, gateways, request defaults, and cache settings from CM_HVNB.

import heavenbase as hb

llm = hb.LLM(preset="mock")
print(llm.chat("Reply with hb-ok"))

The default online provider is OpenRouter through an OpenAI-compatible gateway. Configure credentials through environment variables such as OPENROUTER_API_KEY, or switch to local/mock providers for tests and demos.

Environment Policy

Edit requirements*.txt first. pyproject.toml reads them through setuptools dynamic metadata; bash scripts/sync-env.bash refreshes uv.lock, poetry.lock, and environment-*.yml.

Use this install priority order:

1. uv — uv.lock + uv sync --all-extras after bash scripts/sync-env.bash (default sync installs runtime and all optional extras).
2. pip — pip install -r requirements.txt and pip install -e ".[dev]" / pip install -e ".[full]".
3. pyproject — pip install -e ".[dev]" when only package metadata is available.
4. conda — generated environment-*.yml with -e ".[<extra>]".
5. poetry — optional; poetry install after poetry.lock is refreshed by bash scripts/sync-env.bash.

CI should use bash scripts/sync-env.bash --check as the generated-file drift gate.

Development

Use the repo wrappers:

bash scripts/sync-env.bash
bash scripts/test.bash tests/config/test_config_spec.py::test_config_spec_is_recursively_immutable_and_behaviorally_equal -q
bash scripts/flake.bash -a

For the continuous benchmark suite:

bash scripts/benchmark.bash

External database tests are designed to skip or use the Docker stack when services are unavailable. For direct local service setup, use:

bash ./scripts/docker-restart.bash /d/databases/

Documentation and Work Queue

HeavenBase uses four explicit documentation surfaces:

Active resumable work exists only in docs/tasks.yaml. Architecture pages distinguish current behavior, accepted targets, remaining gaps, and non-goals; an accepted ADR is not a claim that its design has shipped.

Repository Map

Boundaries

• Capability registration does not guarantee provider-native pushdown. Check QueryBuilder.explain() for handler_mode, near_filter_mode, and fallback reasons.
• Multi-backend writes are coordinated by the workspace but not a distributed transaction system. Keep cross-backend invariants simple.
• File backends are local development tools. Treat pickle stores as trusted-local only.
• Workspace persistence is backend-driven. Workspace manifests replay construction config, enabled extensions, and user schemas; row data is not exported yet.

Documentation

• Engineering guide and authority map: docs/README.md
• Current goals and architecture status: docs/goals/current.md
• Roadmap: docs/goals/roadmap.md
• Active task queue: docs/tasks.yaml
• Development log: docs/DEVLOG.md
• Workspace presets: docs/resources/reference/workspace-presets.md
• Configuration: docs/resources/reference/config-spec.md
• Identity rules: docs/resources/reference/id-semantics.md
• MCP: docs/resources/reference/mcp.md
• LLM: docs/resources/reference/llm.md
• Capability matrix: docs/resources/reports/capabilities.md