fastdb

FastDB is an embedded C++ data layer for scientific computing and RPC payload work. It stores typed record tables and object graphs in a compact binary format, and defines fastdb.payload.v1, a portable payload format with canonical identity (RFC 8785 canonical JSON plus SHA-256) and a deterministic binary layout, so the same data means the same bytes in every language.

FastDB is an in-process library, not a database server: there is no network protocol, query planner, or SQL surface. One C++ core owns portable-payload schema compilation, canonicalization, binary layout, and lifetime semantics. The Rust, Python, and TypeScript/WASM portable APIs project that core through a stable C ABI.

What you can do with it

• Typed record tables — declare plain classes with typed fields (U32, F64, STR, ...), fill fixed-size tables in bulk, and read them back row-wise or column-wise. In Python, numeric columns are exposed as NumPy arrays over native storage.
• Object graphs — features can reference other features across tables, including shared and cyclic references, with an explicit ownership/invalidation model for views over native memory.
• Portable payload exchange — compile a payload specification once and get canonical identity, validated binary payloads, checked views, materialization, and stable structured errors, identically in C++, Rust, Python, and TypeScript/WASM. The core can also generate ready-to-use payload code for all four languages.
• Compact binary persistence — save and load databases as files or buffers; the Python binding additionally supports shared-memory handoff between processes.

Packages

The Rust, Python, and TypeScript portable APIs are thin projections of one core; none contains an independent payload parser, layout, or digest implementation (ADR-0001).

Installation

Python (from PyPI):

uv pip install fastdb4py   # or: pip install fastdb4py

TypeScript (from npm):

npm install fastdb4ts

Rust — the safe fastdb crate links against a native Core bundle; it is not a pure-Rust crate. Download the Core bundle matching your crate version from the GitHub release page, then set FASTDB_PAYLOAD_LINK_MODE=system and set FASTDB_PAYLOAD_SYSTEM_LIB_DIR to the absolute extracted lib directory. See the linking contract for details.

C++ — use a Core bundle from a tagged release, or build from source with CMake (see fastcarto/README.md).

The 0.2.2 release record lists Python wheels for macOS arm64/x86_64, manylinux_2_28 x86_64/aarch64 and Windows amd64 on CPython 3.12, 3.13, 3.14 and free-threaded 3.14. Native Core bundles cover Linux x86_64, macOS arm64 and Windows x64. Remaining platform work is tracked in Issue 0001. For platforms without a published wheel or Core bundle, Python builds from source need a C++17 compiler, CMake, SWIG, and NumPy.

Quick start (Python)

import numpy as np
from fastdb4py import RecordEngine, Layout, feature, F64, U32, STR


@feature
class Sample:
    station: U32
    temperature: F64
    label: STR


db = RecordEngine.truncate([Layout(Sample, 3)])
table = db.table(Sample)

table.fill(
    station=np.array([101, 102, 103], dtype=np.uint32),
    temperature=np.array([12.5, 14.8, 9.3]),
    label=["north", "south", "ridge"],
)

# Row readback: fields read straight from native storage.
print(table[1].station, table[1].temperature, table[1].label)
# 102 14.8 south

# Column readback: numeric columns are NumPy arrays over native storage,
# string columns are StringColumn wrappers.
print(table.column.temperature.mean())
print(table.column.label.to_pylist())

# Compact binary persistence.
db.save("samples")
reloaded = RecordEngine.load("samples", from_file=True)
print(reloaded.table(Sample).column.temperature[:])

The Python binding also offers ObjectEngine for dynamically growing tables and reference-heavy graphs, and fdb codegen for generating payload code from a portable specification. See python/README.md for the full API.

Portable payloads and standalone engines

FastDB exposes two complementary APIs:

• The portable payload API (fastdb4py.payload, Rust fastdb, fastdb4ts/payload, C++ fastdb_payload.h/fastdb_payload.hpp) is the cross-language exchange path. A specification is compiled once by the core into an immutable CompiledSpec with a SHA-256 identity; payloads built from it have one validated binary meaning (fastdb.payload.bin.v1) across languages, with checked views, invalidation, detached materialization, and deterministic C++/Rust/Python/TypeScript code generation.
• The standalone engines are the in-process storage paths. RecordEngine manages fixed-size tables of typed records (AoS layout with strided field access), and ObjectEngine manages dynamically growing tables and typed object graphs. These are local libraries for embedding, sharing memory between processes, and file persistence — not services.

RecordEngine/ObjectEngine data and portable payloads are separate subsystems today; see the accepted design for the target scope and Issue 0001 for intentionally deferred capabilities.

Ownership and view lifetimes

Values that stay in Python (@feature instances, materialized copies) are ordinary owned objects. Views into native storage (table rows, numeric and string columns) are tied to a FdbViewOwner: pass FdbViewOwner(checked=True) when integrating with reusable memory leases, call fdb.invalidate(owner_or_view) when the lease ends so later checked access fails loudly, and call fdb.materialize(value) or value.to_owned() before retaining data beyond the backing buffer's lifetime. Standalone tables stay trusted by default and return raw NumPy column views for speed; unsafe_numpy_view() is an explicit escape hatch whose raw arrays cannot be revoked after export. See Backed View Lifetimes.

Documentation

• Python binding (fastdb4py): python/README.md
• TypeScript/WASM binding (fastdb4ts): ts/fastdb4ts/README.md
• Rust crates (fastdb, fastdb-sys): bindings/rust/fastdb/README.md
• C++ core (fastcarto/fastdb): fastcarto/README.md
• Accepted design: portable payload foundation
• Decisions and status: docs/decisions/, implementation status
• Releases: 0.2.2 record, 0.2.1 record, 0.2.0 record, CHANGELOG.md

Development

Build and test the layer you are changing (from the repository root):

uv run pytest tests/python -q      # Python binding tests
uv build                           # fastdb4py sdist + wheel
bash ts/build-wasm.sh              # WebAssembly module for fastdb4ts
npm --prefix ts/fastdb4ts run build
npm run test:ts                    # TypeScript tests

Layer requirements: Python binding — C++17 compiler, CMake ≥ 3.16, SWIG ≥ 4.4, NumPy; TypeScript/WASM — Emscripten, Node.js, npm; core — C++17 compiler and CMake. When C++ binary or ABI behavior changes, revalidate the C++, Rust, Python, and TypeScript consumers together. See fastcarto/README.md for native internals and AGENTS.md for repository conventions.

License

MIT. Distribution archives retain the notices in THIRD_PARTY_NOTICES.txt.