Skip to main content

Foldlite DB

Immutable documents. Rebuildable live views. Rust and WebAssembly.

Foldlite DB is a CouchDB-inspired map/reduce engine built around a simple rule: keep immutable documents, derive everything else. The Rust engine appends accepted document batches to a durable log. Indexes, source checkpoints, group states, and change history form a disposable cache that can be reconstructed from that log. There is one native implementation, exposed to Python; storage needs no database server or external database library.

Write a view in AssemblyScript and compile it to WebAssembly. The same module maps and reduces documents in Rust or entirely in a browser worker. Try the browser playground or read the view authoring guide.

Install

Use Python 3.11+ to install the 0.1.0 release:

python -m pip install foldlitedb==0.1.0

Native wheels are provided for Linux x86-64 and ARM64, macOS Intel and Apple Silicon, and Windows x86-64. Installing a matching wheel needs neither Rust nor Node. The hosted documentation describes the API and bundled portable views.

Build from this checkout

Building from source requires Python 3.11+, Rust/Cargo 1.90+, and a platform C compiler and linker. From the repository root:

python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install .
python examples/portable_metrics.py

On Windows, activate with .venv\Scripts\activate and use an MSVC build environment. Node.js is unnecessary for installation and bundled views. To make a wheel, run python -m pip wheel . --no-deps --wheel-dir dist. Install that wheel on a matching platform with python -m pip install /path/to/the-built.whl; installing a built wheel needs neither Rust nor Node. See the release guide for PyPI Trusted Publisher setup and the 0.1.0 publication workflow.

Append documents, read a live view

Start with a fresh metrics.fold file:

from foldlitedb import Database, View

view = View.bundled("metrics")
with Database("metrics.fold", view) as db:
    documents = db.add_many([
        {"metric": "loss", "value": 4},
        {"metric": "loss", "value": 8},
    ])
    assert list(db.rows())[0].result == {
        "count": 2, "sum": 12, "min": 4, "max": 8, "mean": 6,
    }
    snapshot = db.query()
    documents += (db.add({"metric": "loss", "value": 3}),)
    assert db.changes(since=snapshot.cursor).rows[0].result["mean"] == 5
    retained = tuple(db.documents())
    db.checkpoint()

Rust executes both map and reduce through a bounded Wasm interpreter. metrics groups finite numbers by a string metric; catalogue demonstrates an ordered fold over structured events. Each database holds one view module. Database and View are the primary Python names; PortableDatabase and PortableView are aliases for the same types.

Try the telemetry collector for bulk import, retries, and a persistent changes consumer, or the catalogue example for an ordered item lifecycle. If a payload produces no rows, inspect its map output with db.preview(payload).

Every unnamed append creates a new observation. Give an observation an ID when identical retries should count once. IDs and positions are scoped to a source; conflicting reuse raises an error. Documents cannot be updated or deleted. Append new observations to represent changes, then let the view fold their history.

add_many commits one atomic batch of at most 1024 documents. Its complete Wasm map request must fit 256 KiB, so larger payloads need smaller batches. rows() follows a stable snapshot in bounded pages; query() and changes(since=...) expose continuation cursors. See the Python API for signatures and limits.

A log you keep; a cache you can discard

For a persistent database, metrics.fold contains the immutable document log. metrics.fold.cache is an optional checkpoint; metrics.fold.lock coordinates one active owner. A batch is acknowledged after its log frame is synced. The engine validates and computes a batch before appending it, so a map or reducer failure does not commit a prefix.

checkpoint() writes an atomic cache snapshot; closing cleanly does the same. On reopen, the engine verifies the log and matching cache prefix, then runs Wasm only for batches after the checkpoint. Missing, corrupt, or incompatible caches are rebuilt from the documents. rebuild() explicitly recomputes the view and starts a new cursor generation. Neither operation changes the source documents.

The document log is independent of the view module. Opening it with different module bytes rebuilds the cache in place; retain the exact module if you need to reproduce its previous results. Close the database before backing up the log and view artifacts; Windows locks prevent other handles from reading an active log. The cache is replaceable; the only copy of a document log is not.

Rebuild uses the original batch order. It scans the log linearly and runs each uncached batch through the same map/reduce path as live ingestion. A valid cache avoids repeated Wasm work, but reopening still verifies the complete log. Current indexes and history grow in memory with the data. Cache snapshots are limited to 1 GiB of encoded checkpoint data; a checkpoint or close error reports a failure to save that accelerator, while committed log data remains intact. There is no claim of constant memory or measured rebuild throughput.

To copy retained records into a fresh database:

with Database("copy.fold", view) as copied:
    copied.ingest(retained)
    assert list(copied.rows())[0].result["mean"] == 5
    before = [(r.key, r.state, r.result) for r in copied.rows()]
    copied.rebuild()
    assert [(r.key, r.state, r.result) for r in copied.rows()] == before

Use ingest_many(records, batch_size=100) for larger archives. Iterate its results to perform the import: each yielded result commits one separate bounded batch, and earlier batches remain committed if a later batch fails. See the retry-safe source import for constructing records with upstream positions or stable IDs alone. Preserve record order; preserve original batch boundaries too if you need identical engine sequence values.

Author a view, run it on a website

The AssemblyScript SDK provides JSON helpers and the Wasm ABI. Implement map, identity, add, and finalize; implement merge only for states that can be combined correctly. Inputs and state are explicit, and views must be deterministic. An ordered fold must give equivalent state when the same contributions are split across successive batches.

Rebuild the two bundled examples with Node.js 20+:

npm ci --prefix views
npm run build --prefix views
npm test --prefix views
npm run check --prefix views

The build writes modules and SHA-256 manifests into src/foldlitedb/portable_assets/. It builds the named metrics and catalogue examples. Compile a custom view and generate its manifest with:

npm run build:view --prefix views -- assembly/examples/my-view.ts --out-dir .build

Both paths are relative to views/. The authoring guide includes a complete custom view and the build command to generate its Wasm and digest manifest.

foldlitedb.portable_assets() exposes compiled modules plus client.js, worker.js, and host.js. Copy them to a static site and import BrowserDatabase from client.js. The browser example covers appends, snapshots, changes, export, and replay verification. JavaScript owns in-memory grouping and storage in the browser worker; the Rust log engine is not compiled into this browser build.

Browser state is lost on reload or close. Export resolved documents first. The preview bounds documents, groups, history, and request sizes. Persistent browser storage, log/history compaction, and multiple views per native database remain future work. JSON numbers use IEEE-754 double precision; integers a double cannot represent exactly are rejected, so use strings for exact arbitrary-precision identifiers.

Development and documentation

python -m pip install '.[dev,browser]'
python -m playwright install chromium
python -m pytest
python -m pytest tests/browser
cargo test --locked --manifest-path native/Cargo.toml -p foldlitedb-portable-engine
npm ci --prefix views
npm test --prefix views
npm run check --prefix views
python scripts/build_docs.py
python scripts/check_docs.py --execute --browser
python -m http.server 8000 --directory _site

The docs checker executes annotated Python examples in fresh temporary directories and checks links, search, and desktop/mobile rendering in Chromium. Open http://localhost:8000 for the built docs and playground. An authorized maintainer can publish checked generated docs with python scripts/publish_docs.py; building alone does not publish them.

Measure the native path on your target filesystem with:

python scripts/benchmark_database.py --documents 10000 --keys 16 --batch-size 100 --live-documents 20 --output native-benchmark.json

Use --storage-root /path/to/filesystem to select the filesystem being measured. The report compares durable ingestion, checkpointing, cached reopen, suffix replay, and full reconstruction, while checking unchanged log bytes and matching results. Local measurements describe that fixture and host.

See DESIGN.md for durability, replay, cache validation, and execution contracts.

Metadata

Release files for foldlitedb 0.1.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for foldlitedb 0.1.4
File Size Uploaded Provenance Actions
foldlitedb-0.1.4.tar.gz 368.5 kB With provenance Details for foldlitedb-0.1.4.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for foldlitedb 0.1.4
File Actions
foldlitedb-0.1.4-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for foldlitedb-0.1.4-cp311-abi3-win_amd64.whl
foldlitedb-0.1.4-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.11 abi3 Linux glibc 2.17+ x86-64 Details for foldlitedb-0.1.4-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
foldlitedb-0.1.4-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.11 abi3 Linux glibc 2.17+ ARM64 Details for foldlitedb-0.1.4-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
foldlitedb-0.1.4-cp311-abi3-macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 11.0+ ARM64 Details for foldlitedb-0.1.4-cp311-abi3-macosx_11_0_arm64.whl
foldlitedb-0.1.4-cp311-abi3-macosx_10_12_x86_64.whl CPython 3.11 abi3 macOS 10.12+ x86-64 Details for foldlitedb-0.1.4-cp311-abi3-macosx_10_12_x86_64.whl

Total release size: 7.2 MB

Release files / foldlitedb-0.1.4.tar.gz

Download URL foldlitedb-0.1.4.tar.gz
Size 368.5 kB
Tags Source
SHA-256 checksum
How to use checksums
53104efa7510e818f0a8f1cc249a532a53443f0ce161f21e2071c280dd77a21b
BLAKE2b-256 checksum
How to use checksums
229152fe1399a977a6f2e31d8b381dc54a7e1420ecebcd68b79240e13a67a3a0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Release files / foldlitedb-0.1.4-cp311-abi3-win_amd64.whl

Download URL foldlitedb-0.1.4-cp311-abi3-win_amd64.whl
Size 1.2 MB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
87f49c33d17594c758a539037bc1692e6c6951ed3f462a86916e17582e7a521d
BLAKE2b-256 checksum
How to use checksums
58e77521fa1d9b90ae8ba1f3bd947d38281bb4d8a7e05855ffb3089e678749f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Release files / foldlitedb-0.1.4-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL foldlitedb-0.1.4-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 1.5 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
b8c0684475d93b03d876e01a0e90310303fd5a25e0bdd3064fcd46614981e9ca
BLAKE2b-256 checksum
How to use checksums
399b23a6e94976d15219a30f5f9182a4e447a294b39cd7efb359e3da28ab67c3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Release files / foldlitedb-0.1.4-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL foldlitedb-0.1.4-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 1.4 MB
Tags CPython 3.11 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
66efcdc4a551c41b0caa529c02f61ad5df7e77795592527d0ab55a8d1805a757
BLAKE2b-256 checksum
How to use checksums
bba2bc2c3a827f3c73c36823256efb61c6598898bddf4922035c3e9ae0ac935e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Release files / foldlitedb-0.1.4-cp311-abi3-macosx_11_0_arm64.whl

Download URL foldlitedb-0.1.4-cp311-abi3-macosx_11_0_arm64.whl
Size 1.3 MB
Tags CPython 3.11 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
8c488792f384266198ffa6260d38ceb24255c2167756b9abfbf5cbd09080bafa
BLAKE2b-256 checksum
How to use checksums
21ce7e4de71f3c7ecb21dd1d1e395f95ece7cdaf34fbee15a85471b3e32f9835
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Release files / foldlitedb-0.1.4-cp311-abi3-macosx_10_12_x86_64.whl

Download URL foldlitedb-0.1.4-cp311-abi3-macosx_10_12_x86_64.whl
Size 1.4 MB
Tags CPython 3.11 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
e9a15964fbcd5e754ee486088f917818d62da3c96077a71409b57b8a7de6f59a
BLAKE2b-256 checksum
How to use checksums
50803ef9875d053ff0b01a53a8715640b7b88b3fa50832bdf351de7104836be4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Security for foldlitedb 0.1.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 6 files in this release have been uploaded with provenance metadata.

Select from 6 files

Attestation for foldlitedb-0.1.4.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page