aloelite

Aloelite SQLite Filesystem

Overview (current) | Getting Started | Frequently Asked Questions

Troubleshooting | Requirements Spec | Encryption Spec

Contents

• Installation
• Usage
• Overview
• Getting Started
  • Python API
  • Encryption
  • Pathlib-Style Interface
• Command Line
• Mount API
• FUSE
• Volume Manager and WebUI
  • REST API
  • Backup Sync Pattern
• Security Notes
• Design Background
• License

Installation

pip install aloelite

The core install has no FUSE dependencies and runs anywhere Python does. For FUSE support (Linux only):

sudo apt install fuse3 libfuse3-dev
pip install aloelite[fuse]

For Docker:

docker pull aloecraft/aloelite

Usage

CLI

aloelite -f notebook.fs put report.pdf /docs/report.pdf

Python

with Aloelite("notebook.fs") as fs, fs.mount("docs", create=True) as m:
    m.put("/hello.txt", b"hello world")

Docker (volume manager + WebUI)

docker run -d --privileged --device /dev/fuse -p 8080:8080 \
  -v /aloelite-root:/aloelite-root -v /mnt/aloelite:/mnt:rshared \
  aloecraft/aloelite-manager   # then open http://localhost:8080/admin

FUSE Mount

aloelite-fuse photos.fs photos ~/photos   # now it's just a directory

Overview

Aloelite is a filesystem implemented as a SQLite database. The entire filesystem, (i.e. files, directories, metadata, and content) lives in a single portable .sqlite file that can be copied, versioned, and opened anywhere SQLite runs.

It is designed for situations where you want filesystem semantics (paths, directories, streaming I/O) but need more control than a raw filesystem gives you: portable snapshots, at-rest encryption, content deduplication, and a clean programmatic API. A single file is easier to back up, replicate, and audit than a directory tree.

What It Provides

• Python API — full filesystem semantics with bounded-memory streaming I/O and atomic random-access writes (write_range/truncate)
• CLI (aloelite) — the same operations from the shell, with stdin/stdout piping
• FUSE — mount a volume as a plain directory; any application can use it unmodified
• Volume manager + WebUI — a container that provisions volumes for Docker/Podman over HTTP, with a browser admin panel and file explorer
• At-rest encryption per volume (ChaCha20-Poly1305, Argon2id); the PIN is used only at mount time and never stored
• Content deduplication — identical data is stored once per volume, including across repeated backups
• Live snapshots — export a consistent, self-contained SQLite file while the volume is mounted and in use

Implementation Status

• Core model — nodes, edges, volumes, and mounts are fully realized: path resolution, structural operations (create, move, rename, copy, remove, pack/unpack), advisory locking, and mount-scoped sessions.
• Content storage — content-addressed chunk pool with deduplication, per-version manifests, configurable retention, and bounded-memory streaming I/O; production-validated against files in the tens of gigabytes.
• Random access — write_range and truncate are first-class engine operations: unchanged chunks carry into the new version by reference, so partial overwrites of large files are cheap and bounded-memory.
• Encryption — at-rest at the storage boundary (ChaCha20-Poly1305, Argon2id, per-volume wrapped key), with convergent and random nonce modes.
• FUSE — O_RDWR access through a dirty-extent handle (memory bounded by dirty bytes, flushed atomically on fsync/release); symlinks and permission bits (chmod, executables) persist across remounts; honors utimens; streaming writes commit at flush time, synchronously with the application's close(), so a crashed daemon loses only truly in-flight data; hardened handlers return EIO rather than detaching the mount. Hard links, shared-writable mmap, and byte-range locks across separate mounts are not yet implemented.
• CLI — covers the library verbs for scripting.
• Volume manager (manager/) — exposes volumes as FUSE-mounted directories over an HTTP API with a WebUI; usable as a Docker/Podman volume provisioner.

Reserved but not yet realized:

• cryptographic verification of the node tree (Merkle structure over content and placement)
• content-defined chunking
• key rotation
• graph-shaped namespaces beyond the default hierarchical tree
• and node metadata encryption (currently plaintext in the SQLite schema. (see Security Notes))

Getting Started

New here? GETTING_STARTED.md is the friendly tour, organized by use case (Python / CLI / FUSE / Docker). See also FAQ.md and TROUBLESHOOTING.md.

Python API

from aloelite.aloelite import Aloelite
from aloelite.types import WriteMode, Whence

with Aloelite("photos.sqlite") as fs:
    # Mount by name; create=True bootstraps the volume on first run.
    # (Volumes can also be managed explicitly: create_volume / list_volumes /
    #  resolve_volume_name, then mount by id.)
    with fs.mount("photos", create=True) as m:
        m.put("/hello.txt", b"hi")          # create-or-replace, one atomic op
        m.put("/hello.txt", b"hi again")    # replace
        m.put("/log.txt", b"x\n", append=True)  # append (creates if missing)

        m.mkdir("/2024/trip", parents=True, exist_ok=True)  # mkdir -p

        for e in m.list("/"):
            print(e.path)                   # full path, stamped at fetch time

        m.create_container("/2024")
        m.set_metadata("/2024", {"year": "2024", "album": "trip"})
        m.create_entry("/2024/caption.txt", b"a sunset")

        with m.open_write("/note.txt") as w:
            w.write(b"hello ")
            w.write(b"world")

        print(m.read_all("/note.txt"))   # -> b"hello world"

        with m.open_read("/note.txt") as r:
            head = r.read(5)
            r.seek(-5, Whence.END)
            tail = r.read()

        m.write_range("/note.txt", 6, b"WORLD")  # atomic in-place overwrite
        m.truncate("/note.txt", 5)               # -> b"hello"

        m.rename("/note.txt", "readme.txt")
        m.move("/readme.txt", "/2024/readme.txt")
        m.copy("/2024", "/backup")
        m.remove_recursive("/backup")

    fs.prune()
    print(fs.health_check())   # -> [] when consistent

Encryption

PIN = b"correct-horse-battery-staple"

with Aloelite("vault.sqlite") as fs:
    # create=True with a pin creates the volume encrypted; encryption is
    # decided once, at creation (Argon2id key derivation, ChaCha20-Poly1305).
    with fs.mount("vault", pin=PIN, create=True) as m:
        m.create_entry("/secret.txt", b"eyes only")
        print(m.read_all("/secret.txt"))   # -> b"eyes only"

    # Wrong PIN is rejected at mount time (not at read time)
    from aloelite import errors
    try:
        fs.mount("vault", pin=b"wrong")
    except errors.BadKey:
        print("wrong PIN rejected ✓")

    # Durable mounts (records, not sessions) are listable and re-attachable:
    for info in fs.list_mounts():
        print(info.id, info.mount_path, info.state.value)

Encryption is invisible at the Mount API level. Use enc_mode="random" to trade chunk deduplication for zero equality leakage.

Pathlib-Style Interface

The easiest way to work with files inside a volume. No FUSE required. Any Mount doubles as a path root:

from aloelite.aloelite import Aloelite

with Aloelite("photos.sqlite") as fs:
    vol = fs.create_volume("photos")

    with fs.mount(vol.id) as m:
        docs = m / "docs"                        # Mount / str -> AloelitePath
        docs.mkdir(parents=True, exist_ok=True)

        note = docs / "note.txt"
        note.write_text("hello world")
        print(note.read_text())                  # -> "hello world"

        with (docs / "big.bin").open("wb") as w: # bounded-memory streaming
            w.write(b"chunk " * 100_000)

        for child in docs.iterdir():
            print(child, child.stat().size)

        for txt in m.path("/").rglob("*.txt"):   # '*' and '**' globbing
            print(txt)

        note.set_metadata({"author": "mg"})      # NODE-6 metadata
        note.copy("/docs/note.bak")
        note.rename("/docs/renamed.txt")         # full move, returns new path

AloelitePath is pure sugar over the Mount API — it adds nothing to the contract, so everything above is atomic, deduplicated, and encryption-transparent exactly like the underlying operations.

Mount API

Everything in Aloelite goes through one interface: the Mount API. It is defined once, language-neutrally, in config/mount-api.yaml, and implemented in aloelite/operations.py (the Python reference). The CLI, FUSE driver, volume manager, and pathlib wrapper are all thin consumers of the same operations — there is no second path into the file.

The mental model is four nouns:

• A volume is a filesystem tree inside the file (one file can hold many).
• A mount is a durable access point into a volume — all reads and writes are brokered through one.
• Below that, a volume is made of nodes (files and directories) and edges (placements) — you rarely touch these directly, but they are why moves are cheap and history is recoverable.

Every operation you see in the Python examples below (put, list, move, open_write, ...) is a Mount API operation. The CLI exposes the same verbs; FUSE translates kernel calls into them. Learn the API once and every interface is familiar.

Command line

Mount API usage from the command line

# installed with `pip install aloelite`

aloelite -f notebook.fs volumes
aloelite -f notebook.fs ls -l /
aloelite -f notebook.fs put report.pdf /docs/report.pdf
cat log | aloelite -f notebook.fs put - /logs/today --append
aloelite -f notebook.fs get /docs/report.pdf -   # to stdout
aloelite -f notebook.fs mkdir -p /a/b/c
aloelite -f notebook.fs mv /a.txt /docs/a.txt
aloelite -f notebook.fs rm -r /old
aloelite -f notebook.fs cat /docs/report.pdf
aloelite -f notebook.fs cp /docs/report.pdf /backup/report.pdf   # dedup: near-free
aloelite -f notebook.fs stat /docs/report.pdf
aloelite -f notebook.fs tree /
aloelite -f notebook.fs prune --vacuum
aloelite -f notebook.fs mounts # List Mounts (ACC-1a)

note: Set ALOELITE_FILE env var to skip -f entirely (export ALOELITE_FILE=notebook.fs).

-v NAME_OR_ID selects a volume (name, or uuid7 with/without dashes); omit it when the file holds exactly one. Encrypted volumes take the same --pin / --pin-file / --pin-env flags as aloelite-fuse, or prompt interactively.

FUSE

Mount an Aloelite volume as a regular directory (Linux, requires fuse3):

# Plain volume (--create bootstraps a missing volume; a typo'd name errors instead)
aloelite-fuse photos.sqlite photos /mnt/photos --create

# Encrypted volume — three ways to supply the PIN
aloelite-fuse vault.sqlite vault /mnt/vault --pin "my secret"
aloelite-fuse vault.sqlite vault /mnt/vault --pin-file ~/.vaultpin
aloelite-fuse vault.sqlite vault /mnt/vault --pin-env VAULT_PIN

# Unmount
fusermount3 -u /mnt/photos

The FUSE driver uses bounded-memory I/O throughout — a 15 GB copy does not buffer in RAM. Sequential writes stream one chunk at a time; random access (O_RDWR, partial overwrites, truncation) buffers only the dirty byte ranges and commits them as atomic in-place writes on fsync/release. Handlers are hardened: an unexpected error returns EIO to the caller rather than detaching the mount.

Running applications on a mounted volume

Ordinary applications work on a mounted volume unmodified. Aloelite has been validated backing a mail server (docker-mailserver) and a full git workflow — clone, push, gc, packfile reads, executable hooks — directly on a mount. SQLite databases run correctly in rollback-journal mode (PRAGMA journal_mode=PERSIST or TRUNCATE with a busy_timeout; see Troubleshooting for the recipe).

Two categories of software are not yet supported: anything that persists state via shared-writable mmap (WAL-mode SQLite, LMDB, boltdb, Dovecot index files), and anything that requires hard links. Both usually have a configuration escape hatch — point the mmap-backed store at a regular directory, or switch the journal mode — while the payload data stays on the volume.

Volume Manager and WebUI

The volume manager is a privileged container that manages multiple Aloelite volumes and exposes each as a FUSE-mounted subdirectory, accessible to other containers via bind mount.

Run

# Host directories (once)
sudo mkdir -p /aloelite-root /mnt/aloelite

docker run -d --privileged \
  -v /aloelite-root:/aloelite-root \
  -v /mnt/aloelite:/mnt:rshared \
  --device /dev/fuse \
  -p 8080:8080 \
  aloecraft/aloelite-manager

/aloelite-root holds the backing SQLite files and persists across restarts. /mnt/aloelite is the host-visible mount root; FUSE mounts inside the container propagate here via rshared. --privileged (or at minimum CAP_SYS_ADMIN) is required.

API

Mounting with {"persist": true} makes the mount survive container restarts (auto-mount at startup). Encrypted volumes additionally need "pin_env" or "pin_file" naming where the PIN is read from at each startup — the PIN itself is never stored. An explicit unmount revokes the persist flag.

# Create and mount
curl -s -X POST http://localhost:8080/volumes \
  -H 'Content-Type: application/json' \
  -d '{"name": "myphotos"}' | tee /tmp/vol.json

VID=$(jq -r .id /tmp/vol.json)
curl -s -X POST http://localhost:8080/volumes/$VID/mount \
  -H 'Content-Type: application/json' -d '{}'

# The volume is now a plain directory on the host
ls /mnt/aloelite/$VID

# Consume from another container
docker run --rm -v /mnt/aloelite/$VID:/data alpine ls /data

# Backup: poll stat, export on change
curl -s http://localhost:8080/volumes/$VID/stat | jq
curl -s http://localhost:8080/volumes/$VID/export -o snapshot.sqlite

# Encrypted volume
curl -s -X POST http://localhost:8080/volumes \
  -H 'Content-Type: application/json' \
  -d '{"name": "vault", "encrypted": true, "pin": "correct-horse"}'
# Mount with: -d '{"pin": "correct-horse"}'

The export endpoint runs WAL_CHECKPOINT(TRUNCATE) before streaming, producing a complete self-contained SQLite file with no accompanying WAL. The volume does not need to be unmounted to export — SQLite's read consistency guarantees a coherent snapshot regardless of active writes.

The admin panel at /admin includes a per-volume file explorer for any mounted volume: browse with breadcrumbs, upload, download, create folders, and delete. Each volume card also has a Mounts button listing the durable engine mounts inside the backing file (label, path, state), independent of any live FUSE session. The file endpoints operate through the live FUSE mountpoint, so they work identically for plain and encrypted volumes (the mount session already holds the key).

Backup Sync Pattern

loop:
    poll GET /volumes/<id>/stat
    if mtime > last_known_mtime:
        GET /volumes/<id>/export  →  write to temp file  →  rename into place
        last_known_mtime = mtime
    sleep(interval)

The rename into place is atomic; a failed export leaves the previous replica intact.

Security Notes

Chunk data is encrypted at the storage boundary (ChaCha20-Poly1305, Argon2id key derivation). The SQLite file is opaque without the PIN.

Node metadata (paths, timestamps, node IDs, directory structure) is stored in plaintext in the SQLite schema. An observer with access to the file can read the filesystem tree even without the PIN. For sensitive deployments, place the backing file on an encrypted volume (LUKS, encrypted home directory, etc.) or use the pack primitive to seal a subtree before transport.

The volume manager API is intended for trusted networks. PINs are transmitted in request bodies and never logged or persisted; the derived key is held only for the duration of the mount session.

Design Background

The original design abstract and discussion, from which doc/requirements.md was authored.

Abstract

This document specifies the design of a portable filesystem implemented on top of SQLite. The system models a filesystem as a small set of relational primitives (i.e. nodes, edges, volumes, and mounts) rather than as a fixed on-disk layout, deferring byte packing, page management, and durability to SQLite's mature storage engine. It is deliberately interface-agnostic: it presents a coherent internal model of files, directories, placement, and access without committing to any single external protocol, while remaining structurally amenable to exposing one (WebDAV, FUSE, or others) in the future. The design favors a hierarchical tree as its default arrangement but encodes that hierarchy as a relaxable constraint rather than a structural assumption, leaving a clear path toward a more general graph-shaped namespace. Supporting concerns (e.g. content storage, archival, and verifiable modification) are accommodated as first-class parts of the model even where their full implementation is staged for later.

Discussion

The motivation for building on SQLite is portability and reach. A filesystem expressed as a SQLite database is a single, self-describing file that can be opened, moved, and inspected anywhere SQLite runs, which is nearly everywhere, and it inherits decades of work on storage layout and transactional integrity for free. The cost of that choice is that the filesystem's structure must be expressed relationally; the contribution of this design is a set of primitives that do so cleanly while keeping future capabilities reachable rather than precluded.

The model separates four concerns that filesystems often conflate. A node is an identity: a file (Entry) or a directory (Container), bearing a stable time-ordered identifier and its own name. An edge is a placement: a directed, immutable relationship that situates a node beneath a container within a particular volume. A volume is an origin: the root to which a coherent tree of placements ultimately refers. A mount is an access context: a durable, volume-bound access point, anchored at an explicit node, through which operation on the filesystem is brokered. Holding these four apart is what gives the design its flexibility. Because a node's name and existence are independent of where it sits, the same node can in principle be reachable from more than one place, which is the seam through which links, mounts, and an eventual graph layout enter without disturbing the core. Because placement lives in immutable edges, every structural change is expressed as the creation of a new edge rather than the mutation of an existing one, which keeps the history of where things have been available and gives later features (e.g. ordering, verification, recovery) a stable substrate to build on. Because origins are modeled explicitly rather than inferred, the boundary of a volume is a real, referenceable thing rather than a convention. And because access is brokered through mounts rather than ambient, the system has a concrete answer to a question filesystems usually answer with the operating system: who holds a handle, who holds a lock, and what to reclaim when a session ends.

File contents are held apart from node metadata, so that traversing and resolving the namespace touches only small, frequently-accessed rows and never drags large payloads along. Reading and writing a whole file is an atomic operation in the ordinary case, with a streaming, descriptor-like access path for large or incremental I/O. That access path is mediated by mounts: because the filesystem has no native notion of a process, a mount stands in as the session identity that holds open handles and locks, and locks are scoped to the mount that acquired them, so that ending a session has a well-defined effect on everything it held. This advisory locking coexists with rather than commandeers SQLite's own transactional concurrency. Archival packs a subtree into a portable serialized form within the safety of a single transaction, so that the act of consolidating data cannot lose it. And the design reserves room for cryptographic verification of modification (e.g. a Merkle structure over the tree) by ensuring that mutations flow through a single, well-defined path where such bookkeeping can later be attached. None of these later-stage capabilities is fully realized in the first iteration; the purpose of the model described here is to make each of them an addition rather than a redesign.

License

Apache 2.0. See LICENSE.