aloelite

Aloelite Single-File Filesystem

A portable encrypted filesystem stored inside one file

Overview (current) | Getting Started | Frequently Asked Questions

Troubleshooting | Requirements Spec | Encryption Spec

Contents

• Installation
• Usage
• Overview
• Getting Started
  • Python API
  • Encryption
  • Pathlib-Style Interface
• Command Line
• Mount API
• FUSE
• Volume Manager and WebUI
  • API
  • Backup Sync Pattern
• Security Notes
• Design Background
• License

Installation

pip install aloelite

The core install has no FUSE dependencies and runs anywhere Python does. For FUSE support (Linux only):

sudo apt install fuse3 libfuse3-dev
pip install aloelite[fuse]

For Docker:

docker pull aloecraft/aloelite

Usage

CLI

aloelite -f notebook.fs               # creates the file (with a default volume) or shows what's inside
aloelite -f notebook.fs put report.pdf /docs/report.pdf

Python

with Aloelite("notebook.fs") as fs, fs.mount("docs", create=True) as m:
    m.put("/hello.txt", b"hello world")

WebUI (no Docker, no FUSE, no sudo — runs anywhere Python does)

aloelite-web
# open http://localhost:8080 — create a volume, drag files in and out,
# download the .sqlite file and take it with you

Docker (volume manager + WebUI + FUSE provisioning)

docker run -d --privileged --device /dev/fuse -p 8080:8080 \
  -v /aloelite-root:/aloelite-root -v /mnt/aloelite:/mnt:rshared \
  aloecraft/aloelite-manager   # then open http://localhost:8080/admin

FUSE Mount

aloelite-fuse photos.fs photos ~/photos   # now it's just a directory

Overview

Aloelite is a filesystem implemented as a SQLite database. The entire filesystem, (i.e. files, directories, metadata, and content) lives in a single portable .sqlite file that can be copied, versioned, and opened anywhere SQLite runs.

It is designed for situations where you want filesystem semantics (paths, directories, streaming I/O) but need more control than a raw filesystem gives you: portable snapshots, at-rest encryption, content deduplication, and a clean programmatic API. A single file is easier to back up, replicate, and audit than a directory tree.

What It Provides

• Python API — full filesystem semantics with bounded-memory streaming I/O and atomic random-access writes (write_range/truncate)
• CLI (aloelite) — the same operations from the shell, with stdin/stdout piping
• FUSE — mount a volume as a plain directory; any application can use it unmodified
• Volume manager + WebUI — a browser admin panel with a per-volume file explorer (drag-and-drop upload, download, import/export of whole filesystem files) that needs no FUSE or container; optionally also a privileged container that provisions FUSE-mounted volumes for Docker/Podman over HTTP
• At-rest encryption per volume (ChaCha20-Poly1305, Argon2id); the PIN is used only at mount time and never stored
• Content deduplication — identical data is stored once per volume, including across repeated backups
• Live snapshots — export a consistent, self-contained SQLite file while the volume is mounted and in use

Implementation Status

• Core model — nodes, edges, volumes, and mounts are fully realized: path resolution, structural operations (create, move, rename, copy, remove, pack/unpack), advisory locking, and mount-scoped sessions.
• Content storage — content-addressed chunk pool with deduplication, per-version manifests, configurable retention, and bounded-memory streaming I/O; production-validated against files in the tens of gigabytes.
• Random access — write_range and truncate are first-class engine operations: unchanged chunks carry into the new version by reference, so partial overwrites of large files are cheap and bounded-memory.
• Encryption — at-rest at the storage boundary (ChaCha20-Poly1305, Argon2id, per-volume wrapped key), with convergent and random nonce modes.
• FUSE — O_RDWR access through a dirty-extent handle (memory bounded by dirty bytes, flushed atomically on fsync/release); symlinks and permission bits (chmod, executables) persist across remounts; honors utimens; streaming writes commit at flush time, synchronously with the application's close(), so a crashed daemon loses only truly in-flight data; hardened handlers return EIO rather than detaching the mount. Hard links, shared-writable mmap, and byte-range locks across separate mounts are not yet implemented.
• CLI — covers the library verbs for scripting.
• Volume manager (manager/) — an HTTP API + WebUI with two frontends per volume: direct (a held engine session serving the browser file explorer — no FUSE, no privileges, runs anywhere Python does) and FUSE (volumes exposed as directories for Docker/Podman consumers). Filesystem files can be imported and exported through the WebUI.

Reserved but not yet realized:

• cryptographic verification of the node tree (Merkle structure over content and placement)
• content-defined chunking
• key rotation
• graph-shaped namespaces beyond the default hierarchical tree
• and node metadata encryption (currently plaintext in the SQLite schema. (see Security Notes))

Getting Started

New here? GETTING_STARTED.md is the friendly tour, organized by use case (Python / CLI / FUSE / Docker). See also FAQ.md and TROUBLESHOOTING.md.

Python API

from aloelite.aloelite import Aloelite
from aloelite.types import WriteMode, Whence

with Aloelite("photos.sqlite") as fs:
    # Mount by name; create=True bootstraps the volume on first run.
    # (Volumes can also be managed explicitly: create_volume / list_volumes /
    #  resolve_volume_name, then mount by id.)
    with fs.mount("photos", create=True) as m:
        m.put("/hello.txt", b"hi")          # create-or-replace, one atomic op
        m.put("/hello.txt", b"hi again")    # replace
        m.put("/log.txt", b"x\n", append=True)  # append (creates if missing)

        m.mkdir("/2024/trip", parents=True, exist_ok=True)  # mkdir -p

        for e in m.list("/"):
            print(e.path)                   # full path, stamped at fetch time

        m.create_container("/2024")
        m.set_metadata("/2024", {"year": "2024", "album": "trip"})
        m.create_entry("/2024/caption.txt", b"a sunset")

        with m.open_write("/note.txt") as w:
            w.write(b"hello ")
            w.write(b"world")

        print(m.read_all("/note.txt"))   # -> b"hello world"

        with m.open_read("/note.txt") as r:
            head = r.read(5)
            r.seek(-5, Whence.END)
            tail = r.read()

        m.write_range("/note.txt", 6, b"WORLD")  # atomic in-place overwrite
        m.truncate("/note.txt", 5)               # -> b"hello"

        m.rename("/note.txt", "readme.txt")
        m.move("/readme.txt", "/2024/readme.txt")
        m.copy("/2024", "/backup")
        m.remove_recursive("/backup")

    fs.prune()
    print(fs.health_check())   # -> [] when consistent

Encryption

PIN = b"correct-horse-battery-staple"

with Aloelite("vault.sqlite") as fs:
    # create=True with a pin creates the volume encrypted; encryption is
    # decided once, at creation (Argon2id key derivation, ChaCha20-Poly1305).
    with fs.mount("vault", pin=PIN, create=True) as m:
        m.create_entry("/secret.txt", b"eyes only")
        print(m.read_all("/secret.txt"))   # -> b"eyes only"

    # Wrong PIN is rejected at mount time (not at read time)
    from aloelite import errors
    try:
        fs.mount("vault", pin=b"wrong")
    except errors.BadKey:
        print("wrong PIN rejected ✓")

    # Durable mounts (records, not sessions) are listable and re-attachable:
    for info in fs.list_mounts():
        print(info.id, info.mount_path, info.state.value)

Encryption is invisible at the Mount API level. Use enc_mode="random" to trade chunk deduplication for zero equality leakage.

Pathlib-Style Interface

The easiest way to work with files inside a volume. No FUSE required. Any Mount doubles as a path root:

from aloelite.aloelite import Aloelite

with Aloelite("photos.sqlite") as fs:
    vol = fs.create_volume("photos")

    with fs.mount(vol.id) as m:
        docs = m / "docs"                        # Mount / str -> AloelitePath
        docs.mkdir(parents=True, exist_ok=True)

        note = docs / "note.txt"
        note.write_text("hello world")
        print(note.read_text())                  # -> "hello world"

        with (docs / "big.bin").open("wb") as w: # bounded-memory streaming
            w.write(b"chunk " * 100_000)

        for child in docs.iterdir():
            print(child, child.stat().size)

        for txt in m.path("/").rglob("*.txt"):   # '*' and '**' globbing
            print(txt)

        note.set_metadata({"author": "mg"})      # NODE-6 metadata
        note.copy("/docs/note.bak")
        note.rename("/docs/renamed.txt")         # full move, returns new path

AloelitePath is pure sugar over the Mount API — it adds nothing to the contract, so everything above is atomic, deduplicated, and encryption-transparent exactly like the underlying operations.

Mount API

Everything in Aloelite goes through one interface: the Mount API. It is defined once, language-neutrally, in config/mount-api.yaml, and implemented in aloelite/operations.py (the Python reference). The CLI, FUSE driver, volume manager, and pathlib wrapper are all thin consumers of the same operations — there is no second path into the file.

The mental model is four nouns:

• A volume is a filesystem tree inside the file (one file can hold many).
• A mount is a durable access point into a volume — all reads and writes are brokered through one.
• Below that, a volume is made of nodes (files and directories) and edges (placements) — you rarely touch these directly, but they are why moves are cheap and history is recoverable.

Every operation you see in the Python examples below (put, list, move, open_write, ...) is a Mount API operation. The CLI exposes the same verbs; FUSE translates kernel calls into them. Learn the API once and every interface is familiar.

config/mount-api.yaml is not documentation that can drift: the error set, enums, records, and operation list are asserted against the Python projection by tests/test_spec_projection.py, so adding an operation without declaring it fails the build.

The contract a second implementation needs lives in conformance/:

• conformance/scenarios/ — operation sequences and the state they must produce, as data. Every implementation runs the same scenarios from its own runner instead of hand-translating another language's tests.
• conformance/vectors/ — fixed inputs to exact bytes, pinning content addressing, chunking, and the ENC-2 key ladder. This is what lets a port be written independently and still be known to interoperate.

Python's runners are tests/test_conformance_suite.py and tests/test_format_vectors.py. See conformance/README.md for the scenario format.

Command line

Mount API usage from the command line

# installed with `pip install aloelite`

aloelite -f notebook.fs volumes
aloelite -f notebook.fs ls -l /
aloelite -f notebook.fs put report.pdf /docs/report.pdf
cat log | aloelite -f notebook.fs put - /logs/today --append
aloelite -f notebook.fs get /docs/report.pdf -   # to stdout
aloelite -f notebook.fs put -r ./project /code   # a whole tree in
aloelite -f notebook.fs get -r /code ./restored  # a whole tree out
aloelite -f notebook.fs mkdir -p /a/b/c
aloelite -f notebook.fs mv /a.txt /docs/a.txt
aloelite -f notebook.fs rm -r /old
aloelite -f notebook.fs cat /docs/report.pdf
aloelite -f notebook.fs cp /docs/report.pdf /backup/report.pdf   # dedup: near-free
aloelite -f notebook.fs stat /docs/report.pdf
aloelite -f notebook.fs tree /
aloelite -f notebook.fs prune --vacuum
aloelite -f notebook.fs mounts # List Mounts (ACC-1a)
aloelite --pin -f notebook.fs volume create vault   # explicit (encrypted) volume
aloelite -f notebook.fs volume ls                   # alias of `volumes`
aloelite --version

Running aloelite -f FILE with no command creates the file on first run (bootstrapping a default volume named main — encrypted if a --pin* flag is given, plain otherwise) and prints a status summary on later runs.

Quick one-liners write stdin without a subcommand:

echo "hello" | aloelite -f notebook.fs --in /file.txt       # create/overwrite
echo "more"  | aloelite -f notebook.fs --append /file.txt   # append (creates)

put -r and get -r move whole trees, one file at a time (bounded memory — nothing is staged). The destination follows cp -r: an existing container (put) or directory (get) receives the source inside it as DST/<name>; anything else becomes the tree's new root.

aloelite -f notebook.fs put -r ./project /code    # /code is the tree
aloelite -f notebook.fs mkdir /backup
aloelite -f notebook.fs put -r ./project /backup  # /backup/project
aloelite -f notebook.fs get -r /code ./restored

Empty directories survive the round trip. Symlinked files are copied by content; symlinked directories are skipped (never descended, so a cycle cannot hang the transfer), as is anything that is not a regular file — each skip is named on stderr and counted in the one-line summary. These are CLI conveniences: the loop lives in the CLI, and every step is an ordinary single-node Mount API call.

aloelite fuse ... and aloelite web ... delegate to aloelite-fuse and aloelite-web, so one command covers every interface (a helpful message appears if the FUSE extra isn't installed).

Maintenance lives in aloelite-admin (alias: aloelite admin) — it operates on volumes, keys, and the file itself, where the main CLI operates on files inside volumes:

aloelite-admin -f notebook.fs pin change     # rotate a PIN (data untouched)
aloelite-admin -f notebook.fs snapshot friday          # in-place fork
aloelite-admin -f notebook.fs export backup.fs         # volume -> another file
aloelite-admin -f backup.fs import notebook.fs         # the same, other way
aloelite-admin -f notebook.fs verify --deep  # integrity-check every chunk
aloelite-admin -f notebook.fs health         # structural consistency check
aloelite-admin -f notebook.fs info           # one-screen file dossier

note: Set ALOELITE_FILE env var to skip -f entirely (export ALOELITE_FILE=notebook.fs).

-v NAME_OR_ID selects a volume (name, or uuid7 with/without dashes); omit it when the file holds exactly one. Encrypted volumes take the same --pin / --pin-file / --pin-env flags as aloelite-fuse; a bare --pin (no value; place it before the subcommand, followed by another flag, e.g. aloelite --pin -f file.fs ls /) prompts via getpass, and an encrypted volume with no pin flag prompts interactively too. Pin flags against an unencrypted volume are rejected up front with a pointed error. When a prompted PIN creates a volume, it is asked for twice (confirmation), since a mistyped creation PIN is unrecoverable.

FUSE

Mount an Aloelite volume as a regular directory (Linux, requires fuse3):

# Plain volume (--create bootstraps a missing volume; a typo'd name errors instead)
aloelite-fuse photos.sqlite photos /mnt/photos --create

# Encrypted volume — three ways to supply the PIN
aloelite-fuse vault.sqlite vault /mnt/vault --pin "my secret"
aloelite-fuse vault.sqlite vault /mnt/vault --pin-file ~/.vaultpin
aloelite-fuse vault.sqlite vault /mnt/vault --pin-env VAULT_PIN

# Unmount
fusermount3 -u /mnt/photos

The FUSE driver uses bounded-memory I/O throughout — a 15 GB copy does not buffer in RAM. Sequential writes stream one chunk at a time; random access (O_RDWR, partial overwrites, truncation) buffers only the dirty byte ranges and commits them as atomic in-place writes on fsync/release. Handlers are hardened: an unexpected error returns EIO to the caller rather than detaching the mount.

Running applications on a mounted volume

Ordinary applications work on a mounted volume unmodified. Aloelite has been validated backing a mail server (docker-mailserver) and a full git workflow — clone, push, gc, packfile reads, executable hooks — directly on a mount. SQLite databases run correctly in rollback-journal mode (PRAGMA journal_mode=PERSIST or TRUNCATE with a busy_timeout; see Troubleshooting for the recipe).

Two categories of software are not yet supported: anything that persists state via shared-writable mmap (WAL-mode SQLite, LMDB, boltdb, Dovecot index files), and anything that requires hard links. Both usually have a configuration escape hatch — point the mmap-backed store at a regular directory, or switch the journal mode — while the payload data stays on the volume.

Volume Manager and WebUI

The volume manager serves Aloelite volumes over an HTTP API with a browser admin panel. Each volume is served by one of two frontends:

• Direct — the manager holds a live engine session and the browser file explorer talks straight to the Mount API. No FUSE, no privileges, no container; runs anywhere Python does (including WSL). This is the lowest-commitment way to use Aloelite: install, run, open a browser.
• FUSE — the volume is exposed as a plain directory that other containers bind-mount (Linux; typically run as the privileged container below).

Run (direct only — no FUSE, no container)

aloelite-web

That's the whole setup: direct mode, bound to 127.0.0.1:8080, data in ~/.aloelite. No sudo, no directories to prepare. Flags (see aloelite-web --help): -p/--port, --host, --root, and --fuse to run the container-grade FUSE provisioning mode instead; the matching ALOELITE_* environment variables are honored when a flag is absent. The manager API has no authentication — keep the default loopback bind and put a reverse proxy with auth in front if you need remote access.

Open http://localhost:8080: New volume creates a filesystem file, a volume inside it, and opens the file explorer in one step. Drag files in (with upload progress), preview images, PDFs, and text in place, rename/move/copy from each row, Download the .sqlite file from its card to take it with you, and Import any Aloelite file to pick up where you left off. Encrypted volumes prompt for their PIN when opened.

Run (container, with FUSE provisioning)

# Host directories (once)
sudo mkdir -p /aloelite-root /mnt/aloelite

docker run -d --privileged \
  -v /aloelite-root:/aloelite-root \
  -v /mnt/aloelite:/mnt:rshared \
  --device /dev/fuse \
  -p 8080:8080 \
  aloecraft/aloelite-manager

/aloelite-root holds the backing SQLite files and persists across restarts. /mnt/aloelite is the host-visible mount root; FUSE mounts inside the container propagate here via rshared. --privileged (or at minimum CAP_SYS_ADMIN) is required for FUSE mounts; direct-mode volumes work in this container too.

API

Mounting with {"persist": true} makes a FUSE mount survive container restarts (auto-mount at startup). Encrypted volumes additionally need "pin_env" or "pin_file" naming where the PIN is read from at each startup — the PIN itself is never stored. An explicit unmount revokes the persist flag. Direct sessions end with the manager process, so persist with "mode": "direct" is rejected; re-open the volume from the WebUI after a restart.

# Create and mount
curl -s -X POST http://localhost:8080/volumes \
  -H 'Content-Type: application/json' \
  -d '{"name": "myphotos"}' | tee /tmp/vol.json

VID=$(jq -r .id /tmp/vol.json)
curl -s -X POST http://localhost:8080/volumes/$VID/mount \
  -H 'Content-Type: application/json' -d '{}'

# The volume is now a plain directory on the host
ls /mnt/aloelite/$VID

# Consume from another container
docker run --rm -v /mnt/aloelite/$VID:/data alpine ls /data

# Backup: poll stat, export on change
curl -s http://localhost:8080/volumes/$VID/stat | jq
curl -s http://localhost:8080/volumes/$VID/export -o snapshot.sqlite

# Encrypted volume
curl -s -X POST http://localhost:8080/volumes \
  -H 'Content-Type: application/json' \
  -d '{"name": "vault", "encrypted": true, "pin": "correct-horse"}'
# Mount with: -d '{"pin": "correct-horse"}'

The export endpoint runs WAL_CHECKPOINT(TRUNCATE) before streaming, producing a complete self-contained SQLite file with no accompanying WAL. The volume does not need to be unmounted to export — SQLite's read consistency guarantees a coherent snapshot regardless of active writes.

The admin panel at /admin shows one card per filesystem file with its volumes inside. Open on a volume unlocks it (prompting for a PIN when encrypted) and drops you into the file explorer: browse with breadcrumbs, drag-and-drop upload (folders included), download, create folders, and delete. Each file card has Download (export the .sqlite) and a rename control; Import at the top of the page accepts any Aloelite file. Operator features — FUSE mount, checkpoint, stat, and the durable engine-mount listing — live in each volume's overflow menu. Explorer operations go through the direct engine session (or the live FUSE mountpoint for FUSE-fronted volumes), so plain and encrypted volumes behave identically once open.

Backup Sync Pattern

loop:
    poll GET /volumes/<id>/stat
    if mtime > last_known_mtime:
        GET /volumes/<id>/export  →  write to temp file  →  rename into place
        last_known_mtime = mtime
    sleep(interval)

The rename into place is atomic; a failed export leaves the previous replica intact.

Security Notes

Chunk data is encrypted at the storage boundary (ChaCha20-Poly1305, Argon2id key derivation). The SQLite file is opaque without the PIN.

Node metadata (paths, timestamps, node IDs, directory structure) is stored in plaintext in the SQLite schema. An observer with access to the file can read the filesystem tree even without the PIN. For sensitive deployments, place the backing file on an encrypted volume (LUKS, encrypted home directory, etc.) or use the pack primitive to seal a subtree before transport.

The volume manager API is intended for trusted networks. PINs are transmitted in request bodies and never logged or persisted; the derived key is held only for the duration of the mount session.

Design Background

The original design abstract and discussion, from which doc/requirements.md was authored.

Abstract

This document specifies the design of a portable filesystem implemented on top of SQLite. The system models a filesystem as a small set of relational primitives (i.e. nodes, edges, volumes, and mounts) rather than as a fixed on-disk layout, deferring byte packing, page management, and durability to SQLite's mature storage engine. It is deliberately interface-agnostic: it presents a coherent internal model of files, directories, placement, and access without committing to any single external protocol, while remaining structurally amenable to exposing one (WebDAV, FUSE, or others) in the future. The design favors a hierarchical tree as its default arrangement but encodes that hierarchy as a relaxable constraint rather than a structural assumption, leaving a clear path toward a more general graph-shaped namespace. Supporting concerns (e.g. content storage, archival, and verifiable modification) are accommodated as first-class parts of the model even where their full implementation is staged for later.

Discussion

The motivation for building on SQLite is portability and reach. A filesystem expressed as a SQLite database is a single, self-describing file that can be opened, moved, and inspected anywhere SQLite runs, which is nearly everywhere, and it inherits decades of work on storage layout and transactional integrity for free. The cost of that choice is that the filesystem's structure must be expressed relationally; the contribution of this design is a set of primitives that do so cleanly while keeping future capabilities reachable rather than precluded.

The model separates four concerns that filesystems often conflate. A node is an identity: a file (Entry) or a directory (Container), bearing a stable time-ordered identifier and its own name. An edge is a placement: a directed, immutable relationship that situates a node beneath a container within a particular volume. A volume is an origin: the root to which a coherent tree of placements ultimately refers. A mount is an access context: a durable, volume-bound access point, anchored at an explicit node, through which operation on the filesystem is brokered. Holding these four apart is what gives the design its flexibility. Because a node's name and existence are independent of where it sits, the same node can in principle be reachable from more than one place, which is the seam through which links, mounts, and an eventual graph layout enter without disturbing the core. Because placement lives in immutable edges, every structural change is expressed as the creation of a new edge rather than the mutation of an existing one, which keeps the history of where things have been available and gives later features (e.g. ordering, verification, recovery) a stable substrate to build on. Because origins are modeled explicitly rather than inferred, the boundary of a volume is a real, referenceable thing rather than a convention. And because access is brokered through mounts rather than ambient, the system has a concrete answer to a question filesystems usually answer with the operating system: who holds a handle, who holds a lock, and what to reclaim when a session ends.

File contents are held apart from node metadata, so that traversing and resolving the namespace touches only small, frequently-accessed rows and never drags large payloads along. Reading and writing a whole file is an atomic operation in the ordinary case, with a streaming, descriptor-like access path for large or incremental I/O. That access path is mediated by mounts: because the filesystem has no native notion of a process, a mount stands in as the session identity that holds open handles and locks, and locks are scoped to the mount that acquired them, so that ending a session has a well-defined effect on everything it held. This advisory locking coexists with rather than commandeers SQLite's own transactional concurrency. Archival packs a subtree into a portable serialized form within the safety of a single transaction, so that the act of consolidating data cannot lose it. And the design reserves room for cryptographic verification of modification (e.g. a Merkle structure over the tree) by ensuring that mutations flow through a single, well-defined path where such bookkeeping can later be attached. None of these later-stage capabilities is fully realized in the first iteration; the purpose of the model described here is to make each of them an addition rather than a redesign.

License

Apache 2.0. See LICENSE.