dirark

Content-addressed directory archival with deduplication and remote sync.

Archives directories into compressed, deduplicated stores indexed by BLAKE2b checksums. Supports incremental archiving (idempotent re-runs), merging arks, and pushing/pulling to remote storage via rsync.

Requirements

• Python >= 3.12
• System utilities: rsync, tar, zstd

Install

uv add dirark
# or
pip install dirark

CLI

# Archive a directory → creates <src>.ark.d
dirark archive <src_dir>

# Restore all files from an ark
dirark restore <ark_dir> <dest_dir>

# Print a single file's contents to stdout
dirark read <ark_dir> <file_path>

# Merge an ark into a remote one and back (local path or user@host:/path);
# only objects the other side lacks are transferred, nothing is overwritten
dirark push <local_ark> <remote>
dirark pull <remote> <local_ark>

# Merge src_ark into dst_ark
dirark merge <src_ark> <dst_ark>

# Archive a directory and add it to a remote ark (created if absent)
dirark add <src_dir> <remote_ark>

dirark --help-all prints every command's arguments and flags.

Python API

from dirark import archive_dir, restore_ark, ArkReader
from dirark import push_ark, pull_ark, merge_arks, add_dir_to_remote_ark
from pathlib import Path

# Archive
archive_dir(Path("my_dir"))             # creates my_dir.ark.d
archive_dir(Path("my_dir"), ark_out=Path("existing.ark.d"))  # merge into existing

# Restore
restore_ark(Path("my_dir.ark.d"), Path("restored/"))

# Read individual files without full restore
with ArkReader(Path("my_dir.ark.d")) as ark:
    files = ark.list_files()
    checksum = ark.get_checksum("path/to/file.txt")
    data = ark.read_file("path/to/file.txt")

# Remote sync: index-first merges, so arks that diverged (one per machine) can share a remote
push_ark(Path("my_dir.ark.d"), "user@host:/backups/my_dir.ark.d")
pull_ark("user@host:/backups/my_dir.ark.d", Path("local.ark.d"))

# Merge two local arks
merge_arks(Path("src.ark.d"), Path("dst.ark.d"))

# Archive locally and push to remote in one step
add_dir_to_remote_ark(Path("my_dir"), "user@host:/backups/archive.ark.d")

Blob Store

BlobStore exposes the same compressed, deduplicated storage as a content-addressed bytes interface — useful for caching computed artifacts keyed by their content.

from dirark import BlobStore
from pathlib import Path

store = BlobStore(Path("cache"))

# write() stages bytes and returns their BLAKE2b checksum; it is idempotent
checksum = store.write(b"...payload...")
assert store.has(checksum)
assert store.read(checksum) == b"...payload..."

# write() is safe from many processes at once (atomic staging). Once writers
# have stopped, commit() packs staged blobs into the ark's tar.zst archives.
store.commit()

# Pull blobs from another store (e.g. a per-worker shard) into this one
store.merge_from(BlobStore(Path("worker-0")))

# Sync committed blobs with a store rooted at a remote path (staging stays local)
store.push("user@host:/backups/cache")
store.pull("user@host:/backups/cache")

Checksums are computed with the same BLAKE2b-512 hashing used by archive_dir, so a blob and an archived file with identical content share a key and deduplicate against each other — including across merge_from.

Storage Format

An ark is a directory containing:

• index.sqlite — maps file paths to BLAKE2b checksums, and checksums to tar archives
• data-NNNNN.tar.zst — zstd-compressed tars holding the actual file data (batched at MAX_TAR_MB of source bytes each, smaller in a BlobStore, where every read decompresses a whole tar; a single larger file gets its own tar)

Files are deduplicated by checksum. Re-archiving a directory that hasn't changed is a no-op.

Development

Performance benchmarks live in benchmarks/ and exist to settle implementation trade-offs with measurement rather than assumption:

uv run python benchmarks/bench.py            # full
uv run python benchmarks/bench.py --quick    # smaller inputs, fewer reps