flyfile

Agent-native data transfer. Push/pull anything (text, files, directories) through a central server, or stream it directly client-to-client. Built for AI agents: JSON output everywhere, stable exit codes, content-addressed dedup, burn-after-read.

pip install flyfile

# server (single worker; data lives in LMDB via flaxkv2)
flyfile serve --port 8632 --token SECRET

# client
export FLYFILE_SERVER=http://host:8632 FLYFILE_TOKEN=SECRET
echo "build log" | flyfile push - --name buildlog --tag ci --ttl 2h
flyfile push ./model.bin --reads 1            # burn after one read
flyfile push ./dataset/                       # dirs stream as tar, no temp files
flyfile push ./repo/ --exclude .init/          # .gitignore applies by default; add rules as needed
flyfile push ./repo/ --no-gitignore            # explicitly transfer ignored paths too
flyfile push ./release.tar --keep             # objects expire in 7d unless --keep
flyfile ls --tag ci --json                    # → {"total": N, "items": [...]}
flyfile rm --until 30d --dry-run              # bulk delete by filters, preview first
flyfile preview k3x9m2pq                      # peek without consuming reads
flyfile pull k3x9m2pq -o ./model.bin

# client → client (server relays the stream, nothing is stored)
flyfile send ./results/          # prints: code: amber-falcon
flyfile recv amber-falcon        # on the other machine

Agent contract

• JSON everywhere: --json, or automatic when stdout is not a TTY. Progress goes to stderr, data to stdout. Never prompts.
• Exit codes (stable): 0 ok · 2 usage · 3 not found · 4 auth · 5 conflict · 6 expired/burned · 7 network (retryable) · 1 other.
• Errors are JSON on stderr: {"error", "message", "retryable", "suggestion"}.
• Idempotent push: content-addressed (sha256). Re-pushing the same bytes is instant ("deduped": true).
• Default retention is 7 days, enforced server-side (so the HTTP API and the Python client behave the same). --ttl adjusts it; --keep stores forever.
• flyfile preview <id> reads the head of an object (or a dir's file manifest) without downloading and without consuming burn-after-read counts. Because it does not consume a read, the response is capped at 64 KiB and sets truncated when it clips.
• flyfile schema [cmd] dumps the command tree as JSON for introspection.
• flyfile send --json emits NDJSON events; the code event arrives before the transfer starts, so an agent can hand it to the receiver immediately.

Breaking changes in 0.4.0

• Directory push now applies the source root's .gitignore by default. Use --no-gitignore (or use_gitignore=False in Python) to upload ignored paths too. Repeatable --exclude PATTERN rules are applied after .gitignore rules.
• Directory uploads now reject absolute or source-escaping symlinks and special nodes such as FIFOs unless they are excluded. This guarantees that a successful upload can be restored by the safe pull extractor.

Breaking changes in 0.3.0

• The on-disk format changed and is not backward compatible. Chunks are now namespaced by upload rather than by content hash, which is what makes concurrent uploads of the same content safe and makes abandoned chunks reclaimable. A server started against a 0.2.x data directory fails at startup with instructions rather than corrupting or silently leaking data. To upgrade: pull anything you still need with 0.2.x, then delete the data directory (default ~/.local/share/flyfile).
• GET /objects/{id}/chunks/{idx} for idx > 0 now requires the x-ff-lease header issued with chunk 0. Previously any caller could fetch later chunks without claiming a read, which made --reads 1 meaningless for multi-chunk objects. The bundled client handles this transparently; custom clients that fetch chunks directly must pass the lease through.

Breaking changes in 0.2.0

• GET /objects (and FlyfileClient.ls()) now returns {"total": N, "items": [...]} instead of a bare list, so a paged query can report how many objects matched. flyfile ls --json changes shape accordingly. Upgrade server and clients together.
• Uploads without an explicit TTL now expire after 7 days instead of being kept forever. Pass --keep (CLI) or ttl=0 (client/x-ff-ttl: 0) for the old behavior.

Design notes

• Storage is flaxkv2 (LMDB): metadata and 8 MiB content chunks in one env. The LMDB file does not shrink after deletes (free pages are reused; file size ≈ historical peak).
• Compression (zstd-3) happens on the client; the server stores/relays compressed bytes. Each 8 MiB chunk of a large upload is an independent zstd frame, so parallel upload, resume, and parallel download all work per-chunk.
• Burn-after-read: the read is claimed atomically when chunk 0 (or /content) is fetched. That claim issues a short-lived lease (x-ff-lease, default 5 min) which an in-flight parallel download presents to fetch the remaining chunks, so a burnt object stays unreadable to everyone else while the legitimate download finishes. A sweeper grace period (default 15 min) keeps the chunks around for the life of the lease.
• Every chunk belongs to exactly one owner at all times — an upload ledger entry or a blob record — and ownership moves in a single transaction. That invariant is what makes crashes, aborted uploads, and concurrent uploads of the same content unable to strand or clobber data. See docs/architecture.md.
• Run exactly one uvicorn worker: relay pairing and the burn-claim lock are in-process.

Development

uv venv && uv pip install -e ".[dev]"
.venv/bin/pytest
scripts/bench.sh   # 1 GiB loopback throughput smoke test