filetx

Transactional filesystem operations — batch moves, copies and deletes that either all commit or all roll back.

Stack: Python 3.10+, standard library only, zero runtime dependencies.

pip install filetx

from filetx import Transaction

with Transaction(journal="release.jsonl") as tx:
    tx.mkdir("dist", exist_ok=True)
    tx.move("build/app.bin", "dist/app.bin")
    tx.copy("build/manifest.json", "dist/manifest.json")
    tx.delete("build", missing_ok=True)

Nothing happens until the block exits. If any step fails, every completed step is undone. If the process is killed part-way through, filetx undo release.jsonl finishes the rollback from the journal it left behind.

1. The problem

Multi-step filesystem work has no rollback. A release script that creates a directory, moves three artefacts into it and deletes the build tree has four chances to fail, and if it fails on the third the disk is left in a state that matches neither the before nor the after. The usual responses are to copy everything to a backup directory first — which costs time proportional to the data, so nobody does it for large trees — or to write bespoke cleanup code per script, which is only exercised on the day it is needed and is therefore usually wrong.

Python has shutil for individual operations and atomicwrites for making a single file write atomic. Neither gives you a batch that is all-or-nothing. filetx is that missing piece: record the operations, commit them, and get the tree back unchanged if anything goes wrong — including if the process dies.

This started as the transaction log inside a file-organiser tool I built, where a half-finished reorganisation of someone's photo library is a genuinely bad outcome. The engine turned out to be the interesting part, so it was extracted, generalised and published.

2. What it operates on

There is no dataset here — this is a library, and its input is whatever tree the caller points it at.

Non-goals. Not a distributed transaction manager, not crash-safe against media failure, and not safe against another process mutating the same paths concurrently — see Limitations.

3. Architecture

flowchart TD
    A["tx.move / copy / delete / mkdir<br/>(recorded, not performed)"] --> B["commit()"]
    B --> C["plan_undo — validate,<br/>compute undo record"]
    C -->|invalid| R
    C --> D["journal — write undo record, fsync"]
    D --> E["apply — change the filesystem"]
    E -->|OSError| R
    E --> F{"more operations?"}
    F -->|yes| C
    F -->|no| G["journal — commit"]
    G --> H["purge staged material"]
    R["revert applied operations,<br/>in reverse order"] --> S{"all reverted?"}
    S -->|yes| T["journal — rollback<br/>raise TransactionError"]
    S -->|no| U["journal left unsettled<br/>raise RollbackError"]
    U -.->|"later: filetx undo"| R

The undo record is written to the journal before the change it describes is attempted. That is the whole design: a record written afterwards is useless to a process that died in between.

Writing it first requires knowing, in advance, where a file is about to be moved to — so staged names are derived deterministically from (path, transaction id, plan index) rather than allocated at random.

The remaining ambiguity is that the journal can describe a change that never actually happened. Every revert therefore inspects the real filesystem before acting instead of trusting the record, which makes reverting idempotent: running it twice, or against an operation that never applied, does nothing.

4. Key decisions & tradeoffs

5. Results

Measured on Windows 11 (NTFS), Python 3.13.9, 12-core CPU / 64 GB RAM, median of 3 runs. Reproduce with uv run python scripts/benchmark.py --files 5000 --size 1024. Run-to-run variation on this machine is roughly ±15%, and the speedup ratio has ranged 1,000–1,300× across runs — the figure worth trusting is the order of magnitude, not the exact multiple.

The benchmark tree is synthetic and generated by the script above. The comparison is against shutil, i.e. what you would write by hand, not against another library.

6. How to run

git clone https://github.com/Prithv122/filetx.git
cd filetx
uv sync
uv run pytest --cov=src --cov-report=term-missing

No environment variables, services or datasets are needed. Other useful commands:

uv run python scripts/benchmark.py     # reproduce the numbers in section 5
uv run ruff check . && uv run ruff format --check .
uv build && uv run --with twine twine check --strict dist/*

Recovering an interrupted run

filetx inspect release.jsonl   # what was this transaction doing, and how far did it get?
filetx undo release.jsonl      # put it back

inspect marks each operation done, PARTIAL or -. undo rolls an unsettled transaction back, cleans up after a committed one, and reports — but never deletes — staged data whose original path has since been reoccupied.

Limitations

• No locking. Two transactions touching the same paths concurrently will interfere. Serialise them yourself.
• fsync durability only. The journal is fsync-ed before each change, so a process kill or power loss is recoverable, but a lying disk cache or media failure is not. Pass fsync=False to trade that away for speed.
• Staged material is visible. During a transaction, .filetx-* entries exist alongside your files. A crash leaves them until filetx undo runs; filetx.STAGE_PREFIX is exported so scanners can skip them.
• Cross-filesystem moves are O(size). They fall back to copying.

7. What I'd change at 100× scale

The design assumes a batch you can hold in memory and a journal you read whole. At 100× — hundreds of thousands of operations per transaction — three things break, in this order:

1. The plan is written to the journal as one record. A single JSON line holding 500k operations has to be serialised and fsync-ed before the first change happens, and re-parsed in full during recovery. I would stream the plan as one record per operation and make read_journal incremental.
2. fsync per operation dominates. Measured at 1.57 ms per record here (versus 0.01 ms unflushed), so 500k operations is about 13 minutes of waiting on the disk before any useful work happens. I would batch the flush — group commits of N records — accepting a bounded window where the journal lags reality, and handle it by making recovery re-verify the tail against the filesystem, which the state-checking reverts already support.
3. Rollback is serial. Undoing 500k renames one at a time wastes an SSD's parallelism. Reverts of operations on disjoint paths could run in a thread pool; the ordering constraint is only between operations whose paths nest, which is a partial order the planner could compute up front.

What I would not change is the staging strategy — it is the part that gets better with scale, not worse.

References

The write-ahead ordering and the do/undo/purge split follow standard database recovery practice; ARIES (Mohan et al., 1992) is the canonical description, and the naming of the three phases here is deliberately borrowed from it. No implementation was consulted — the filesystem constraints are different enough that the resemblance is conceptual.

Licence

MIT — see LICENSE.