LazyFork

Fork running containers with 1 MB of overhead.

A fork is an exact copy of a running container mid-execution: same processes, memory and filesystem, continuing from the same instruction. Forks can be forked again.

How it works:

• One CRIU dump of the container, rewritten so its memory lives in files on tmpfs; many restores from it.
• Every fork maps those files privately. Reads share the page cache; the kernel copies a page only when a fork writes it.
• Untouched and read-only memory costs nothing per fork, and no daemon sits on the page-fault path.
• The filesystem is an overlay stack; each fork writes to its own layer.
• No kernel or CRIU patches.

Install

1. Environment macOS: LazyFork runs its daemon in a Linux VM managed by Lima. Linux: a host with CRIU 4.1+, userfaultfd, overlayfs and cgroup v2, for example Ubuntu 26.04.
2. LazyFork

 pip install lazyfork

Example

Start a container that counts, look inside it, fork it, and watch the fork continue the count:

lazyfork daemon start
lazyfork create python:3.12-slim --name counter -- \
    python3 -c 'import itertools, time; [print(i, flush=True) or time.sleep(1) for i in itertools.count()]'
lazyfork attach counter          # a shell inside: try `cat /var/log/lazyfork/output.log`, then `exit`
lazyfork fork counter --name twin
lazyfork attach twin            # twin picks up the count where counter was; Ctrl-C to detach

Fork a 250 MB container 100 times

A container holding a 256 MB heap, forked a hundred times:

lazyfork create python:3.12-slim --name big -- \
    python3 -c 'import itertools, time; heap = bytearray(256 << 20); [print(i, flush=True) or time.sleep(1) for i in itertools.count()]'
lazyfork fork big -n 100
lazyfork list                   # big: 260 MB; each fork: 1.7 MB; all hundred together: 171 MB
lazyfork status

The snapshot's memory is written once into files on tmpfs (258 MB, 41 ms). Every fork maps them and owns only what it writes.

 id             name   state     parent   depth      pid   procs      memory
 ──────────────────────────────────────────────────────────────────────────────
 055bf112e881   big    running   -            0   383402       1    260.5 MB
 fcfa15f04e7c   -      running   big          1   383421       1      1.7 MB
 6d796ab30396   -      running   big          1   383434       1      1.7 MB
 45eefddd251d   -      running   big          1   383447       1      1.7 MB
 bf6954b745eb   -      running   big          1   383460       1      1.7 MB
 7f037b6cbc9b   -      running   big          1   383473       1      1.7 MB
...
 101 total                                                          431.7 MB

Mechanism

A fork is a CRIU restore of a snapshot whose memory has been rewritten into file-backed private mappings. The kernel's normal handling of MAP_PRIVATE file mappings then provides the copy-on-write.

1. The snapshot moves memory into files. criu dump writes the container's pages into its image as usual. LazyFork then rewrites the image: for every private anonymous VMA that holds pages, it creates a sparse file on tmpfs, copies the VMA's pages into it at their VMA offsets with copy_file_range, and edits the VMA entry from MAP_ANONYMOUS to a private mapping of that file at offset 0. The pages are removed from the image's pagemap and a file entry for /.lazyfork/mem/<vma>.mem is added. CRIU already restores private file mappings for ordinary programs, so nothing in CRIU changes.

2. A restore maps the files. criu restore opens each file and calls mmap(addr, len, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, fd, 0). No page data is copied and the page tables for those regions start empty. A fork therefore holds about 1 MB right after restore, whatever its heap size.

3. Reads map shared pages. The files live on tmpfs, so their contents are page-cache pages already in RAM. The first read of an address raises a page fault; the kernel looks the page up in the file's page cache and installs a read-only page-table entry pointing at that physical page (plus up to 15 cached neighbours through fault-around). Every fork reading the same address maps the same 4 KB, so the parent's memory exists once no matter how many forks read it.

4. Writes copy one page. A write to a page mapped read-only raises a protection fault. Because the mapping is MAP_PRIVATE, the kernel allocates a fresh anonymous page, copies the 4 KB into it, and repoints the fork's page-table entry at the copy, now writable. The file, the parent and every other fork keep the original. A page a fork never writes stays shared for its whole life; a page it never touches has no page-table entry and no memory charge.

5. Nothing else is on the fault path. Steps 3 and 4 are the kernel's standard file-mapping fault handlers. No daemon, socket or userfaultfd is involved, which is what makes a first-touch read cost a minor fault rather than a round trip to user space.

6. Forks of forks. When a fork is snapshotted, its VMAs are still private file mappings, and CRIU dumps only the pages whose page-table entries point at private copies. That delta, typically a few hundred pages, is written on top of the same shared files when a grandchild restores. Once a fork's delta exceeds a quarter of the shared pages, the daemon rebases its snapshot: base extents plus delta are copied into fresh files and the image is rewritten as in step 1.

Where a file mapping differs from anonymous memory. Two behaviours had to be reconciled. mremap growth keeps reading the file past the original VMA, so each file is truncated to 2^47 bytes and growth lands on holes that read as zero. MADV_DONTNEED on a private file mapping restores the file's bytes rather than zeros, which glibc's thread arenas rely on when trimming a heap; each container therefore sees overcommit_memory = 2, which makes glibc trim with mmap(PROT_NONE) instead. Allocators that keep the zero-fill assumption (jemalloc, mimalloc) are listed under Limitations.

Architecture

 macOS / any client                      Linux host or Lima VM (root)
 ┌──────────────────────┐   unix socket  ┌──────────────────────────────────────────────┐
 │ lazyfork CLI         │ ──────────────▶│ lazyfork daemon (Python, HTTP/JSON)          │
 │ lazyfork.client SDK  │                │  engine: containers, snapshots, purge, gc    │
 └──────────────────────┘                │  runtime: overlayfs · cgroup v2 · namespaces │
                                         │           criu · memfile (image rewrite)     │
                                         │  store: SQLite                               │
                                         └──────────────────────────────────────────────┘
                                                      │ per container
                                         ┌────────────┴───────────────────────────────┐
                                         │ pid/mount/uts/ipc namespaces, own cgroup   │
                                         │ / = overlay (image + frozen layers + upper)│
                                         │ /.lazyfork/mem = family memory files (ro)  │
                                         └────────────────────────────────────────────┘

Use

On Linux, run lazyfork daemon run as root. On macOS, the daemon lives in the VM and these commands manage it; everything else runs from the Mac.

lazyfork daemon start           # creates and boots the VM; the first run takes a few minutes
lazyfork daemon status
lazyfork daemon clean           # deletes all stopped containers
lazyfork daemon stop
lazyfork daemon delete          # removes the VM and everything in it

Every container has a generated id, an optional name, and a parent. Commands take a name, an id, or a unique id prefix. lfork is a shorthand for lazyfork.

lazyfork create python:3.12-slim --name agent -- python3 -c 'import time; [time.sleep(1) for _ in iter(int, 1)]'
lazyfork fork agent -n 10           # ten running copies of agent, right now; agent keeps running
lazyfork fork agent --name worker   # a named copy
lazyfork fork worker -n 5           # copies of a copy
lazyfork stop agent                 # snapshot and end the process; still forkable, holds no memory
lazyfork delete worker              # its children keep running

lazyfork exec worker -- hostname    # run a command inside
lazyfork shell worker               # interactive shell inside
lazyfork attach worker              # stream its output
lazyfork freeze worker              # pause; thaw resumes

lazyfork list
lazyfork tree
lazyfork status                     # host, daemon and container totals
lazyfork stats worker               # memory, snapshot, restore timing; --csv for one row

from lazyfork.client import Client

agent = Client().container("agent")
with agent.fork(10) as forks:
    outputs = [f.exec(["hostname"])["stdout"] for f in forks]
    grandchildren = forks[0].fork(3)

Optional runtime hook, for runtimes that want a say in when they are forked:

• Listen on a unix socket inside the container, given with --hook.
• QUIESCE arrives before a snapshot, RESEED <id> after a restore.
• lazyfork/hook.py is a drop-in implementation; examples/agent.py uses it.

Benchmark

Setup:

• python:3.12-slim container running a Python agent with a 128 MB heap (137 MB of memory files).
• Snapshotted once, forked 1 to 100 times.
• Each fork runs 5 seconds, reading a few dozen pages and writing one per second, then its cgroup memory is read.
• Lineage rows: forks of forks, nine levels deep.
• VM: 4 vCPU, 8 GiB, Ubuntu 26.04 arm64, CRIU 4.2. Scripts and CSVs in bench/.

Memory a fork only reads stays shared: after 5 seconds a fork maps about 19 MB from the memory files and owns 2.5 MB of its own pages. A fork's snapshot contains only the pages it wrote (300 to 700 pages here), and its children start from the same shared files.

Limitations

• Memory is a private file mapping inside a fork, so MADV_DONTNEED returns the snapshot's bytes rather than zeros. glibc is handled (each container sees overcommit_memory = 2, which makes it shrink heaps with mmap instead), and CPython and Go never rely on it, but allocators that do, such as jemalloc and mimalloc, are not supported inside forks.
• A fork's /proc/self/maps shows its heap as files under /.lazyfork/mem.
• Snapshots live on tmpfs and do not survive a reboot. No network namespace; runtimes reconnect after a fork.