turbofile

Real async file I/O for Python.
A Rust core drives the best completion mechanism each OS has (io_uring on Linux, POSIX AIO on macOS) behind an aiofiles-compatible asyncio API.

Installation · Usage · Benchmarks · Backends · Limitations · Development

Existing Python async-file libraries dispatch every call to a thread pool. turbofile submits the I/O to the kernel and completes your await when the kernel says the data moved:

• Zero-copy reads and writes. The kernel fills the bytes object your await f.read(n) returns; writes pin your buffer and hand the kernel its pointer. No intermediate copies on the hot paths.
• Completion batching. A burst of completions costs one event-loop wakeup: a byte on a pipe the loop watches, so the driver thread never takes the GIL to wake the loop, and the batch is drained entirely in Rust.
• Parallel large reads. Read-all on a large file is split into 512 KiB chunks filled concurrently into one buffer, which spreads the page-cache copy across the kernel's I/O threads and beats a single blocking read.
• Whole-file ops. read_bytes/write_bytes do open+read/write+close as a single submission: one round trip per file. Above 1 MiB, read_bytes takes the parallel-chunk fill instead, so the kernel writes the returned bytes directly with no intermediate copy.

Installation

First release pending; until it's on PyPI, build from source:

git clone https://github.com/4thel00z/turbofile
cd turbofile
make develop   # uv + maturin, builds the extension into the venv

Usage

import turbofile

async def main() -> None:
    async with turbofile.open("data.bin", "rb") as f:
        payload = await f.read()

    async with turbofile.open("log.txt", "a", encoding="utf-8") as f:
        await f.write("one line\n")

    data = await turbofile.read_bytes("data.bin")   # whole file, one op
    text = await turbofile.read_text("notes.md")

turbofile.open mirrors aiofiles.open: binary and text modes, buffering, encoding/errors/newline with full universal-newline semantics, seek/tell (text cookies follow CPython's _pyio scheme), readline, async iteration, readinto, truncate, fsync via sync. Migration is import turbofile as aiofiles for the open API.

Inside a running loop, read_bytes and write_bytes return the completion future itself rather than a coroutine, so asyncio.gather over many files schedules no task per file. They can still be awaited directly or wrapped in asyncio.create_task; outside a loop they return a coroutine, so asyncio.run(turbofile.read_bytes(path)) works too.

f.read_at(pos, n) and f.readinto_at(pos, buffer) read at a position without moving the file's own. Inside a running loop they return the completion future as well, and pages already resident settle it at once, so asyncio.gather over many positions reads them concurrently with no task per read.

Benchmarks

make bench compares against aiofiles on your machine: p50 per operation, median of three runs. On an Apple-silicon Mac (macOS 26.4, POSIX AIO backend, page-cache-hot files, mains power):

Both 8 MiB reads finish in about 0.31 ms, against 0.27 ms for a blocking read of the same bytes and 0.44 ms for aiofiles: at that size the copy is the floor and aiofiles pays its thread hop on top of it. The sixteen chunks copying in parallel on the kernel's AIO threads show at 64 MiB, where the parallel fill takes 2.7 ms against 8.9 ms for a read in a thread. Sequential writes await one chunk at a time, so one copy is in flight and the write row sits at the copy as well; turbofile wins where per-op overhead and concurrency dominate, which is what an asyncio application does.

Reads whose pages are already resident never leave the event-loop thread: no submission, no driver-thread hop, no completion wakeup. On Linux the kernel decides, through preadv2(RWF_NOWAIT); on macOS mincore on a mapping of the file reports which pages the buffer cache holds, and a plain pread copies them. Data that is not resident takes the async path. The table's 32 concurrent random reads take this route through read_at. make ladder isolates the read: its file_read rung is BinaryFile's positional read of a hot 4 KiB, its pread rung the blocking syscall for the same bytes.

On Linux read_bytes takes the same inline route for the whole file, open included, through openat2(RESOLVE_CACHED): 5.3 us for a hot 4 KiB file. macOS has no cached-only open, and an inline open there stalls on cold paths and endpoint-security scans (about 300 us for the first open of a file on a machine running Jamf Protect), so read_bytes keeps its open and close on the driver thread.

Benchmark on a real filesystem: tmpfs sets no FMODE_NOWAIT, which disables the fast path and makes io_uring punt every read to a kernel worker, so /tmp numbers are not comparable. make bench and make ladder take FS=<dir> and print which filesystem they measured. See perf/README.md for the analysis harness.

Backends

TURBOFILE_BACKEND=compio selects the compio driver on macOS (benchmarking, or as an escape hatch). Windows (IOCP via compio) is planned.

Limitations

• opener= and integer file descriptors are not supported.
• Cancelling an await sends a kernel-level abort request (aio_cancel on macOS; the compio driver stops before the next chunk), best-effort by nature: an op the kernel finishes first settles with its result. Either way the CancelledError arrives only once the op has settled, so your buffer is never touched after the await raises.

Development

make develop   # build the extension into the venv (uv + maturin)
make test      # cargo test + pytest
make lint      # clippy -D warnings
make bench     # benchmark against aiofiles

License

MIT OR Apache-2.0