turbofile

Real async file I/O for Python. A Rust core drives the best completion mechanism each OS has — io_uring on Linux, POSIX AIO on macOS — behind an aiofiles-compatible asyncio API.

Existing Python async-file libraries dispatch every call to a thread pool. turbofile submits the I/O to the kernel and completes your await when the kernel says the data moved:

• Zero-copy reads and writes. The kernel fills the bytes object your await f.read(n) returns; writes pin your buffer and hand the kernel its pointer. No intermediate copies on the hot paths.
• Completion batching. A burst of completions costs one event-loop wakeup (one call_soon_threadsafe doorbell per burst, drained entirely in Rust).
• Parallel large reads. Read-all on a large file is split into chunks filled concurrently into one buffer.
• Whole-file ops. read_bytes/write_bytes do open+read/write+close as a single submission — one round trip per file.

import turbofile

async def main() -> None:
    async with turbofile.open("data.bin", "rb") as f:
        payload = await f.read()

    async with turbofile.open("log.txt", "a", encoding="utf-8") as f:
        await f.write("one line\n")

    data = await turbofile.read_bytes("data.bin")   # whole file, one op
    text = await turbofile.read_text("notes.md")

turbofile.open mirrors aiofiles.open: binary and text modes, buffering, encoding/errors/newline with full universal-newline semantics, seek/tell (text cookies follow CPython's _pyio scheme), readline, async iteration, readinto, truncate, fsync via sync. Migration is import turbofile as aiofiles for the open API.

Benchmarks

make bench compares against aiofiles on your machine: p50 per operation, median of three runs. On an Apple-silicon Mac (macOS 26.4, POSIX AIO backend, page-cache-hot files, mains power):

Large sequential transfers are memory-bandwidth-bound in the page cache, so every implementation converges there; turbofile wins where per-op overhead and concurrency dominate, which is what an asyncio application does.

Reads whose pages are already resident never leave the event-loop thread: no submission, no driver-thread hop, no completion wakeup. On Linux the kernel decides, through preadv2(RWF_NOWAIT); on macOS mincore on a mapping of the file reports which pages the buffer cache holds, and a plain pread copies them. Data that is not resident takes the async path. make ladder isolates that read: its file_read rung is BinaryFile's positional read of a hot 4 KiB, its pread rung the blocking syscall for the same bytes.

On Linux read_bytes takes the same inline route for the whole file, open included, through openat2(RESOLVE_CACHED): 5.3 us for a hot 4 KiB file. macOS has no cached-only open, and an inline open there stalls on cold paths and endpoint-security scans (about 300 us for the first open of a file on a machine running Jamf Protect), so read_bytes keeps its open and close on the driver thread.

Benchmark on a real filesystem: tmpfs sets no FMODE_NOWAIT, which disables the fast path and makes io_uring punt every read to a kernel worker, so /tmp numbers are not comparable. make bench and make ladder take FS=<dir> and print which filesystem they measured. See perf/README.md for the analysis harness.

Backends

TURBOFILE_BACKEND=compio selects the compio driver on macOS (benchmarking, or as an escape hatch). Windows (IOCP via compio) is planned.

Limitations

• opener= and integer file descriptors are not supported.
• read_bytes on very large files pays one buffer copy; prefer open(...).read() for multi-megabyte files.
• Cancelling an await detaches the future; the kernel op still completes (and, for readinto, may still write into the buffer) — standard completion-model semantics.

Development

make develop   # build the extension into the venv (uv + maturin)
make test      # cargo test + pytest
make lint      # clippy -D warnings
make bench     # benchmark against aiofiles

License

MIT OR Apache-2.0