ayafileio

English | 简体中文

"The fastest file I/O in Gensokyo, swift as the Wind God Maiden." — Aya Shameimaru

Async file I/O on real kernel backends: IOCP on Windows, io_uring on Linux (5.1+), Dispatch I/O on macOS. No thread pools pretending to be async; ayafileio.open() picks the backend for you.

Changelog

See CHANGES.md. For the 2.0 architecture and dependency rules, see architecture.md. For the full usage guide, see USAGE.md. Using ayafileio in an interesting way? See STORY.md.

Backends

Features

• Kernel backends avoid Python executor offloading
• Kernel-level completion: IOCP / io_uring / Dispatch I/O
• Cache-hit inline fast paths: hot small reads complete on the calling thread (IOCP sync-completion / io_uring COOP_TASKRUN / macOS mincore + pread)
• Thousands of concurrent operations on a single file handle
• aiofiles-compatible API, plain async/await
• Text and binary modes with automatic encoding/decoding
• Async whole-file copy: acopy() with OS-level fast paths (zero-copy on Linux)
• Runtime-tunable configuration shared by all backends
• Python 3.10–3.15, including 3.14t/3.15t free-threading

Installation

pip install ayafileio

Requires Python 3.10+, Windows 7+ / Linux (kernel 5.1+ for io_uring) / macOS 10.10+. No external dependencies; precompiled wheels are available.

Quick start

import asyncio
import ayafileio

async def main():
    # Write to a file — fast as the wind
    async with ayafileio.open("example.txt", "w") as f:
        await f.write("Hello, async world!\n")

    # Read with automatic decoding
    async with ayafileio.open("example.txt", "r", encoding="utf-8") as f:
        content = await f.read()
        print(content)

    # Binary operations
    async with ayafileio.open("data.bin", "rb") as f:
        data = await f.read(1024)
        await f.seek(0, 0)

asyncio.run(main())

Open once, read many times

Open/close overhead is already in the microsecond range, but reopening the same file in a loop still costs you a coroutine round-trip per iteration:

# Slow: repeated open/close
for i in range(10000):
    async with ayafileio.open("data.bin", "rb") as f:
        data = await f.read()

# Fast: one handle, many operations (~6x faster)
async with ayafileio.open("data.bin", "rb") as f:
    for i in range(10000):
        await f.seek(0)
        data = await f.read()

Checking the active backend

import ayafileio

print(ayafileio.get_backend_info())
# Windows: {'platform': 'windows', 'backend': 'iocp', 'os_version': '10.0.19045', ...}
# Linux:   {'platform': 'linux', 'backend': 'io_uring', 'os_version': 'Linux 6.5.0-...', ...}
# macOS:   {'platform': 'macos', 'backend': 'dispatch_io', 'os_version': 'Darwin 24.5.0', ...}

Capability discovery

get_capabilities() returns everything in get_backend_info() plus two extra dicts: features (library-level capabilities with uniform keys across platforms) and backend_detail (natively probed capabilities of the active backend). Probing asks the kernel directly — io_uring opcodes are checked via IORING_REGISTER_PROBE rather than version-number tables, so backported kernels report what they actually support. Results are cached process-wide.

caps = ayafileio.get_capabilities()

caps["features"]
# {'positional_io': True, 'batch_positional_io': True, 'zero_copy_readinto': True,
#  'chunked_streaming': True, 'async_open': True, 'adaptive_batching': True,
#  'fast_file_copy': True}   # fast_file_copy: copy_file_range / CopyFile2 available

caps["backend_detail"]   # keys depend on the backend, e.g. Linux io_uring:
# {'available': True, 'completion_model': True, 'sqpoll': True,
#  'features': {'single_mmap': True, 'fast_poll': True, ...},
#  'opcodes': {'read': True, 'write': True, 'statx': True, ...}}  # None on kernels < 5.6

# Typical use: gate optional features on what the kernel actually offers
if caps["backend_detail"].get("sqpoll"):
    ayafileio.configure({"io_uring_sqpoll": True})

Configuration

All backends share one runtime configuration:

import ayafileio

print(ayafileio.get_config())

ayafileio.configure({
    "io_worker_count": 8,
    "buffer_size": 131072,      # 128 KB
    "close_timeout_ms": 2000,
})

ayafileio.reset_config()  # back to defaults

API reference

AsyncFile

class AsyncFile(Generic[T]):
    def __init__(
        self, path: str | Path, mode: str = "rb",
        encoding: str | None = None,
        newline: str | None = None,
        errors: str | None = None,
        auto_flush: bool = False
    ): ...

    # Reading
    async def read(self, size: int = -1) -> T: ...
    async def readline() -> T: ...
    async def readlines(hint: int = -1) -> list[T]: ...
    async def readall() -> T: ...                        # alias for read(-1)
    async def readinto(buf: bytearray | memoryview) -> int: ...  # zero-copy [binary only]
    async def chunk(chunk_size: int, *, buf: bytearray | memoryview | None = None) -> AsyncGenerator[memoryview, None]: ...  # streaming chunks [binary only]
    async def read_at(offset: int, size: int = -1) -> bytes: ...  # positioned read (pread), leaves file position untouched [binary only]
    async def read_many(spans: Iterable[tuple[int, int]]) -> list[bytes]: ...  # batched positioned reads, submitted in one event-loop turn [binary only]
    async def write_at(offset: int, data: bytes | bytearray | memoryview) -> int: ...
    async def write_many(writes: Iterable[tuple[int, bytes | bytearray | memoryview]]) -> list[int]: ...

    # Writing
    async def write(self, data: str | bytes) -> int: ...
    async def writelines(lines) -> None: ...

    # Position
    async def seek(self, offset: int, whence: int = 0) -> int: ...
    async def tell() -> int: ...
    async def truncate(size: int) -> None: ...

    # Control
    async def flush(self) -> None: ...
    async def close(self) -> None: ...

    # Properties
    @property
    def closed(self) -> bool: ...
    @property
    def name(self) -> str: ...
    @property
    def mode(self) -> str: ...

    # State
    def readable() -> bool: ...
    def writable() -> bool: ...
    def seekable() -> bool: ...
    def fileno() -> int: ...
    def isatty() -> bool: ...

    # Iteration
    def __aiter__(self) -> AsyncFile[T]: ...
    async def __anext__(self) -> T: ...

Supported modes

Positioned and batched writes

async with ayafileio.open("data.bin", "w+b") as f:
    n = await f.write_at(1024, b"payload")
    counts = await f.write_many([(0, b"header"), (4096, b"block")])
    assert await f.tell() == 0

write_at(offset, data) returns the number of bytes written. write_many(writes) accepts an iterable of (offset, data) pairs and returns counts in input order. Both require a writable binary file, reject append mode, and preserve the logical file position. They accept bytes, bytearray, and contiguous memoryview buffers. Writes are zero-copy: the backend holds the caller's buffer until the write completes, so don't resize or modify it in place while the write is in flight (same contract as readinto()). Positioned writes rewind and discard any readline() read-ahead so subsequent reads see the updated content. An empty batch returns []; an empty buffer returns 0. Offsets must be nonnegative, offset plus length must fit a signed 64-bit integer, and each buffer is limited to 4 GiB minus 1 byte. As with write(), callers must check for short writes.

read_many() submits the whole batch through the native backend and hands back one aggregate Future, so a batch costs a single Python round-trip regardless of span count; write_many() gathers native Futures directly, avoiding a Python Task per item. Submitted requests are drained before reporting submission or I/O errors, but batches are not atomic: some writes may have succeeded when an error is raised. Concurrent overlapping writes have unspecified ordering; await each write_at(...) separately when ordering matters. Cancelling the wait does not undo already submitted system I/O.

Copying files

n = await ayafileio.acopy("model.gguf", "backup/model.gguf")

acopy(src, dst, *, chunk_size=4 MiB, concurrency=8, copy_stat=False) copies a whole file and returns the number of bytes copied. On Linux it uses the in-kernel zero-copy copy_file_range; on Windows CopyFile2, which also preserves metadata. Both run in a worker thread. Where no fast call exists (or the filesystem rejects it), it falls back to a read_at/write_at pipeline with up to concurrency chunks in flight, so memory stays below concurrency × chunk_size. The destination is truncated first; copying a file onto itself raises shutil.SameFileError; copy_stat=True applies shutil.copystat afterwards.

Whole-file operations

data = await ayafileio.read_bytes("model.safetensors")
n    = await ayafileio.write_bytes("out.bin", payload)
text = await ayafileio.read_text("config.yaml", encoding="utf-8")

One await does open + read/write + close — aimed at asyncio.gather over many files (per-layer weight loading, bulk config reads). Semantics are identical to the hand-written async with ayafileio.open(...) form, minus the boilerplate.

Async open

async with ayafileio.aopen("data.bin", "rb") as f:
    data = await f.read()

aopen() takes the same arguments as open() and returns the same AsyncFile — the difference is that the OS open itself runs on the C++ thread pool instead of the event-loop thread. The returned handle is lazy (aiofiles style): async with triggers the open in __aenter__, and await ayafileio.aopen(...) works too. Opening many files this way parallelizes the open syscalls themselves; a plain synchronous open() in the loop still costs one short blocking syscall per file.

Batch whole-file operations

shards  = await ayafileio.read_bytes_many(model_shard_paths)
configs = await ayafileio.read_text_many(config_paths, encoding="utf-8")
counts  = await ayafileio.write_bytes_many([(p, payload) for p, payload in items])
counts  = await ayafileio.write_text_many(items, encoding="utf-8")

One call reads (or writes) many files concurrently: parallel opens via aopen(), async I/O via the platform backend, results in input order (write variants take (path, data) pairs and return byte counts). max_concurrency (default 64) caps in-flight files so huge inputs can't exhaust fds; any failure raises the whole call — batches are not atomic, so a failed write batch may leave some files already written. This is the shortest way to write the pattern that actually uses the async backend — a sequential for loop over open() moves one file at a time no matter how fast each operation is.

Configuration functions

def configure(options: dict) -> None: ...      # apply unified configuration
def get_config() -> dict: ...                  # current configuration
def reset_config() -> None: ...                # reset to defaults
def get_backend_info() -> dict: ...            # active backend information
def get_capabilities() -> dict: ...            # capability matrix (features + backend_detail)

Wrapping an existing file

def wrap_file(fd: int, mode: str = "rb", *, owns_fd: bool = False) -> AsyncFile[bytes]: ...

Wraps an existing file descriptor (int) or a file-like object with fileno() as an AsyncFile. Binary mode only.

Pool management

def drain_buffer_pool() -> None: ...           # release all cached I/O buffers

Useful after bulk tempfile operations or between benchmark rounds.

Benchmarks

Every push runs benchmarks/t_compare.py on all three platforms on CI and attaches the JSON results as artifacts. Numbers below are from a recent run (GitHub Actions, Python 3.14) unless marked local. Reproduce with python benchmarks/t_compare.py.

Random 4 KiB read (Linux, io_uring)

Positioned read_at on a single shared handle holds ~345K ops/s from x16 up; aiofiles has no positioned-read API to compare against. Small 4 KiB sequential reads: 201.6K vs 21.9K ops/s (9.2x). Sequential read throughput: 7,360 MB/s at 64 KiB blocks (6.6x), 20,178 MB/s at 4 MiB (1.3x).

Sequential writes are closer: 1.2–1.3x up to 256 KiB blocks, and aiofiles pulls ahead at 4 MiB (0.4x) on the CI runner — large buffered writes ride the kernel's writeback path and swing between runs. If that workload matters to you, check a recent run's artifact rather than any single number.

Windows and macOS

GitHub's Windows runners throttle raw sequential throughput too hard for those CI cells to be representative — the last windows-2022 run measured 64 KiB sequential read at 0.79x and 4 KiB small reads at 0.62x over aiofiles — so the headline Windows figures remain local NVMe measurements. The concurrency cells do run healthy on that same runner since the 1.11.0 accounting fix (random 4 KiB reads, per-task handles): 12.0K vs 7.4K ops/s at x1 (1.6x), 28.3K vs 5.4K at x16 (5.3x), 31.3K vs 8.7K at x64 (3.6x); sequential writes lead everywhere (64 KiB: 60.9K vs 8.2K ops/s, 7.4x).

• Windows (IOCP), sequential write: 1,363 → 2,858 MB/s at 1 MiB blocks, 929 → 3,046 MB/s at 4 MiB (1.47x over aiofiles). 4 KiB random read_at with 16 in flight: 21.4K ops/s.
• macOS (Dispatch I/O): cache-hit small reads now complete inline on the calling thread (mincore + pread) — 4 KiB sequential read 6.8K → 499K ops/s (27.2x over aiofiles), 4 KiB random read at x1 4.6K → 432K ops/s (54.2x; x64: 416K, 25.7x), 64 KiB sequential read 384 → 6,441 MB/s (6.7x); the write path is untouched (4 MiB sequential write 8.2 GB/s). Measured on the macos-15 CI runner (Python 3.14, benchmarks/t_compare.py methodology); the small-read cells swing ~2x run-to-run on this shared runner — these are the latest run's numbers, after the macOS fd-leak fix stopped taxing every open/close cycle.

Whole-file copy (acopy, local, 512 MiB cache-hot file)

Concurrent logging writes (tests/python/test_loguru.py)

10K records × 128 B written concurrently to 5 files: 3.5–5.2x over sync-writes-in-a-threadpool across runs.

Tuning

The defaults reflect where tuning converged. On very fast NVMe drives, iocp_batch_size 128–256 with buffer_size 128 KiB can squeeze out a bit more on Windows.

Contributing

Contributions welcome: fork, branch, add tests, make sure the benchmarks still pass, open a PR.

License

MIT — see LICENSE.

"Slow is a crime, right?" — Aya Shameimaru, editor-in-chief of Bunbunmaru News