ayafileio

当前是英文 | chinese version

"The fastest file I/O in Gensokyo, swift as the Wind God Maiden."
— Aya Shameimaru, always flying at full speed

Cross-platform asynchronous file I/O library using native async I/O where available.
Windows leverages IOCP (I/O Completion Ports), Linux uses io_uring (kernel 5.1+), and macOS uses Dispatch I/O (GCD) for truly non-blocking file operations.

changes

see -> CHANGES

🏆 True Async on All Three Major Platforms

ayafileio aims to be a single, unified async file API that delivers true kernel-level async I/O across Windows, Linux, and macOS.

📸 Key Features

🛠️ Installation

pip install ayafileio

System requirements:

• Python 3.10+
• Windows 7+ / Linux (kernel 5.1+ for io_uring) / macOS 10.10+
• No external dependencies, precompiled wheels available

🚀 Quick Start

import asyncio
import ayafileio

async def main():
    # Write to a file — fast as the wind
    async with ayafileio.open("example.txt", "w") as f:
        await f.write("Hello, async world!\n")

    # Read with automatic decoding
    async with ayafileio.open("example.txt", "r", encoding="utf-8") as f:
        content = await f.read()
        print(content)

    # Binary operations
    async with ayafileio.open("data.bin", "rb") as f:
        data = await f.read(1024)
        await f.seek(0, 0)

asyncio.run(main())

⚡ Performance Best Practice

ayafileio's file open/close overhead is already in the microsecond range, but for maximum performance, avoid reopening the same file in a loop.

# ❌ DO NOT DO THIS: repeated open/close in a loop
for i in range(10000):
    async with ayafileio.open("data.bin", "rb") as f:
        data = await f.read()

# ✅ DO THIS: open once, operate many times
async with ayafileio.open("data.bin", "rb") as f:
    for i in range(10000):
        await f.seek(0)
        data = await f.read()

The latter is ~6x faster — it eliminates 9999 unnecessary coroutine scheduling round-trips.

🔍 Backend Information

Check which backend is currently in use:

import ayafileio

info = ayafileio.get_backend_info()
print(info)
# Windows: {'platform': 'windows', 'backend': 'iocp', 'is_truly_async': True}
# Linux:   {'platform': 'linux', 'backend': 'io_uring', 'is_truly_async': True}
# macOS:   {'platform': 'macos', 'backend': 'dispatch_io', 'is_truly_async': True}

⚙️ Unified Configuration

ayafileio provides a unified configuration system that allows runtime tuning:

import ayafileio

# View current configuration
config = ayafileio.get_config()
print(config)

# Update configuration
ayafileio.configure({
    "io_worker_count": 8,
    "buffer_size": 131072,      # 128KB buffer
    "close_timeout_ms": 2000,
})

# Reset to defaults
ayafileio.reset_config()

Configuration Options

📚 API Reference

AsyncFile class

class AsyncFile(Generic[T]):
    def __init__(
        self, path: str | Path, mode: str = "rb",
        encoding: str | None = None,
        newline: str | None = None,
        errors: str | None = None,
        auto_flush: bool = False
    ): ...

    # 读取
    async def read(self, size: int = -1) -> T: ...
    async def readline() -> T: ...
    async def readlines(hint: int = -1) -> list[T]: ...
    async def readall() -> T: ...                        # read(-1) 别名
    async def readinto(buf: bytearray | memoryview) -> int: ...  # 零拷贝 [仅二进制]
    async def chunk(chunk_size: int, *, buf: bytearray | memoryview | None = None) -> AsyncGenerator[memoryview, None]: ...  # 流式分块 [仅二进制]
    async def read_at(offset: int, size: int = -1) -> bytes: ...  # 位置读（pread 语义），不动文件位置 [仅二进制]
    async def read_many(spans: Iterable[tuple[int, int]]) -> list[bytes]: ...  # 批量位置读，单事件循环周期提交 [仅二进制]
    async def write_at(offset: int, data: bytes | bytearray | memoryview) -> int: ...
    async def write_many(writes: Iterable[tuple[int, bytes | bytearray | memoryview]]) -> list[int]: ...

    # 写入
    async def write(self, data: str | bytes) -> int: ...
    async def writelines(lines) -> None: ...              # 批量写入

    # 位置
    async def seek(self, offset: int, whence: int = 0) -> int: ...
    async def tell() -> int: ...
    async def truncate(size: int) -> None: ...

    # 控制
    async def flush(self) -> None: ...
    async def close(self) -> None: ...

    # 属性
    @property
    def closed(self) -> bool: ...
    @property
    def name(self) -> str: ...
    @property
    def mode(self) -> str: ...

    # 状态
    def readable() -> bool: ...
    def writable() -> bool: ...
    def seekable() -> bool: ...
    def fileno() -> int: ...
    def isatty() -> bool: ...

    # 迭代器
    def __aiter__(self) -> AsyncFile[T]: ...
    async def __anext__(self) -> T: ...

Supported Modes

Configuration Functions

def configure(options: dict) -> None: ...      # Unified configuration
def get_config() -> dict: ...                   # Get current configuration
def reset_config() -> None: ...                 # Reset to defaults
def get_backend_info() -> dict: ...             # Get backend information

File Wrapping

def wrap_file(fd: int, mode: str = "rb", *, owns_fd: bool = False) -> AsyncFile[bytes]: ...

Wrap an existing file descriptor (int) or a file-like object with fileno() as an AsyncFile, backed by the optimal platform backend. Binary mode only.

Pool Management

def drain_handle_pool() -> None: ...            # Drain all cached file handles
def drain_buffer_pool() -> None: ...            # Drain all cached I/O buffers

Use drain_handle_pool() / drain_buffer_pool() to release pooled resources at runtime — useful after bulk tempfile operations or between benchmark rounds.

🧪 Performance Comparison

Scenario 1: Crawlee-style Dataset Append (open/write/close per record)

Simulating Crawlee's Dataset append pattern — 5,000 records, 50 concurrent writers, each writing a single line and closing the file:

Key findings:

• On Windows enterprise SSD, ayafileio achieves 42x lower P99 latency (0.044ms vs 1.854ms)
• aiofiles shows 96.7% jitter under load; ayafileio only 16.2%
• Even on degraded hardware, ayafileio maintains predictable performance

Test environment: Windows 10/11, Ubuntu 22.04, macOS 14; GitHub Actions enterprise NVMe SSD

Scenario 2: Single-file High-concurrency Random Read

The true test of async I/O — 100,000 concurrent tasks performing random 256B reads on a single shared file handle. No open/close overhead, pure I/O path comparison:

Key findings:

• At low concurrency (1K), all approaches are similar — IOCP setup overhead is amortized
• At 10K+ concurrency, aiofiles' thread pool saturates; throughput drops as concurrency increases — from 7,706 down to 2,130 ops/s (72% degradation)
• ayafileio with IOCP gains throughput at 10K (46,616 ops/s) due to batched completion harvesting via GetQueuedCompletionStatusEx
• At 100K concurrency, ayafileio is 9.1x faster than aiofiles on the same HDD
• The synchronous threadpool (mimicking aiofiles' approach) flatlines at ~8,800 ops/s regardless of concurrency — thread contention ceiling

Test environment: Windows 10, Python 3.14.5, WDC WD10EZEX 7200RPM HDD, 20MB file, 256B random reads

Scenario 3: Extreme Concurrency Stress — 500,000 Concurrent Reads

500,000 asyncio tasks all reading from a single file via IOCP — testing the library's absolute concurrency ceiling:

ayafileio handles half a million concurrent IOCP reads on a single file handle with zero errors. The dual-IOCP worker architecture (2 threads total) processes all 500K completions while aiofiles would require thousands of threads for the same workload — and still be slower.

Tuning: Default Configuration is Optimal

We tested 14 different configuration combinations (iocp_batch_size, buffer_size, buffer_pool_max, io_worker_count) on the HDD at 100K concurrency. Result: every configuration scored within ±3% of the default. The library's auto-tuned defaults already saturate the disk's physical I/O limit — there is no software bottleneck left to tune.

For NVMe SSDs with >500K IOPS capability, increasing iocp_batch_size to 128–256 and buffer_size to 128KB may yield additional gains. Use ayafileio.configure() to experiment:

ayafileio.configure({
    "iocp_batch_size": 128,
    "buffer_size": 131072,
})

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Add tests
4. Ensure benchmarks pass
5. Open a pull request

📄 License

MIT License — see LICENSE for details.

"Slow is a crime, right?"
— Aya Shameimaru, editor-in-chief of Bunbunmaru News

Positioned and batched writes

async with ayafileio.open("data.bin", "w+b") as f:
    n = await f.write_at(1024, b"payload")
    counts = await f.write_many([(0, b"header"), (4096, b"block")])
    assert await f.tell() == 0

write_at(offset, data) returns the number of bytes written. write_many(writes) accepts an iterable of (offset, data) pairs and returns counts in input order. Both require a writable binary file, reject append mode, and preserve the logical file position. They accept bytes, bytearray, and contiguous memoryview buffers, which the backend copies on submission. Positioned writes rewind and discard any readline() read-ahead so subsequent reads see the updated content. An empty batch returns []; an empty buffer returns 0. Offsets must be nonnegative, offset plus length must fit a signed 64-bit integer, and each buffer is limited to 4 GiB minus 1 byte. As with write(), callers must check for short writes.

Batch operations gather native Futures directly, avoiding a Python Task per item; read_many() uses the same optimization. Submitted requests are drained before reporting submission or I/O errors, but batches are not atomic: some writes may have succeeded when an error is raised. Concurrent overlapping writes have unspecified ordering; await each write_at(...) separately when ordering matters. Cancelling the wait does not undo already submitted system I/O.