rapfiles

True async filesystem I/O — no fake async, no GIL stalls.

Overview

rapfiles provides true async filesystem I/O operations for Python, backed by Rust and Tokio. Unlike libraries that wrap blocking I/O in async syntax, rapfiles guarantees that all I/O work executes outside the Python GIL, ensuring event loops never stall under load.

Roadmap Goal: Achieve drop-in replacement compatibility with aiofiles, enabling seamless migration with true async performance. See docs/ROADMAP.md for details.

Why rap*?

Packages prefixed with rap stand for Real Async Python. Unlike many libraries that merely wrap blocking I/O in async syntax, rap* packages guarantee that all I/O work is executed outside the Python GIL using native runtimes (primarily Rust). This means event loops are never stalled by hidden thread pools, blocking syscalls, or cooperative yielding tricks. If a rap* API is async, it is structurally non-blocking by design, not by convention. The rap prefix is a contract: measurable concurrency, real parallelism, and verifiable async behavior under load.

See the rap-manifesto for philosophy and guarantees.

Features

• ✅ True async file reads and writes
• ✅ Native Rust-backed execution (Tokio)
• ✅ Zero Python thread pools
• ✅ Event-loop-safe concurrency under load
• ✅ GIL-independent I/O operations
• ✅ Verified by Fake Async Detector
• ✅ aiofiles compatibility (drop-in replacement)

Feature Categories

• File Operations - Read, write, append (text and binary)
• File Handles - Async context managers (async with)
• Directory Operations - Create, remove, list, walk directories
• File Metadata - Get file statistics, sizes, timestamps
• File Manipulation - Copy, move, rename, remove, links
• Atomic Operations - Atomic writes and moves
• File Locking - Shared/exclusive advisory locks
• Batch Operations - Concurrent read/write/copy of multiple files
• Path Operations - Synchronous path utilities (rapfiles.ospath)

Requirements

• Python 3.8+ (including Python 3.13 and 3.14)
• Rust 1.70+ (for building from source)

Installation

pip install rapfiles

Building from Source

git clone https://github.com/eddiethedean/rapfiles.git
cd rapfiles
pip install maturin
maturin develop

Development Setup

For development with testing support:

git clone https://github.com/eddiethedean/rapfiles.git
cd rapfiles
pip install -e ".[test]"

This installs the package in editable mode along with testing dependencies (pytest, pytest-asyncio, aiofiles).

Run tests:

pytest

---

Usage

Basic File Operations

import asyncio
from rapfiles import read_file, write_file

async def main():
    # Write file asynchronously (true async, GIL-independent)
    await write_file("example.txt", "Hello from rapfiles!")
    
    # Read file asynchronously (true async, GIL-independent)
    content = await read_file("example.txt")
    print(content)
    # Output: Hello from rapfiles!
    
    # Write another file
    await write_file("output.txt", content)

asyncio.run(main())

Binary Files

import asyncio
from rapfiles import read_file_bytes, write_file_bytes

async def main():
    # Write binary data
    await write_file_bytes("image.png", image_data)
    
    # Read binary data
    data = await read_file_bytes("image.png")

asyncio.run(main())

File Handles

import asyncio
from rapfiles import open

async def main():
    # Open file with async context manager
    async with open("file.txt", "r") as f:
        content = await f.read()
        print(content)
    
    # Write mode
    async with open("output.txt", "w") as f:
        await f.write("Hello, world!")
    
    # Binary mode
    async with open("image.png", "rb") as f:
        data = await f.read()
    
    # Read lines
    async with open("file.txt", "r") as f:
        line = await f.readline()
        lines = await f.readlines()

asyncio.run(main())

Concurrent File Operations

import asyncio
from rapfiles import read_file, write_file

async def main():
    # Process multiple files concurrently
    tasks = [
        write_file("file1.txt", "Content 1"),
        write_file("file2.txt", "Content 2"),
        write_file("file3.txt", "Content 3"),
    ]
    await asyncio.gather(*tasks)
    
    # Read all files concurrently
    contents = await asyncio.gather(
        read_file("file1.txt"),
        read_file("file2.txt"),
        read_file("file3.txt"),
    )
    print(contents)
    # Output: ['Content 1', 'Content 2', 'Content 3']

asyncio.run(main())

Documentation

Comprehensive documentation is available in the docs/ directory:

Feature Documentation

• Directory Operations - Creating, removing, listing, and walking directories
• File Metadata - Getting file statistics, sizes, and timestamps
• File Manipulation - Copying, moving, renaming, removing, and linking files
• Atomic Operations - Atomic file writes and moves for data integrity
• File Locking - Advisory file locks for coordinating access
• Batch Operations - Concurrent processing of multiple files
• Path Operations - Synchronous path utilities (rapfiles.ospath)
• API Reference - Complete API documentation

Project Documentation

• Roadmap - Detailed development plans and feature roadmap
• Build and Test - Local development setup instructions
• Release Notes - Version history and changes
• Changelog - Detailed changelog
• Security - Security policy and vulnerability reporting

Benchmarks

This package passes the Fake Async Detector. Benchmarks are available in the rap-bench repository.

Run the detector yourself:

pip install rap-bench
rap-bench detect rapfiles

Current Status (v0.2.0)

Phase 1 Complete ✅:

• ✅ File handle operations (AsyncFile class with async with support)
• ✅ File operations: read(), write(), readline(), readlines(), seek(), tell()
• ✅ Binary file operations: read_file_bytes(), write_file_bytes()
• ✅ Append operations: append_file()
• ✅ Directory operations: create_dir(), create_dir_all(), remove_dir(), remove_dir_all(), list_dir()
• ✅ Path checking: exists(), is_file(), is_dir()
• ✅ Directory traversal: walk_dir() for recursive directory walking
• ✅ File metadata: stat(), metadata(), FileMetadata class
• ✅ Path operations: rapfiles.ospath module (aiofiles.ospath compatible)
• ✅ aiofiles compatibility: Drop-in replacement for basic aiofiles operations

Phase 2 Complete ✅:

• ✅ File manipulation: copy_file(), move_file(), rename(), remove_file()
• ✅ Link operations: hard_link(), symlink(), canonicalize()
• ✅ Atomic operations: atomic_write_file(), atomic_write_file_bytes(), atomic_move_file()
• ✅ File locking: lock_file(), lock_file_shared() with FileLock class
• ✅ Batch operations: read_files(), write_files(), copy_files() with concurrent execution
• ✅ Comprehensive test suite: 188+ tests covering all Phase 1 and Phase 2 features
• ✅ Type stubs: Complete .pyi files for IDE support
• ✅ Code quality: Ruff formatted and linted, clippy checked

Known Limitations:

• buffering, encoding, errors, newline, closefd, opener parameters accepted for API compatibility but not yet fully implemented
• No streaming operations for large files (planned for Phase 3)
• No file watching capabilities (planned for future phases)
• No advanced I/O patterns like zero-copy (planned for future phases)

Roadmap: See docs/ROADMAP.md for planned improvements. Phase 1 (aiofiles compatibility) and Phase 2 (advanced operations) are complete. Future phases will add streaming operations and advanced optimizations.

Related Projects

• rap-manifesto - Philosophy and guarantees
• rap-bench - Fake Async Detector CLI
• rapsqlite - True async SQLite
• rapcsv - Streaming async CSV

Contributing

Contributions are welcome! Please see our contributing guidelines (coming soon).

License

MIT

For detailed release notes, see CHANGELOG.md.