aiogzip 1.6.0

Asynchronous gzip file reader/writer with aiocsv support.

aiogzip ⚡️

An asynchronous library for reading and writing gzip-compressed files.

aiogzip provides a fast, simple, and asyncio-native interface for handling .gz files, making it a useful complement to Python's built-in gzip module for asynchronous applications.

🚀 Read the Documentation

Features

• Truly Asynchronous: Built with asyncio and aiofiles.
• High-Performance: Optimized buffer handling for fast I/O.
• Drop-in Replacement: Mimics gzip.open() with async seek, tell, peek, and readinto support; verified against tarfile-style access patterns and aiocsv workflows.
• Reproducible Archives: Control gzip mtime and embedded filenames.
• Type-Safe: Distinct AsyncGzipBinaryFile and AsyncGzipTextFile.
• aiocsv Ready: Seamless integration for CSV pipelines.
• Predictable Performance: Backward seeks rewind the stream and re-decompress data (same as gzip.GzipFile), so treat random access as O(n) and prefer forward-only patterns when possible.

Append mode and large files

• Append mode ("ab", "at") writes a new gzip member. The file ends up as two (or more) concatenated gzip members. Every standards-compliant reader — including aiogzip, gzip.open(), and command-line gunzip — transparently concatenates the output, but each additional open writes a new member rather than extending the existing deflate stream.
• Backward seeks restart decompression from the beginning of the file, so forward-only access is much faster than mixed-direction access.
• Non-seekable input streams use a bounded rewind cache. By default, up to 128 MiB of compressed input is retained so backward seeks can replay the stream; pass max_rewind_cache_size=<bytes> to tune this, or None to allow an unbounded cache.
• Writes past 4 GiB of uncompressed data produce a gzip trailer whose ISIZE field wraps to size & 0xFFFFFFFF (this matches the gzip format spec and gzip.open()). Pass strict_size=True to refuse writes that would exceed the limit instead.
• Guard against decompression bombs by passing max_decompressed_size=<bytes> when reading untrusted files; the decompressor aborts with OSError once the cap is exceeded.
• Use one file object per task. An open aiogzip file is not safe for concurrent use by multiple asyncio tasks — its internal buffers and decoder/compressor state are mutated without locking, the same contract as standard-library file objects. Give each task its own file object, or serialize access behind your own lock.

Quickstart

pip install aiogzip

import asyncio
from aiogzip import AsyncGzipFile

async def main():
    # Write
    async with AsyncGzipFile("file.gz", "wb") as f:
        await f.write(b"Hello, async world!")

    # Read
    async with AsyncGzipFile("file.gz", "rb") as f:
        print(await f.read())

asyncio.run(main())

# Deterministic metadata
async with AsyncGzipFile(
    "dataset.gz", "wb", mtime=0, original_filename="dataset.csv"
) as f:
    await f.write(b"stable bytes")

Performance

• Text I/O: Often ~2-3x faster than standard gzip in bulk text workflows.
• Binary I/O: Near parity with gzip for bulk writes, with fast bulk reads (a full read(-1) of compressible data runs at several hundred MB/s); can be slower for very small chunk sizes.
• Concurrency: CPU-heavy zlib compress/decompress calls run in the default executor above a 256 KiB threshold, so multiple gzip streams on the same event loop compress and decompress in parallel instead of serializing on the loop thread. The repo's concurrent-I/O benchmark runs ~4x faster on 1.4.0 than on 1.3.x as a result; single-stream throughput stays at parity.
• Memory: Optimized buffer management for stable memory usage.
• JSONL: For large gzipped JSONL files, prefer AsyncGzipTextFile(..., newline="\n", chunk_size=512 * 1024) to reduce line-iteration overhead.

See the Performance Guide for detailed benchmarks.

Contributing

See CONTRIBUTING.md for development instructions.