aiogzip ⚡️

An asynchronous API modeled after Python's gzip module for reading and writing gzip-compressed files.

It can substantially outperform sequential gzip when async work overlaps or optional zlib-ng accelerates bulk decompression; direct single-file line iteration remains faster with synchronous gzip. See Performance and optional acceleration.

Installation

pip install aiogzip

The 2.0 alpha series requires Python 3.11 or newer. Python 3.8 through 3.10 users should remain on the latest compatible 1.x release.

Text-mode quickstart

File methods are asynchronous, and line iteration uses async for:

import aiogzip

async with aiogzip.open("events.jsonl.gz", "rt") as f:
    async for line in f:
        print(line)

Binary-mode quickstart

import aiogzip

async with aiogzip.open("payload.bin.gz", "wb") as f:
    await f.write(b"Hello, async world!")

async with aiogzip.open("payload.bin.gz", "rb") as f:
    payload = await f.read()

For small files that comfortably fit in memory, use the binary whole-file helpers:

import aiogzip

data = await aiogzip.read("payload.bin.gz")
await aiogzip.write("copy.bin.gz", data)

read() and write() load the entire decompressed or uncompressed payload into memory. Use open() to stream large files. The existing AsyncGzipFile() factory remains fully supported for compatibility.

For arbitrary asynchronous byte sources, compress or decompress without adapting the source to a file object:

async for data in aiogzip.decompress_chunks(compressed_source()):
    await consume(data)

async for data in aiogzip.compress_chunks(raw_source(), mtime=0):
    await send(data)

Complete decompression integrity validation occurs only if the iterator is consumed to the end. Compression output is incomplete if its source fails or the consumer exits early. See the async-iterable streaming guide for backpressure, limits, cancellation, metadata, and lifecycle behavior.

For synchronous custom transports, the provisional 2.0 alpha codec performs gzip framing and validation without I/O or executor offload:

import aiogzip

encoder = aiogzip.GzipEncoder(mtime=0)
wire = b"".join(encoder.start())
wire += b"".join(encoder.feed(b"payload"))
wire += b"".join(encoder.finish())

decoder = aiogzip.GzipDecoder()
payload = b"".join(decoder.feed(wire)) + b"".join(decoder.finish())

Every returned CodecOperation must be exhausted before the next codec call; its idempotent close() method handles deterministic early abandonment. Decoder integrity is established only after finish() is exhausted. See the synchronous codec guide for ownership, limits, immutable inputs, and thread-safety details.

See the recipes for JSON Lines, untrusted input, reproducible output, append mode, seeking, cancellation recovery, and external async streams.

Maintained integration examples

Two credential-free examples exercise complete application workflows using only aiogzip's public API:

• fragmented_transport.py drives the synchronous codec over bounded, explicitly length-prefixed transport frames and keeps decoded records provisional until trailer validation succeeds.
• concurrent_jsonl_ingest.py processes independent gzip shards with bounded concurrency and publishes staged JSONL output only after every shard validates.

See the example runbook for requirements, clean-checkout commands, wheel-installed commands, failure scenarios, and the boundary between example application code and supported aiogzip API.

Why use aiogzip?

• Async file I/O built on asyncio and aiofiles, so independent streams can overlap I/O waits.
• Binary and text modes with distinct, typed concrete classes.
• Async read, write, readline, seek, tell, peek, readinto, and line iteration.
• Interoperable gzip output, concatenated-member reads, and append support.
• Configurable gzip metadata for reproducible archives.
• Bounded decompression and rewind-cache controls for untrusted or non-seekable input.
• Pull-driven compression and decompression for AsyncIterable[bytes] sources.
• A synchronous sans-I/O gzip codec for custom transports.
• Optional zlib-ng acceleration without a required runtime dependency.
• Verified tarfile-style access patterns and aiocsv workflows.

Migrating from gzip

The API follows gzip, but file operations must be awaited:

Synchronous code:

import gzip

with gzip.open("events.jsonl.gz", "rt") as f:
    for line in f:
        process(line)

becomes asynchronous code:

import aiogzip

async with aiogzip.open("events.jsonl.gz", "rt") as f:
    async for line in f:
        await process(line)

Important differences and caveats:

• aiogzip defaults to compresslevel=6; gzip.open() defaults to 9. Pass compresslevel=9 when that parity matters.
• Paths (including pathlib.Path) are accepted directly. Supported external asynchronous sources and destinations are passed with filename=None and fileobj=...; their read() or write() methods must be async.
• Append modes ("ab" and "at") create a new gzip member. Both libraries transparently read concatenated members as one decompressed stream.
• One logical task should own an open handle at a time. Separate handles may progress concurrently; serialize intentional shared-handle access with an application lock covering the complete logical operation.
• Gzip has no random-access index. Backward seeks rewind and replay decompression, so mixed-direction access can be O(n).
• Cancelling an executor-backed decompression can leave that reader unusable; close it and open a new handle before continuing.

Compatibility and operational behavior

aiogzip reads and writes standard gzip streams and supports text and binary modes, tarfile-style reads, aiocsv, append mode, and concatenated members. It is an asynchronous API modeled after gzip, not a synchronous drop-in replacement.

• Lifecycle: Prefer async with. When that is impractical, call await f.open() and pair it with await f.close() in finally.
• Seeking: Backward seeks replay decompression from the start. Forward access is fastest. Text tell() may return a handle-bound opaque cookie when decoder state is buffered; do not persist that cookie across reopens.
• Non-seekable sources: Up to 128 MiB of compressed input is cached by default for replay. Tune max_rewind_cache_size, or pass None for an unbounded cache.
• Untrusted input: max_decompressed_size caps cumulative decompressed output for a read pass. Overflow raises OSError without first materializing the complete expansion.
• Task safety: One logical task owns one open handle at a time. Separate handles may progress concurrently. An overlapping call raises ConcurrentOperationError before state corruption; it is a misuse signal, not a lock or retry-based synchronization primitive. Use an application lock around the complete shared operation when ownership cannot be separated.
• Cancellation: If cancellation occurs during executor-backed decompression, later reads and seeks raise OSError. Close and reopen the reader. A similarly cancelled compression makes that output member unusable; discard it and start a new writer.
• Append mode: Each append creates another member instead of extending the existing deflate stream. Standards-compliant readers concatenate the members.
• Large writes: Gzip's 32-bit ISIZE wraps after 4 GiB, as it does in gzip.open(). Pass strict_size=True to reject a member that would cross that limit.
• Compression metadata: mtime and the embedded original filename affect output bytes. Set both explicitly when reproducibility across paths matters.
• Boolean options: Pass exact True or False for fast_compress and strict_size; closefd also accepts None for its ownership default. Integer substitutes such as 0 and 1 raise TypeError before a resource is opened. The direct codec applies the same rule to collect_member_info.

Performance and optional acceleration

aiogzip's performance advantage comes from async concurrency and optional codec acceleration. Comparisons use identical compressed fixtures for reads, compression level 6 for both writers, and median timings from repeated runs.

On a representative Python 3.12 Linux run, the direct I/O cases used 8 MiB inputs and the concurrency case used ten 1 MiB files:

• overlapping ten files with simulated 10 ms latency was about 6.4-6.9x faster than processing them sequentially with gzip;
• optional zlib-ng made a highly compressible bulk read(-1) about 14.5x faster than gzip, and stdlib zlib finished slightly ahead (~1.08x);
• an LF-only universal-newline fast path made the representative zlib-ng bulk text read about 1.9x faster than gzip (the stdlib engine was about 1.25x slower);
• bounded readlines() batches brought an 8 MiB JSONL read-and-parse workload to parity with gzip on both engines and about 9-14% faster than direct iteration;
• equal-level bulk text writes were at parity; and
• direct single-file JSONL iteration remained about 1.5-1.8x slower than gzip because each line crosses an async-iterator boundary — batch with readlines(hint) when that path is hot.

The concurrency result measures overlapped waiting, not a faster deflate codec, and benchmark ratios vary by hardware, storage, Python version, and data. Large codec calls are offloaded to the default executor so independent tasks can keep making progress. Line splitting, readlines(), and writelines() use bounded batching to reduce aiogzip's own coroutine overhead.

For large UTF-8 JSON Lines files with \n terminators, the measured fast path uses newline="\n" and chunk_size=512 * 1024. Tune memory and throughput for your workload rather than assuming one chunk size fits every application. When CPU-bound per-line processing permits it, repeated await f.readlines(hint) calls can process bounded groups of complete lines with fewer async transitions than async for; the hint is an approximate decoded-character target, not a hard memory limit.

Install the optional zlib-ng engine with:

pip install "aiogzip[fast]"

When installed, zlib-ng is selected automatically for decompression. Its gain depends on the input and access pattern: it helps decompression-heavy bulk reads far more than per-line Python iteration. Compression remains on stdlib zlib so installation alone does not change gzip bytes; pass fast_compress=True per writer to opt in. Set AIOGZIP_ENGINE=stdlib to force stdlib behavior. Inspect the default selections for a diagnostic report:

import aiogzip

print(aiogzip.engine_info())

The engine names are informational, not a stable machine-readable interface. See the performance guide for benchmarks and tuning guidance.

Development and contributing

The 1.x line is the last to support Python 3.8 through 3.10. aiogzip 2.0 requires Python 3.11+. Older interpreters continue to resolve the latest compatible 1.x release from PyPI.

See the contributing guide for setup, tests, linting, typing, documentation, and benchmark workflows.