gzip-stream
gzip-stream is a super-tiny library that will help you compress by GZIP on-the-fly.
GZIPCompressedStream class instance acting like an any other stream (in fact, GZIPCompressedStream inherits io.RawIOBase), but wraps another stream and compress it on-the-fly.
from gzip_stream import GZIPCompressedStream
from my_upload_lib import MyUploadClient
upload_client = MyUploadClient()
with open('my_very_big_1tb_file.txt') as file_to_upload:
compressed_stream = GZIPCompressedStream(
file_to_upload,
compression_level=7
)
upload_client.upload_fileobj(compressed_stream)
GZIPCompressedStream does not read entire stream, but instead read it by chunks, until compressed output size will not satisfy read size.
AsyncGZIPDecompressedStream class can async read from another source with zlib and gzip decompression on-the-fly
# aiobotocore example
import aiobotocore
from gzip_stream import AsyncGZIPDecompressedStream
AWS_ACCESS_KEY_ID = "KEY_ID"
AWS_SECRET_ACCESS_KEY = "ACCESS_KEY"
BUCKET = "AWESOME_BUCKET"
upload_client = MyAsyncUploadClient()
session = aiobotocore.get_session()
async with session.create_client(
service_name="s3",
endpoint_url="s3_endpoint",
aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
aws_access_key_id=AWS_ACCESS_KEY_ID,
) as client:
response = await client.get_object(Bucket=BUCKET, Key='my_very_big_1tb_file.txt.gz')
async for decompressed_chunk in GzipAsyncReaderWrapper(response["Body"])):
await upload_client.upload_fileobj(decompressed_chunk)
Module works on Python ~= 3.5.
Installation
pip install gzip-stream
License
Public Domain: CC0 1.0 Universal.
Metadata
Release files for gzip-stream 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| gzip-stream-1.2.0.tar.gz | 6.1 kB | Without provenance | Details for gzip-stream-1.2.0.tar.gz |
Release files / gzip-stream-1.2.0.tar.gz
| Download URL | gzip-stream-1.2.0.tar.gz |
|---|---|
| Size | 6.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4065cd84443de771fe8dfa6463140cbd5d02da83c06381c84a8ac66cfc5f9766
|
|
BLAKE2b-256 checksum How to use checksums |
159f20fb069117a2f1ea514878f51db1da9046109cf76ba38577316bc1964d28
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/59.5.0 requests-toolbelt/0.9.1 tqdm/4.58.0 CPython/3.7.4
|
Security for gzip-stream 1.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.