Skip to main content

chunksv

Python wrapper for csv.reader that can process files in predefined chunks.

purpose

This library allows a user to partition a filestream into partitions of a predefined size. It was initially motivated by the need to process large CSV files from AWS S3 while keeping application code clean.

package installation and usage

The package is available on PyPI:

python -m pip install chunksv

The library can be imported and used as follows:

import chunksv

with open("file.csv", "r") as f:
    rows = chunksv.reader(
        f, 
        max_bytes=<size of each partition>, 
        header=[<optional columns list>]
    )

When the reader object has consumed enough rows to reach the max_bytes limit, it will raise StopIteration. To consume more rows from the input stream, call reader.resume():

while not rows.empty:
    current_partition = [r for r in rows]
    < process partition here >
    rows.resume()

Metadata

Release files for chunksv 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for chunksv 0.1.0
File Size Uploaded Provenance Actions
chunksv-0.1.0.tar.gz 3.6 kB Without provenance Details for chunksv-0.1.0.tar.gz

Release files / chunksv-0.1.0.tar.gz

Download URL chunksv-0.1.0.tar.gz
Size 3.6 kB
Tags Source
SHA-256 checksum
How to use checksums
2f5bb735948cd70d772f87fec8d7497e9af3668cea4be0e9ed5af14aeba99aea
BLAKE2b-256 checksum
How to use checksums
a73ce3b27b119680079d44116c479ccdbb5f48b4fe296e37c2fa29f5c080ed08
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.9.0

Security for chunksv 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page