dask chunked read_text on gzip file
Project description
Dask indexed gzip
An implementation compatible with dask read_text interface, than can chunk a gzipped text file into several partitions, thanks to an index, provided by indexed_gzip
Sample session
>>> import dask_igzip >>> file_path = dask_igzip() >>> dask.igzip.
Dask read_text create a unique partition if you provide it with a gzip file. This is understandable, there is no way to split the gzip file in a predictable yet coherent way. This project provides an implementation where the gzip is indexed, then lines positions are also indexed, so that reading the text can be done by chunk (thus enabling parallelism). On first run, indexes are saved on disk, so that subsequent runs are fast.
Changelog
The format is based on Keep a Changelog and this project tries to adhere to Semantic Versioning.
0.1.0 - 2018-06-19
New
initial release
100% code coverage
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dask-igzip-0.1.0.linux-x86_64.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c45d9941474953cf0d5c001bcf4ac811c07fe4b9ababb514480cf1dd4f9b6dde |
|
MD5 | 726a2500fb5552a72800758439a2f19e |
|
BLAKE2b-256 | b99018b2fa4a02bbdd4f6d33908596132440343606b0e321b51af4dd30aced3a |
Hashes for dask_igzip-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8ba4a036d69c42f9af3a33f719692fef43300cd768b6e7740a1b5e4107822108 |
|
MD5 | 6a65d8614112a14a7adfc1e31b32218c |
|
BLAKE2b-256 | de01f7fd4f15de000738c300a89ecd626ffd3d159768f7fa306d9cb0b992f167 |