About
ParPar (Parallel Parser) is a light tool which makes it easy to distribute a function across a large file.
ParPar is meant to work on serialized data where some values are highly repeated across records for a given field. e.g.
a 1 1
a 1 2
a 2 3
a 2 4
a 3 5
a 3 6
b 1 7
b 1 8
b 2 9
b 2 10
b 3 11
b 3 12
although we have 12 records, for column 1 there are only 2 unique values. Likewise, for column 2 there are only 3 unique values. We could break this file up into smaller files under a directory:
<out-dir>/<col-1-value>/<col-2-value>
or vis versa.
How to use.
- Start by importing the ParPar class:
from parpar import ParPar
- Initialize an instance:
ppf = ParPar()
- Shard a large file into sub-files:
# shard by columns
ppf.shard(
<input-file>, <output-directory>,
<columns>, <delim>, <newline>
)
# shard by lines
ppf.shard_by_lines(
<input-file>, <output-directory>,
<number_of_lines>, <delim>, <newline>
)
- Check to make sure the number of records are the same:
files = ppf.shard_files(<output-directory>)
records = ppf.sharded_records(files)
from parpar.utils import filelines
print(records == filelines(<input-file>))
- Map an arbitrary function across each line of all shared files:
def foo(line, *args, **kwargs):
pass
args = [1, 2, 3]
kwargs = {'a': 'x', 'b': 'y'}
ppf.shard_line_apply(<output-directory>,
foo, args, kwargs,
processes=<number-of-processes>
)
- Map an arbitrary function across all shared files:
def bar(file, *args, **kwargs):
pass
args = [1, 2, 3]
kwargs = {'a': 'x', 'b': 'y'}
ppf.shard_file_apply(<output-directory>,
bar, args, kwargs,
processes=<number-of-processes>
)
1 maintainer
Metadata
1 maintainer
Release files for parpar 0.0.19
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| parpar-0.0.19.tar.gz | 7.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| parpar-0.0.19-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 15.0 kB
Release files / parpar-0.0.19.tar.gz
| Download URL | parpar-0.0.19.tar.gz |
|---|---|
| Size | 7.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d72349456f083121b9995e10bb1f69eb2be2aaab6c1cdab0df5976c8d4c7bc3c
|
|
BLAKE2b-256 checksum How to use checksums |
f359f96aa503c15fd01725779cd369be5d0ba890215d1964e22fcb7fdb353253
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.34.0 CPython/3.6.8
|
Release files / parpar-0.0.19-py3-none-any.whl
| Download URL | parpar-0.0.19-py3-none-any.whl |
|---|---|
| Size | 8.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
cbf554f3ae7775558b00c67062f3d72dc9337c05c97e80ce742da7583d4b4d4a
|
|
BLAKE2b-256 checksum How to use checksums |
b67dbcef68c5809355278d259d5f7a747576dea3ffd10d5a28cea0fb15c34e02
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.34.0 CPython/3.6.8
|
