a set utilities that ake advantage of python's 'multiprocessing' module to distribute CPU-intensive tasks
# Para(llel) processing utilities This library implements a simple set of parallel processing utilities that take advantage of python’s multiprocessing module to distribute processing over multiple CPUs on a single machine. The most salient feature of this library is the map() function that can be used to distribute CPU-intensive processing of a collection of items over multiple cores.
## Basic usage
>>> import para >>> import gzip >>> >>> items = ["examples/big-file1.gz", "examples/big-file2.gz", ... "examples/big-file3.gz"] >>> def log_lines(path): ... with gzip.open(path, 'rt') as f: ... for lineno, line in enumerate(f): ... if len(line) > 50: ... yield path, lineno, line ... >>> for path, lineno, line in para.map(log_lines, items): ... print(path, lineno, repr(line)) ... examples/big-file1.gz 2 'this line is going to be much longer than 80 chars -- at least I hope it will\n' examples/big-file3.gz 0 'again with the long lines -- this is going to show up in the output, I hope\n'
## Authors * Aaron Halfaker – https://github.com/halfak
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|File Name & Checksum SHA256 Checksum Help||Version||File Type||Upload Date|
|para-0.0.5.tar.gz (5.0 kB) Copy SHA256 Checksum SHA256||–||Source||Sep 18, 2015|
|para-0.0.5.zip (9.7 kB) Copy SHA256 Checksum SHA256||–||Source||Sep 18, 2015|