Skip to main content

a set utilities that ake advantage of python's 'multiprocessing' module to distribute CPU-intensive tasks

Project description

# Para(llel) processing utilities This library implements a simple set of parallel processing utilities that take advantage of python’s multiprocessing module to distribute processing over multiple CPUs on a single machine. The most salient feature of this library is the map() function that can be used to distribute CPU-intensive processing of a collection of items over multiple cores.

  • Installation pip install para

## Basic usage

>>> import para
>>> import gzip
>>>
>>> items = ["examples/big-file1.gz", "examples/big-file2.gz",
...          "examples/big-file3.gz"]
>>> def log_lines(path):
...     with gzip.open(path, 'rt') as f:
...         for lineno, line in enumerate(f):
...             if len(line) > 50:
...                 yield path, lineno, line
...
>>> for path, lineno, line in para.map(log_lines, items):
...     print(path, lineno, repr(line))
...
examples/big-file1.gz 2 'this line is going to be much longer than 80 chars -- at least I hope it will\n'
examples/big-file3.gz 0 'again with the long lines -- this is going to show up in the output, I hope\n'

## Authors * Aaron Halfaker – https://github.com/halfak

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

para-0.0.8.tar.gz (5.4 kB view details)

Uploaded Source

Built Distribution

para-0.0.8-py3-none-any.whl (6.5 kB view details)

Uploaded Python 3

File details

Details for the file para-0.0.8.tar.gz.

File metadata

  • Download URL: para-0.0.8.tar.gz
  • Upload date:
  • Size: 5.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.6.3 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.5.6

File hashes

Hashes for para-0.0.8.tar.gz
Algorithm Hash digest
SHA256 46c3232ae9d8ea9d886cfd08cdd112892202bed8645f40b6255597ba4cfef217
MD5 4bda06e38cc2d7d865b2fb38e632605d
BLAKE2b-256 a6ac192f715b8260eb05d7dfd822e9626583e2aa658f33afd407e2febbb37373

See more details on using hashes here.

File details

Details for the file para-0.0.8-py3-none-any.whl.

File metadata

  • Download URL: para-0.0.8-py3-none-any.whl
  • Upload date:
  • Size: 6.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.6.3 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.5.6

File hashes

Hashes for para-0.0.8-py3-none-any.whl
Algorithm Hash digest
SHA256 c63b030658cafd84f8fabfc000142324d51c7440e50ef5012fd1a54972ca25f4
MD5 c84b59cc1c939029d9cb8ca4fc5cff30
BLAKE2b-256 716255f612856021efa8fe7425289914f3641cca7eb9bc9374bf888597f37ce7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page