Skip to main content

pyumpf

Python Unified Multiprocessing Parallel Functions

This is a small library that aims to bring together the standard library map/reduce functions with both the multiprocessing module in the standard library and the PP module (www.parallelpython.com).

Each has their own niche, and there own advantages and disadvantages. The map/reduce functions in the standard library are good for implementing a functional paradigm, but are limited by the GIL to only run one task at a time. The multiprocessing module is good for single machines with multiple CPUs, but can be awkward to debug and use. The PP module is great for clusters, but is even more awkward to debug and use. Plus all of these have different APIs, and so cannot be drop-in replacements.

pyumpf gets around this by providing a unified interface via umpf.map and umpf.reduce that can easily be expanded to use multiprocessing or PP when needed, but also collapsed to the built-in map/reduce for simpler debugging.

To install: python setup.py install

For an example, see umpf_test.py Note that this example gives PP a poor performance. This is because it is running on a single machine, and this example is too small to make best use of the parallelism. Ideally, tasks should run for several seconds each and have easily pickleable arguments and returned values.

By default, umpf defaults to pythons built-in map/reduce. These are single-threaded.

To use multiprocessing, do the following:

import umpf import multiprocessing as mp umpf.Hub.pool = mp.Pool()

To use Parallel Python, do the following:

import umpy import pp umpy.Hub.pool = pp.Server()

Metadata

Release files for pyumpf 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyumpf 0.1.1
File Size Uploaded Provenance Actions
pyumpf-0.1.1.tar.gz 3.5 kB Without provenance Details for pyumpf-0.1.1.tar.gz

Release files / pyumpf-0.1.1.tar.gz

Download URL pyumpf-0.1.1.tar.gz
Size 3.5 kB
Tags Source
SHA-256 checksum
How to use checksums
2f2e5a5f592ca6adb97c2cbe6e11cd1a8e99d0f9925ac0d065d72fa65bd83e1f
BLAKE2b-256 checksum
How to use checksums
38f2973765bd9b801320c200d90026fc7d25740c327cb310c97b2d645b46da20
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for pyumpf 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page