dispy
dispy is a comprehensive, yet easy to use framework for creating and using compute clusters to execute computations in parallel across multiple processors in a single machine (SMP), among many machines in a cluster, grid or cloud. dispy is well suited for data parallel (SIMD) paradigm where a computation is evaluated with different (large) datasets independently with no communication among computation tasks (except for computation tasks sending intermediate results to the client).
dispy works with Python versions 2.7+ and 3.1+ on Linux, Mac OS X and Windows; it may work on other platforms (e.g., FreeBSD and other BSD variants) too.
Features
dispy is implemented with pycos, an independent framework for asynchronous, concurrent, distributed, network programming with tasks (without threads). pycos uses non-blocking sockets with I/O notification mechanisms epoll, kqueue and poll, and Windows I/O Completion Ports (IOCP) for high performance and scalability, so dispy works efficiently with a single node or large cluster(s) of nodes. pycos itself has support for distributed/parallel computing, including transferring computations, files etc., and message passing (for communicating with client and other computation tasks). While dispy can be used to schedule jobs of a computation to get the results, pycos can be used to create distributed communicating processes, for broad range of use cases.
Computations (Python functions or standalone programs) and their dependencies (files, Python functions, classes, modules) are distributed automatically.
Computation nodes can be anywhere on the network (local or remote). For security, either simple hash based authentication or SSL encryption can be used.
After each execution is finished, the results of execution, output, errors and exception trace are made available for further processing.
Nodes may become available dynamically: dispy will schedule jobs whenever a node is available and computations can use that node.
If callback function is provided, dispy executes that function when a job is finished; this can be used for processing job results as they become available.
Client-side and server-side fault recovery are supported:
If user program (client) terminates unexpectedly (e.g., due to uncaught exception), the nodes continue to execute scheduled jobs. If client-side fault recover option is used when creating a cluster, the results of the scheduled (but unfinished at the time of crash) jobs for that cluster can be retrieved later.
If a computation is marked reentrant when a cluster is created and a node (server) executing jobs for that computation fails, dispy automatically resubmits those jobs to other available nodes.
dispy can be used in a single process to use all the nodes exclusively (with JobCluster - simpler to use) or in multiple processes simultaneously sharing the nodes (with SharedJobCluster and dispyscheduler program).
Cluster can be monitored and managed with web browser.
Dependencies
dispy requires pycos for concurrent, asynchronous network programming with tasks. pycos is automatically installed if dispy is installed with pip. Under Windows efficient polling notifier I/O Completion Ports (IOCP) is supported only if pywin32 is installed; otherwise, inefficient select notifier is used.
Installation
To install dispy, run:
python -m pip install dispy
Release Notes
Short summary of changes for each release can be found at News. Detailed logs / changes are at github commits.
Links
Documentation is at dispy.org.
Release files for dispy 4.15.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| dispy-4.15.2.tar.gz | 427.9 kB | Without provenance | Details |
Release files / dispy-4.15.2.tar.gz
| Download URL | dispy-4.15.2.tar.gz |
|---|---|
| Size | 427.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8e88bf23673a8269ba34887ebcf4a7c8a51016fbd27f898fefef57dd2257258c
|
|
BLAKE2b-256 checksum How to use checksums |
5988b2bd984a81db9ba0d73a47645ded9da8e0bcfddc231644f9017668aeabcc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.0 CPython/3.10.7
|
Security for dispy 4.15.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
