Parallel Python

Parallel Python is a Python module which provides a mechanism for parallel execution of Python code on SMP (systems with multiple processors or cores) and clusters (computers connected via network).

It is light, easy to install and integrate with other Python software. Parallel Python is an open source and cross-platform module written in pure Python — no third-party runtime dependencies.

Website: www.parallelpython.com

Why processes instead of threads?

The most simple and common way to write parallel applications for SMP computers is to use threads. However, if the application is computation-bound, the threading module will not allow Python byte-code to run in parallel: the interpreter uses the GIL (Global Interpreter Lock), so only one byte-code instruction executes at a time even on an SMP machine.

Parallel Python overcomes this limitation with worker processes and IPC. All the complexity of processes, pipes, sockets and scheduling is handled for you — your application just submits jobs and retrieves their results. The same jobs can run on local worker processes or on remote ppserver nodes, with dynamic load-balancing across both.

Quick start, SMP

import pp

def square(x):
    return x * x

# 1) Start the pp execution server with one worker per processor
job_server = pp.Server()

# 2) Submit all the tasks for parallel execution
f1 = job_server.submit(square, (1,))
f2 = job_server.submit(square, (2,))

# 3) Retrieve the results as needed
r1 = f1()
r2 = f2()

job_server.destroy()

See examples/ for complete programs: summing primes, reversing MD5 hashes, dynamic worker counts, callbacks, auto-differentiation and a benchmark.

Quick start, clusters

On the nodes, start a server on each remote computational node:

node-1> ppserver
node-2> ppserver
node-3> ppserver

On the client:

import pp

ppservers = ("node-1", "node-2", "node-3")
job_server = pp.Server(ppservers=ppservers)

f1 = job_server.submit(func1, args1, depfuncs1, modules1)
f2 = job_server.submit(func2, args2, depfuncs2, modules2)

r1 = f1()
r2 = f2()

Auto-discovery

Instead of listing nodes explicitly, run servers with auto-discovery enabled and let the client find them over UDP broadcast:

node-1> ppserver -a
node-2> ppserver -a

import pp

job_server = pp.Server(ppservers=("*",))

By default the discovery destination is derived from the wildcard pattern ("*" uses 255.255.255.255). On networks without broadcast delivery you can point discovery at a specific address instead — on the client with the broadcast argument, on ppserver with -b BROADCAST:

job_server = pp.Server(ppservers=("*",), broadcast="192.168.1.255")

Features

• Parallel execution of Python code on SMP machines and clusters
• Job-based parallelization that is easy to understand and convert from serial code
• Automatic detection of the optimal configuration (the number of worker processes defaults to the number of effective processors)
• Dynamic processor allocation — the worker count can be changed at runtime with set_ncpus()
• Low overhead for repeated jobs: identical function packages are shipped once and then referenced by content hash
• Dynamic load balancing — jobs are distributed between workers at runtime
• Fault tolerance — if a worker or node fails, tasks are rescheduled on others
• Auto-discovery of computational resources over UDP broadcast
• Dynamic allocation of computational resources (a consequence of auto-discovery and fault tolerance)
• SHA-based authentication for network connections
• Cross-platform portability (Windows, Linux, Unix, Mac OS X)
• Standard library only — no third-party runtime dependencies

Requirements

• Python 3.10 or newer
• No third-party runtime dependencies (standard library only)

Installation

pip install pp

From a source checkout:

pip install -e .

The install provides the pp module and the ppserver command-line tool.

Running a network server

ppserver -i 0.0.0.0 -p 60001 -s mysecret -w 4

Then from any client on the network:

job_server = pp.Server(ppservers=("192.168.1.10:60001",), secret="mysecret")

ppserver options:

Security note: always use a non-trivial secret key. A default secret is used when none is configured, which is not suitable for untrusted networks.

API overview

job_server = pp.Server(ncpus="autodetect", ppservers=(), secret=None,
                       restart=False, proto=4, socket_timeout=3600,
                       loglevel=None)
job = job_server.submit(func, args=(), depfuncs=(), modules=(),
                        callback=None, callbackargs=(), group="default",
                        globals=None)
result = job()                  # blocks until the job finishes
job.wait()                      # block until the job finishes
job_server.wait("group")        # wait for a group of jobs
job_server.set_ncpus(4)         # resize the local pool at runtime
job_server.get_active_nodes()   # {node: ncpus}
job_server.get_stats()          # job execution statistics
job_server.print_stats()        # print the statistics
job_server.destroy()            # kill workers and close files

template = pp.Template(job_server, func, depfuncs=(), modules=(),
                       callback=None, callbackargs=(), group="default")
job = template.submit(1, 2, 3)  # reuse the same job with new args

Notes:

• submit serializes the function by its source, so functions and classes defined in your script work out of the box. Functions that live in importable modules can instead be shipped by name via the modules= argument. Built-in callables (math.sqrt, str.upper, ...) are shipped as an import-and-bind.
• Lambdas must be assigned to a variable before submitting (f = lambda x: x * x), since workers resolve functions by name.
• A job that raises an exception returns None and prints the traceback; the worker keeps serving subsequent jobs.

Examples

examples/sum_primes.py        # sum of primes across workers
examples/reverse_md5.py       # brute-force hash search
examples/dynamic_ncpus.py     # change the worker count at runtime
examples/callback.py          # result callbacks
examples/auto_diff.py         # automatic differentiation
examples/benchmark.py         # serial vs. parallel throughput

Testing

pip install pytest pytest-timeout
pytest

All tests are self-contained (local subprocesses and loopback sockets only) and each is held to a 60-second timeout by pytest-timeout.

Project layout

src/pp/
    __init__.py    # public API
    _version.py    # version metadata
    _common.py     # shared utilities
    _transport.py  # length-framed pipe/socket transports + caching
    _server.py     # Server, Template, DestroyedServerError, scheduler
    _worker.py     # worker subprocess (python -m pp._worker)
    _auto.py       # UDP auto-discovery
    cli.py         # ppserver entry point (NetworkServer + CLI)
examples/          # runnable examples
tests/             # pytest test suite

License

Apache-2.0 (see LICENSE and NOTICE). Questions and support: support@parallelpython.com.