pckit 0.5.3

A simple package for parallel computing with Python

pckit

This is a simple package for parallel computing with Python.

Usage

Multiprocessing

If you want to use any solver from the package you have to wrap your functions into a model. Here the example with square of 2 and 3 are evaluated by 2 workers. MyModel is a subclass of the package Model. The method results is required.

import pckit


class MyModel(pckit.Model):
    def results(self, x: int) -> int:
        # Solve here problem f(x) = x^2
        return x ** 2


if __name__ == '__main__':
    model = MyModel()
    worker = pckit.MultiprocessingWorker(model)
    with pckit.get_solver(worker, workers_num=2) as solver:
        # Create tasks to solve
        tasks = [2, 3]
        results = solver.solve(tasks)
        print(results)
        # >>> [4, 9]

Workers number can be controlled by workers_num argument.

MPI

You can easily run scripts on the cluster with mpi4py implementation on MPI (See mpi4py installation docs). Simply change MultiprocessingWorker to MPIWorker in the previous example and start the script with MPI mpiexec -np <n> python -m mpi4py your_script.py, where <n> is a number of workers.

worker = pckit.MPIWorker(model)

By default, zero rank process is also used as a worker. It can be controlled by zero_rank_usage argument of get_solver function.

Single thread

Single threaded execution is also available with Worker

worker = pckit.Worker(model)

Examples

More examples

Features

Cache

Dict based cache is available by caching argument in get_solver(). Tasks are required to be hashable.

with pckit.get_solver(worker, caching=True) as solver:
    tasks = [2, 2]

The second task's solution will be reused from the cache.

You can create your own cache by implementing __contains__, __getitem__, __setitem__ of the BaseCache class. See example for more details.

Custom iterators

You can send emails or print anything during evaluation with custom iterator. tqdm is also supported.

import tqdm

results = solver.solve(tasks, iterator=tqdm.tqdm)

See example to create your own iterator.

Logging with MPI

See example

Comsol Models, Solvers, Workers

Based on MPh package.

TBDocumented