A simple package for parallel computing with Python
Project description
pckit
This is a simple package for parallel computing with Python.
Usage
Multiprocessing
If you want to use any solver from the package you have to wrap your functions into a model.
Here the example with square of 2 and 3 are evaluated by 2 workers.
MyModel
is a subclass of the package Model
. The method results
is required.
import pckit
class MyModel(pckit.Model):
def results(self, x: int) -> int:
# Solve here problem f(x) = x^2
return x ** 2
if __name__ == '__main__':
model = MyModel()
worker = pckit.MultiprocessingWorker(model)
with pckit.get_solver(worker, workers_num=2) as solver:
# Create tasks to solve
tasks = [2, 3]
results = solver.solve(tasks)
print(results)
# >>> [4, 9]
MPI
You can easily run scripts on the cluster with mpi4py implementation on MPI (See mpi4py installation docs).
Simply change MultiprocessingWorker
to MPIWorker
in the previous example and start the script with MPI mpiexec -np 3 python -m mpi4py your_script.py
worker = pckit.MPIWorker(model)
Moreover, a multiprocessing solver can be started inside an MPI solver.
Single thread
Single threaded execution is also available with Worker
worker = pckit.Worker(model)
Examples
Features
Cache
Dict based cache is available by caching
argument in get_solver()
.
Tasks are required to be hashable.
with pckit.get_solver(worker, caching=True) as solver:
tasks = [2, 2]
The second task's solution will be reused from the cache.
Custom iterators
You can send emails or print anything during evaluation with custom iterator. tqdm is also supported.
import tqdm
results = solver.solve(tasks, iterator=tqdm.tqdm)
See example to create your own iterator.
Comsol Models, Solvers, Workers
Based on MPh package.
TBDocumented
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.