Simpler multiprocessing framework
Today’s computers are almost always multicore machines. However, due to CPython’s GIL, exploiting these multiple cores are not simple. The multithreading module, for instance, are practically rendered impotent as multithreading won’t happen unless a thread is being tied up in an I/O operation.
However, Python provides a way to actually harness multicore parallelism: The multiprocessing module.
The problem with multiprocessing is that you are given a very ‘low-level’ plumbing mechanism. It provides you with tools to create multiple (sub-)processes, tools to communicate between the (sub-)processes, tools to synchronize between the (sub-)processes… but that’s about it. When you start making programs to harness multiprocessing’s powers, you’ll quickly find out that you have to keep building the exact same code structure. And along the way, you keep ‘re-creating the wheel’, so to speak.
This package aims to provide Python programmers with a ‘slightly higher level’ view of multiprocessing, by removing the necessity of keep having to create the ‘harness’ code.
In a nutshell, this code aims to be to multiprocessing what requests did for urllib: Abstracting all the plumbing and providing a more intuitive toolset to implement multicore-aware programs.