Massively Parallel operation made easy
Project description
Rationale
You wrote a program a.out with some parameters
You need to explore the space of parameters
Minionize is a solution to spawn a legion of a.out in a massively parallel manner.
How does it work
A classical pattern to do the above is to apply the master/worker pattern where a master give tasks to workers. Workers repeatedly fetch a new task from a queue , run it and report back somewhere its status.
Minionize encapsulates a.out so that it can takes its inputs from a queue.
Currently we support:
execo based queue: the queue is stored in a shared file system in your cluster
Google pub/sub based queue: the queue is hosted by Google.
Some examples
- Simplest use:
with Execo engine:
# generate some input parameters
$) python -c "from execo_engine.sweep import ParamSweeper; ParamSweeper('sweeps', sweeps=range(10), save_sweeps=True)"
# start your minions
$) minionize echo hello
hello 0
hello 1
hello 2
hello 3
hello 4
hello 5
hello 6
hello 7
hello 8
hello 9
- examples/process.py: run your program as a subprocess each time a new parameter comes / shows how to transform the input parameters
use it with Execo engine:
# generate the queue of task python -c "from execo_engine.sweep import ParamSweeper, sweep; ParamSweeper('sweeps', sweeps=sweep({'a': [0, 1], 'b': ['x', 't"]}), save_sweeps=True)" # start your minions MINION_ENGINE=execo python process.py
use it with GooglePubSub engine:
# start your minions MINION_ENGINE=google \ GOOGLE_PROJECT_ID=gleaming-store-288314 \ GOOGLE_TOPIC_ID=TEST \ GOOGLE_SUBSCRIPTION=tada \ GOOGLE_APPLICATION_CREDENTIALS=``/.gcp/gleaming-store-288314-2444b0d20a52.json \ python process.py
Roadmap
Easy integration as docker entrypoint
Support new queues (Redis stream, RabbitMQ, Kakfa …)
Support new abstractions to run container based application (docker, singularity…)
Automatic encapsulation using a .minionize.yml
Keep in touch (matthieu dot simonin at inria dot fr)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for minionize-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 79da85ced281d448dd8df3fc4144824dd361b8bf5521223b755498eb875163eb |
|
MD5 | 22c9856ed8bfce462d61d8b2f8bb7539 |
|
BLAKE2b-256 | 4a875414d6807a003e9816112c000206653a9d94aec8041e486c39e2fbb6df7b |