Massively Parallel operation made easy
Project description
Rationale
You wrote a program a.out with some parameters
You need to explore the space of parameters
Minionize is a solution to spawn a legion of a.out in a massively parallel manner.
How does it work
A classical pattern to do the above is to apply the master/worker pattern where a master give tasks to workers. Workers repeatedly fetch a new task from a queue , run it and report back somewhere its status.
Minionize encapsulates a.out so that it can takes its inputs from a queue.
Currently we support:
execo based queue: the queue is stored in a shared file system in your cluster
Google pub/sub based queue: the queue is hosted by Google.
Some examples
- Simplest use:
with Execo engine:
# Create the queue of params # You'll have to run this prior to launching your minions (adapt to # your need / make a regular script) $) python -c "from execo_engine.sweep import ParamSweeper; ParamSweeper('sweeps', sweeps=range(10), save_sweeps=True)" # start your minions $) MINION_ENGINE=execo minionize echo hello hello 0 hello 1 hello 2 hello 3 hello 4 hello 5 hello 6 hello 7 hello 8 hello 9
- examples/process.py: run your program as a subprocess each time a new parameter comes / shows how to transform the input parameters
use it with Execo engine:
# generate the queue of task python -c "from execo_engine.sweep import ParamSweeper, sweep; ParamSweeper('sweeps', sweeps=sweep({'a': [0, 1], 'b': ['x', 't"]}), save_sweeps=True)" # start your minions MINION_ENGINE=execo python process.py
use it with GooglePubSub engine:
# start your minions MINION_ENGINE=google \ GOOGLE_PROJECT_ID=gleaming-store-288314 \ GOOGLE_TOPIC_ID=TEST \ GOOGLE_SUBSCRIPTION=tada \ GOOGLE_APPLICATION_CREDENTIALS=``/.gcp/gleaming-store-288314-2444b0d20a52.json \ python process.py
Roadmap
Easy integration as docker entrypoint
Support new queues (Redis stream, RabbitMQ, Kakfa …)
Support new abstractions to run container based application (docker, singularity…)
Automatic encapsulation using a .minionize.yml
Keep in touch (matthieu dot simonin at inria dot fr)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for minionize-0.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afe282ef5bd50a42bb7b2899620b20715818c955a54b918e5f7de1efafcba63e |
|
MD5 | b0eac79361016b1b11022cd7797a4cc8 |
|
BLAKE2b-256 | fdffd50dfd6fceafec65769e509e9d4e5c3665b86ab00bdfac286227a8c7bd97 |