Skip to main content

A utility to transport files and execute at remote in prarallel

Project description

Description:
This is a framework to execute programs in multiple nodes concurrently.
It consists of two modules: controller and executor. The controller controls the whole process, it first dispatches the executor
and user-defined program files to all the nodes specified by the user. Then it starts the executor remotely and waits for all the executors
to end. The executor run the program specified by the user and reports to the controller peroidly. When it finishes, it notifys the
controller and then exits. After all the executors have reported their status or a speicified timeout is reached, the controller exits.
The user can then send another request to pull the results.
To monitor every executor's status, the executor reports to the controller peroidly. When one executor fails to report in a
predefined timeout, the controller will mark it as dead. And if the controller receives the report later, it changes the status to be alive
again. The controller never waits for a dead node.

Features:
Heart beat monitor
Complete parallel execution
Easy tool for cloning files
Agile and simple:
unlike program in other languages, this is simple and the executor is only one single file.
Minimal resource exhaustion:
the execution is distributed in all nodes, and the controller exausted little resource(mainly for monitor heartbeat)

Environment:
This is just tested on python2.7 and python2.6 in linux environment.
And it based on ssh and scp for net communication, before your exection, you should authenticated your controller with all executors.

Commands:
corunner-run: to run program in multiple nodes concurrently.
corunner-cp: to dispatch files from or to multiple nodes concurrently.

Example:
1. Execute the myscript in all machines from 192.168.101 to 192.168.200 conrrently with heartbeat switched off:
Run in controller: corunner-run -n 192.168.100.101..200 -f myscript -r "/tmp/corunner" -i 1 python /tmp/corunner/myscript
2. Collect all output files from above to the controller and put in seperated directory:
Run in controller: corunner-cp -n 192.168.100.101..200 -i -s /tmp/corunner/ouput -d /temp/corunner/all --divide

Other:
If you have any problem or suggestion, welcome to contact me, my email is zwsun<sun33170161@gmail.com>. I'm pleasant if it can help
you and improve your efficiency when working with many machines.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

corunner-1.0.3.beta.tar.gz (23.0 kB view details)

Uploaded Source

File details

Details for the file corunner-1.0.3.beta.tar.gz.

File metadata

  • Download URL: corunner-1.0.3.beta.tar.gz
  • Upload date:
  • Size: 23.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for corunner-1.0.3.beta.tar.gz
Algorithm Hash digest
SHA256 9603e09b02ec74a6c5a7f5f9c014eb778dcd29409d5be0a07f3ebdfcec3a86cf
MD5 b0329e3b93fbf8f00d3bdf7820bfb939
BLAKE2b-256 804f6d61da4df108e2b559230acc5102998d4cc9d61762e101586e5adc4dd6cc

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page