Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Distribute commands to many nodes

Project description

cooperate is a shell command that execute commands in a cooperative manner, by distributing them to many nodes.

It requires Python >= 3.3 and asyncio.

For example:

cooperate --local -- echo FOO

Will execute the job “echo FOO” locally.

This one:

cooperate --ssh me@my.node -- echo FOO

Is barelly equivalent to:

ssh me@my.node echo FOO

You can declare as many nodes as you want. For example:

cooperate --local --ssh me@my.node --ssh me@my.second.node -- echo FOO

Is equivalent to:

echo FOO
ssh me@my.node echo FOO
ssh me@my.second.node echo FOO

You can also declare many jobs at once. For example:

cooperate --local --command "echo FOO" --command "echo BAR"

Is equivalent to:

echo FOO
echo BAR

Installation

pip install cooperate

Nodes

Commands can be distribued thru these kind of nodes:

  • –local execute locally
  • –ssh execute thru ssh
  • –docker execute in a local docker container
  • –lxc execute in a local lxc container

These options can be repeated as often as needed.

Modes

By default, it spawns every commands to every nodes.

The -m, –mode allow to configure the desired mode.

The all mode executes all commands in all nodes:

cooperate --local --ssh me@my.node --ssh me@my.second.node \
    --command="echo FOO" --command="echo BAR"

Is equivalent to:

echo FOO
echo BAR
ssh me@my.second.node echo FOO
ssh me@my.second.node echo BAR

The distribute mode share the commands out among all the nodes:

cooperate --local --ssh me@my.node --ssh me@my.second.node \
    --command="echo FOO" --command="echo BAR" --mode=distribute

Is equivalent to:

echo FOO
ssh me@my.second.node echo BAR

Concurrency

By default, it executes all jobs simultaneously.

The -b, –batch option allows to execute on only a specify number of jobs at a time. Both percentages and finite numbers are supported:

cooperate --local --concurrence 1 \
    --command="echo FOO" --command="echo BAR" --command="echo BAZ"

cooperate --local --concurrence 33% \
    --command="echo FOO" --command="echo BAR" --command="echo BAZ"

The concurrency system maintains a window of running jobs. When a job returns then it starts a remnant job and so on.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for cooperate, version 0.3.1
Filename, size File type Python version Upload date Hashes
Filename, size cooperate-0.3.1-py3-none-any.whl (9.0 kB) File type Wheel Python version 3.4 Upload date Hashes View hashes
Filename, size cooperate-0.3.1.tar.gz (5.5 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page