Skip to main content

Multiprocessing from the command-line

Project description

Description
===========

Multiprocessing from the command-line.

Multiproc allows one to use a pool of workers with bash. It provides a
reliable way to launch a lot of concurrent processes while keeping the
control of the number of these processes.

This program uses python3.

Documentation
=============

Usage: multiproc [-h] [-0] [-v] [-p N] [-1] FORMAT

Arguments:
FORMAT The command to be run in each process.
"%s" input string
"%n" number of the process
"%%" litteral "%"

Options:
-h, --help Print this help and exit.
-v, --verbose Set verbose output
-0 Use NUL as delimiter
This doesn't cover potential command output.
-p, --process N Number of processes to be used.
Default is the number of CPU.
-1, --uniq Equivalent to -p1

Example: download multiple files where urls.txt contains the urls list

$ cat urls.txt | multiproc -p 10 "wget %s -O file.%n"

Example
=======

Download multiple files where urls.txt contains the urls list with 10 processes:

$ cat urls.txt | multiproc -p 10 "wget %s -O file.%n"

Work with files

$ find . -iname "*.c" -print0 | multiproc -v -0 "gcc %n"


Install
=======

The easiest is

pip install multiproc

or, in this directory

python3 setup install


Dependencies
============

docopt https://github.com/docopt/docopt or "pip install docopt"

License
=======

This program is under the GPLv3 License.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

Contact
=======

Main developper: Cédric Picard
Email: cedric.picard@efrei.net
Twitter: @Cym13
GPG: 383A 76B9 D68D 2BD6 9D2B 4716 E3B9 F4FE 5CED 42CB

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

multiproc-1.0.1.tar.gz (3.3 kB view details)

Uploaded Source

File details

Details for the file multiproc-1.0.1.tar.gz.

File metadata

  • Download URL: multiproc-1.0.1.tar.gz
  • Upload date:
  • Size: 3.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for multiproc-1.0.1.tar.gz
Algorithm Hash digest
SHA256 6c8a33ca3076e5154efd543d5423d0578bfdf0a7349bf0d736e901b0f4768d97
MD5 d1bf3b0b95677fd0ab2c4bd8d19ce8fa
BLAKE2b-256 cb95c623ff6042f5b5eb195dcfd35a3a2325f712a8379921918162075630e924

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page