Skip to main content

A Task Based Parallelization Framework

Project description

Jug allows you to write code that is broken up into tasks and run different tasks on different processors.

It uses the filesystem to communicate between processes and works correctly over NFS, so you can coordinate processes on different machines.

Jug is a pure Python implementation and should work on any platform.

Website: http://luispedro.org/software/jug

Documentation: http://packages.python.org/Jug

Video: On vimeo or showmedo

Mailing List: http://groups.google.com/group/jug-users

Short Example

Here is a one minute example. Save the following to a file called primes.py:

from jug import TaskGenerator
from time import sleep

@TaskGenerator
def is_prime(n):
    sleep(1.)
    for j in xrange(2,n-1):
        if (n % j) == 0:
            return False
    return True

primes100 = map(is_prime, xrange(2,101))

Of course, this is only for didactical purposes, normally you would use a better method. Similarly, the sleep function is so that it does not run too fast.

Now type jug status primes.py to get:

Task name                  Waiting       Ready    Finished     Running
----------------------------------------------------------------------
primes.is_prime                  0          99           0           0
......................................................................
Total:                           0          99           0           0

This tells you that you have 99 tasks called primes.is_prime ready to run. So run jug execute primes.py &. You can even run multiple instances in the background (if you have multiple cores, for example). After starting 4 instances and waiting a few seconds, you can check the status again (with jug status primes.py):

Task name                  Waiting       Ready    Finished     Running
----------------------------------------------------------------------
primes.is_prime                  0          63          32           4
......................................................................
Total:                           0          63          32           4

Now you have 32 tasks finished, 4 running, and 63 still ready. Eventually, they will all finish and you can inspect the results with jug shell primes.py. This will give you an ipython shell. The primes100 variable is available, but it is an ugly list of jug.Task objects. To get the actual value, you call the value function:

In [1]: primes100 = value(primes100)

In [2]: primes100[:10]
Out[2]: [True, True, False, True, False, True, False, False, False, True]

What’s New

version 0.8: - Tasklets - Fix bugs in sleep-until and cleanup

version 0.7.4: - Fix case where ~/.jug/configrc does not exist - Print host name to lock file on file_store - Refactored implementation of options - Fix unloading tasks that have not run - Fix mapreduce for empty input

Version 0.7.3: - Parse ~/.jug/configrc - Fix bug with waiting times - Special case saving of numpy arrays - Add more expressive jugdir syntax - Save dict_store backend to disk

Version 0.7.2: - included missing files in the distribution

Version 0.7.1: - sleep-until subcommand - bugfixes

Version 0.7 (starting with 0.6.9 in testing): - barrier() - better shell command

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Jug-0.8-b0.tar.gz (33.8 kB view details)

Uploaded Source

File details

Details for the file Jug-0.8-b0.tar.gz.

File metadata

  • Download URL: Jug-0.8-b0.tar.gz
  • Upload date:
  • Size: 33.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for Jug-0.8-b0.tar.gz
Algorithm Hash digest
SHA256 513a2ea0096e77305f187e808e3f5fca63cdf1a3bd040e30429874be4dc57ba9
MD5 0a57bb7e5043f78820b7a57b11f44d2f
BLAKE2b-256 39496b47d4af4e4149bd709531c0029c91466239787f25fcec7f9964299c5c70

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page