Skip to main content

Lightweight pipelining: using Python functions as pipeline jobs.

Project description

Joblib is a set of tools to provide lightweight pipelining in Python. In particular, joblib offers:

  1. transparent disk-caching of the output values and lazy re-evaluation (memoize pattern)

  2. easy simple parallel computing

  3. logging and tracing of the execution

Joblib is optimized to be fast and robust in particular on large, long-running functions and has specific optimizations for numpy arrays.


joblib is BSD-licensed.

Vision

Joblib came out of long-running data-analysis Python scripts. The long term vision is to provide tools for scientists to achieve better reproducibility when running jobs, without changing the way numerical code looks like. However, Joblib can also be used to provide a light-weight make replacement.

The main problems identified are:

  1. Lazy evaluation: People need to rerun over and over the same script as it is tuned, but end up commenting out steps, or uncommenting steps, as they are needed, as they take long to run.

  2. Persistence: It is difficult to persist in an efficient way arbitrary objects containing large numpy arrays. In addition, hand-written persistence to disk does not link easily the file on disk to the corresponding Python object it was persists from in the script. This leads to people not a having a hard time resuming the job, eg after a crash and persistence getting in the way of work.

The approach taken by Joblib to address these problems is not to build a heavy framework and coerce user into using it (e.g. with an explicit pipeline). It strives to leave your code and your flow control as unmodified as possible.

Current features

  1. Transparent and fast disk-caching of output value: a make-like functionality for Python functions that works well with large numpy arrays. The goal is to separate operations in a set of steps with well-defined inputs and outputs, that are saved and reran only if necessary, by using standard Python functions:

    >>> from joblib import Memory
    >>> mem = Memory(cachedir='/tmp/joblib')
    >>> import numpy as np
    >>> a = np.vander(np.arange(3))
    >>> square = mem.cache(np.square)
    >>> b = square(a)
    ________________________________________________________________________________
    [Memory] Calling square...
    square(array([[0, 0, 1],
           [1, 1, 1],
           [4, 2, 1]]))
    ___________________________________________________________square - 0.0s, 0.0min
    
    >>> c = square(a)
    >>> # The above call did not trigger an evaluation
  2. Embarrassingly parallel helper: to make is easy to write readable parallel code and debug it quickly:

    >>> from joblib import Parallel, delayed
    >>> from math import sqrt
    >>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
    [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
    
  3. Logging/tracing: The different functionalities will progressively acquire better logging mechanism to help track what has been ran, and capture I/O easily. In addition, Joblib will provide a few I/O primitives, to easily define define logging and display streams, and provide a way of compiling a report. We want to be able to quickly inspect what has been run.

Contributing

The code is hosted on github. It is easy to clone the project and experiment with making your own modifications. If you need extra features, don’t hesitate to contribute them.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

joblib-0.4.1.dev.tar.gz (142.1 kB view details)

Uploaded Source

Built Distribution

joblib-0.4.1.dev-py2.6.egg (76.5 kB view details)

Uploaded Egg

File details

Details for the file joblib-0.4.1.dev.tar.gz.

File metadata

  • Download URL: joblib-0.4.1.dev.tar.gz
  • Upload date:
  • Size: 142.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for joblib-0.4.1.dev.tar.gz
Algorithm Hash digest
SHA256 64d089d85a7ac35e3b3e38c5e3ef09324a1b8e5f5000300187afb992b3bce82a
MD5 7cfc8f6b663b5a20f12d4f9e1c0afc6c
BLAKE2b-256 697c48aa180f02746fdad3594491127e49d8fcd70dc757d8b6380a652c232901

See more details on using hashes here.

File details

Details for the file joblib-0.4.1.dev-py2.6.egg.

File metadata

File hashes

Hashes for joblib-0.4.1.dev-py2.6.egg
Algorithm Hash digest
SHA256 a0ba3f8a971ff52603775332904c68bd67d13b8dd89cdcf1755b21551032be8a
MD5 12ec5e137051965b7f4732b3e903181c
BLAKE2b-256 8eb636fbb33faaff4480909b8b15ae86b5b76750fc0227e48dd162dc16d01179

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page