Skip to main content

Lightweight pipelining: using Python functions as pipeline jobs.

Project description

Joblib is a set of tools to provide lightweight pipelining in Python. In particular, joblib offers:

  1. transparent disk-caching of the output values and lazy re-evaluation (memoize pattern)

  2. easy simple parallel computing

  3. logging and tracing of the execution

Joblib is optimized to be fast and robust in particular on large data and has specific optimizations for numpy arrays. It is BSD-licensed.

User documentation:

http://pythonhosted.org/joblib

Download packages:

http://pypi.python.org/pypi/joblib#downloads

Source code:

http://github.com/joblib/joblib

Report issues:

http://github.com/joblib/joblib/issues

Vision

The vision is to provide tools to easily achieve better performance and reproducibility when working with long running jobs.

  • Avoid computing twice the same thing: code is rerun over an over, for instance when prototyping computational-heavy jobs (as in scientific development), but hand-crafted solution to alleviate this issue is error-prone and often leads to unreproducible results

  • Persist to disk transparently: persisting in an efficient way arbitrary objects containing large data is hard. Using joblib’s caching mechanism avoids hand-written persistence and implicitly links the file on disk to the execution context of the original Python object. As a result, joblib’s persistence is good for resuming an application status or computational job, eg after a crash.

Joblib strives to address these problems while leaving your code and your flow control as unmodified as possible (no framework, no new paradigms).

Main features

  1. Transparent and fast disk-caching of output value: a memoize or make-like functionality for Python functions that works well for arbitrary Python objects, including very large numpy arrays. Separate persistence and flow-execution logic from domain logic or algorithmic code by writing the operations as a set of steps with well-defined inputs and outputs: Python functions. Joblib can save their computation to disk and rerun it only if necessary:

    >>> from joblib import Memory
    >>> mem = Memory(cachedir='/tmp/joblib')
    >>> import numpy as np
    >>> a = np.vander(np.arange(3)).astype(np.float)
    >>> square = mem.cache(np.square)
    >>> b = square(a)                                   # doctest: +ELLIPSIS
    ________________________________________________________________________________
    [Memory] Calling square...
    square(array([[ 0.,  0.,  1.],
           [ 1.,  1.,  1.],
           [ 4.,  2.,  1.]]))
    ___________________________________________________________square - 0...s, 0.0min
    
    >>> c = square(a)
    >>> # The above call did not trigger an evaluation
  2. Embarrassingly parallel helper: to make it easy to write readable parallel code and debug it quickly:

    >>> from joblib import Parallel, delayed
    >>> from math import sqrt
    >>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
    [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
  3. Logging/tracing: The different functionalities will progressively acquire better logging mechanism to help track what has been ran, and capture I/O easily. In addition, Joblib will provide a few I/O primitives, to easily define logging and display streams, and provide a way of compiling a report. We want to be able to quickly inspect what has been run.

  4. Fast compressed Persistence: a replacement for pickle to work efficiently on Python objects containing large data ( joblib.dump & joblib.load ).

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

joblib-0.10.3.tar.gz (497.4 kB view details)

Uploaded Source

Built Distributions

joblib-0.10.3-py3.5.egg (299.2 kB view details)

Uploaded Egg

joblib-0.10.3-py2.py3-none-any.whl (166.5 kB view details)

Uploaded Python 2Python 3

File details

Details for the file joblib-0.10.3.tar.gz.

File metadata

  • Download URL: joblib-0.10.3.tar.gz
  • Upload date:
  • Size: 497.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for joblib-0.10.3.tar.gz
Algorithm Hash digest
SHA256 29b2965a9efbc90a5fe66a389ae35ac5b5b0c1feabfc7cab7fd5d19f429a071d
MD5 455401ccfaf399538d8e5333086df2d3
BLAKE2b-256 5a429adc0c9addcdbf1877c715cc3cf10ddf885ca987602c709a62218f1c674c

See more details on using hashes here.

File details

Details for the file joblib-0.10.3-py3.5.egg.

File metadata

  • Download URL: joblib-0.10.3-py3.5.egg
  • Upload date:
  • Size: 299.2 kB
  • Tags: Egg
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for joblib-0.10.3-py3.5.egg
Algorithm Hash digest
SHA256 81dbf3d75b8309a7a2bad8f28eb26e95e804afc8e902bdfa3ac4af328c57c815
MD5 3e99e803abba3c76eb54788577e61636
BLAKE2b-256 d7d4071c9aed52426dc9f032bc12db16d5bccac602b3cc674015f9255bdd23af

See more details on using hashes here.

File details

Details for the file joblib-0.10.3-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for joblib-0.10.3-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 90cda39dcef85cb6f64bff51cb812ce6eed9df7217d8e78b8c693ab1c370653e
MD5 2afdf96eb324f465be1c3638f9062279
BLAKE2b-256 23ad17c2e3f1e1f58f1a3efb5d6eb4478ea61b5a782c0361bb4e0e65b93cf013

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page