Provenance and caching library for functions
provenance is a Python library for function-level provenance. By decorating functions you are able to cache the results, i.e. artifacts, to memory, disk, or S3. The artifact stores can be layered, e.g. you can write to a local disk store and and a global team S3 one. Every artifact also keeps metadata associated with it on how it was created (i.e. the function and arguments used) so you can always answer the question “Where did this come from?”. The most featureful metadata store is backed by Postgres and is recommended for any serious use. The library is general purpose but was built for machine learning pipelines and plays nicely with numpy and other pydata libraries. You can basically think of this as joblib’s memoize but on steroids.
The library was extracted from a production system and has been used help collaboration on a number of other projects (e.g. sharing models and features easily over s3). The API should be pretty stable but as of now the only documentation are the tests. We will be adding proper documentation and logging soon.
>>> import provenance as p
conda install -c conda-forge provenance or pip install provenance
Assuming you have conda installed, the following commands can be used to create a development environment.
Initial environment creation
conda env create source activate provenance-dev pip install -r requirements.txt pip install -r test_requirements.txt
Reactivating the environment after it has been created
source activate provenance-dev
Release history Release notifications
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size provenance-0.9.4.1-py3-none-any.whl (36.2 kB)||File type Wheel||Python version py3||Upload date||Hashes View hashes|
|Filename, size provenance-0.9.4.1.tar.gz (46.8 kB)||File type Source||Python version None||Upload date||Hashes View hashes|
Hashes for provenance-0.9.4.1-py3-none-any.whl