CacheFlow is a caching workflow engine, capable of executing dataflows while reusing previous results where appropriate, for efficiency. It is very extensible and can be used in many projects.
Goals
☑ Python 3 workflow system
☑ Executes dataflows from JSON or YAML files
☑ Extensible: can add new components, new storage formats, new caching mechanism, new executors
☐ Pluggable: extensions can be installed from PyPI without forking
☑ Re-usable: can execute workflows by itself, but can also be embedded into applications. Some I plan on developing myself:
☑ Literate programming app: snippets or components embedded into a markdown file, which are executed on render (similar to Rmarkdown). Results would be cached, making later rendering fast
☐ Integrate in some of my NYU research projects (VisTrails, Vizier, D3M)
Other ideas:
☐ Use Jupyter kernels as backends to execute code (giving me quick access to all the languages they support)
☐ Isolate script execution (to run untrusted Python/… code, for example with Docker)
Non-goals
Make a super-scalable and fast workflow execution engine: I’d rather make executors based on Spark, Dask, Ray than try to re-implement those from scratch.
Status
Basic structures are here, extracted from D3M. Execution works. Very few components available. Working on web interface.
1 maintainer
Release files for cacheflow 0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| cacheflow-0.3.tar.gz | 232.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| cacheflow-0.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 472.4 kB
Release files / cacheflow-0.3.tar.gz
| Download URL | cacheflow-0.3.tar.gz |
|---|---|
| Size | 232.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3e9aabef06ae7cbfceb25f3a9e0b672a3ba18a620f8cf957513ddc14caeb0702
|
|
BLAKE2b-256 checksum How to use checksums |
2d02293a2299acd5a4f63188dda52ce341ebbd1f22b4c6e70b28414d225bbb07
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.8.0 tqdm/4.19.9 CPython/3.6.3
|
Release files / cacheflow-0.3-py3-none-any.whl
| Download URL | cacheflow-0.3-py3-none-any.whl |
|---|---|
| Size | 240.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
552569e6207558c132c2fe26eb18fb7b529ef9b0d9622cca7921f2a4af3f1830
|
|
BLAKE2b-256 checksum How to use checksums |
11188677bb2c3b3ca37c286b22abbe61cd0f65fc3f6831f01709670b70082536
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.8.0 tqdm/4.19.9 CPython/3.6.3
|
