This release is a pre-release and may not be stable for production use.
pynb-dag-runner
pynb-dag-runner is a Python library that can run ML/data pipelines on stateless compute infrastructure (that may be ephemeral or serverless).
This means that pynb-dag-runner does not need a tracking server (or database) to record task outcomes (like logged ML metrics, models, artifacts).
Instead, pipeline outputs are emitted using the OpenTelemetry standard.
Since structured logs can be directed to a file (as one option), this can be used to run pipelines on limited or no cloud infrastructure;
after pipeline execution one only needs to preserve the structured logs.
Documentation and architecture
Demo
-
The below shows a demo ML training pipeline that uses only Github infrastructure (that is: Github actions for compute; Build artifacts for storage; and Github Pages for reporting). This uses
pynb-dag-runnerand a fork of MLFlow that can be deployed as a static website (see, https://github.com/pynb-dag-runner/mlflow). -
Codes for pipeline (MIT): https://github.com/pynb-dag-runner/mnist-digits-demo-pipeline
Roadmap and project planning
Install via PyPI
Latest release
pip install pynb-dag-runner- https://pypi.org/project/pynb-dag-runner
Snapshot of latest commit to main branch
pip install pynb-dag-runner-snapshot- https://pypi.org/project/pynb-dag-runner-snapshot
Any feedback/ideas welcome!
License
(c) Matias Dahl 2021-2022, MIT, see LICENSE.md.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file composable_logs_snapshot-0.0.10.dev1675012618-py3-none-any.whl.
File metadata
- Download URL: composable_logs_snapshot-0.0.10.dev1675012618-py3-none-any.whl
- Upload date:
- Size: 37.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/4.0.1 CPython/3.8.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a11d13f8e43fa3d40ace69f31a8d610009fe21d62eb84bec483dda21056ce79a
|
|
| MD5 |
aeb459a3e3157ed8f546614675e4409b
|
|
| BLAKE2b-256 |
3156c888f36f429c944f3b41999bd520e11c673e0a5521f70e631b27db34c2ea
|
