No project description provided
Project description
pynb-dag-runner
pynb-dag-runner
is a Python library that can run ML/data pipelines on stateless compute infrastructure (that may be ephemeral or serverless).
This means that pynb-dag-runner
does not need a tracking server (or database) to record task outcomes (like logged ML metrics, models, artifacts).
Instead, pipeline outputs are emitted using the OpenTelemetry standard.
Since structured logs can be directed to a file (as one option), this can be used to run pipelines on limited or no cloud infrastructure;
after pipeline execution one only needs to preserve the structured logs.
Documentation and architecture
Demo
-
The below shows a demo ML training pipeline that uses only Github infrastructure (that is: Github actions for compute; Build artifacts for storage; and Github Pages for reporting). This uses
pynb-dag-runner
and a fork of MLFlow that can be deployed as a static website (see, https://github.com/pynb-dag-runner/mlflow). -
Codes for pipeline (MIT): https://github.com/pynb-dag-runner/mnist-digits-demo-pipeline
Roadmap and project planning
Install via PyPI
Latest release
pip install pynb-dag-runner
- https://pypi.org/project/pynb-dag-runner
Snapshot of latest commit to main branch
pip install pynb-dag-runner-snapshot
- https://pypi.org/project/pynb-dag-runner-snapshot
Any feedback/ideas welcome!
License
(c) Matias Dahl 2021-2022, MIT, see LICENSE.md.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
File details
Details for the file composable_logs_snapshot-0.0.10.dev1674409880-py3-none-any.whl
.
File metadata
- Download URL: composable_logs_snapshot-0.0.10.dev1674409880-py3-none-any.whl
- Upload date:
- Size: 44.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.8.10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f6e713fdc46006e0b0a6494abf8298c8796970fa5754fb1000736df3cee9306d |
|
MD5 | 8ea64031e053368ecbaab26947fb3ef9 |
|
BLAKE2b-256 | 76e5efc1130058475b5f31bd6fceacb31d451505b87e8a8105e87d96e2d6e291 |