No project description provided
Project description
pynb-dag-runner
pynb-dag-runner
is a Python library that can run ML/data pipelines on stateless compute infrastructure (that may be ephemeral or serverless).
This means that pynb-dag-runner
does not need a tracking server (or database) to record task outcomes (like logged ML metrics, models, artifacts).
Instead, pipeline outputs are emitted using the OpenTelemetry standard.
Since structured logs can be directed to a file (as one option), this can be used to run pipelines on limited or no cloud infrastructure;
after pipeline execution one only needs to preserve the structured logs.
Documentation and architecture
Demo
-
The below shows a demo ML training pipeline that uses only Github infrastructure (that is: Github actions for compute; Build artifacts for storage; and Github Pages for reporting). This uses
pynb-dag-runner
and a fork of MLFlow that can be deployed as a static website (see, https://github.com/pynb-dag-runner/mlflow). -
Codes for pipeline (MIT): https://github.com/pynb-dag-runner/mnist-digits-demo-pipeline
Roadmap and project planning
Install via PyPI
pip install pynb-dag-runner
(latest release)pip install pynb-dag-runner-snapshot
(dev-release, latest commit to main branch)
Any feedback/ideas welcome!
License
(c) Matias Dahl 2021-2022, MIT, see LICENSE.md.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for pynb_dag_runner_snapshot-0.0.8.dev1664040708-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8367bff8d7f281cdad4f759de8c3a3366a47422d8425da16df884975688be514 |
|
MD5 | 8714cf0a803bf04630e111e7f2409c92 |
|
BLAKE2b-256 | ebac088fa60e46a19e348ee4e4957b7a1b34b0ad52c1b146c72cafbf84046bda |