pynb-dag-runner-snapshot 0.0.9.dev1667143816

No project description provided

pynb-dag-runner

pynb-dag-runner is a Python library that can run ML/data pipelines on stateless compute infrastructure (that may be ephemeral or serverless).

This means that pynb-dag-runner does not need a tracking server (or database) to record task outcomes (like logged ML metrics, models, artifacts). Instead, pipeline outputs are emitted using the OpenTelemetry standard. Since structured logs can be directed to a file (as one option), this can be used to run pipelines on limited or no cloud infrastructure; after pipeline execution one only needs to preserve the structured logs.

Documentation and architecture

• https://pynb-dag-runner.github.io/pynb-dag-runner

Demo

• The below shows a demo ML training pipeline that uses only Github infrastructure (that is: Github actions for compute; Build artifacts for storage; and Github Pages for reporting). This uses pynb-dag-runner and a fork of MLFlow that can be deployed as a static website (see, https://github.com/pynb-dag-runner/mlflow).

  

• Codes for pipeline (MIT): https://github.com/pynb-dag-runner/mnist-digits-demo-pipeline

Roadmap and project planning

• https://github.com/orgs/pynb-dag-runner/projects/2/views/3

Install via PyPI

Latest release

• pip install pynb-dag-runner
• https://pypi.org/project/pynb-dag-runner

Snapshot of latest commit to main branch

• pip install pynb-dag-runner-snapshot
• https://pypi.org/project/pynb-dag-runner-snapshot

---

Any feedback/ideas welcome!

License

(c) Matias Dahl 2021-2022, MIT, see LICENSE.md.