Skip to main content

DAGpy

DAGpy is a data science collaboration tool based on iPython notebooks enabling data science teams to:

  • easily collaborate by branching out of others’ notebooks

  • minimize code duplication

  • give a clean overview of the project

  • cache intermediate outputs so team members can use them without re-evaluation

  • automate the process of code execution upon data changes or on schedule

  • provide a clean interface to the data visualization dashboard designers and developers

DAGpy manages a DAG (directed acyclic graph) of blocks of code, with each block being a sequence of iPython notebook cells, together with their outputs. It is designed to work seamlessly with popular VC systems like git and can be run locally or as a server application.

Author: Ivan Bestvina

Example project

To play around with the example project, you can:

  • view the project DAG: python program.py view

  • run all the blocks: python program.py execute -a

  • add blocks through flows (with block B as a parent) and run them automatically: python program.py makeflow B -r

  • commit the changes: python program.py submitflow dagpy_flow.ipynb

  • explore other DAGpy options with python program.py -h

Please note that notebook execution time includes a significant overhead of over a second, because a kernel must be started for each one. In future, we plan on adding support for non-notebook plane python blocks. These would also be edited through a flow notebook view, but would be saved as .py scripts, and executed without noticable overhead.

Dependencies:

  • python 3

  • jupyter

  • dill

  • networkx, matplotlib (for DAG view)

Release files for dagpy 0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dagpy 0.1
File Size Uploaded
dagpy-0.1.zip 17.0 kB Details

Release files / dagpy-0.1.zip

Download URL dagpy-0.1.zip
Size 17.0 kB
Tags Source
SHA-256 checksum
How to use checksums
da452c5f8f010b1c4c5956fa947334cb806d29ab3f18036934a2e6c9217b4742
BLAKE2b-256 checksum
How to use checksums
9450b05bc54fc864f32cba106ddaf12c956836f74b25dbb1087b265fb4153453
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

0.3.2

1 release file

0.3

1 release file

0.2

1 release file

This release

0.1 This release

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page