Skip to main content

A lightweight Python-3.6+ lib for solving & executing graphs of functions

Project description

Supported Python versions of latest release in PyPi Development Status Latest release in GitHub Latest version in PyPI (build-version: v10.0.0, build-date: 2020-07-19T14:48:12.578522) Travis continuous integration testing ok? (Linux) ReadTheDocs ok? cover-status Code Style Apache License, version 2.0

Github watchers Github stargazers Github forks Issues count

It’s a DAG all the way down!

sample graphtik plot

Lightweight computation graphs for Python

Graphtik is an an understandable and lightweight Python module for executing (& plotting) a graph of functions (a.k.a pipeline) on hierarchical data.

  • The API posits a fair compromise between features and complexity, without precluding any.

  • It can be used as is to build machine learning pipelines for data science projects.

  • It should be extendable to act as the core for a custom ETL engine, a workflow-processor for interdependent tasks & files like GNU Make, or an Excel-like spreadsheet.

Features

  • Can assemble existing functions without modifications into pipelines.

  • dependency resolution can bypass calculation cycles based on data given and asked.

  • Support functions with optional <optionals> input args and/or varargs <varargish>.

  • Support functions with partial outputs; keep working even if certain endured operations fail.

  • Support alias of function provides to avoid the need for trivial conveyor operations.

  • Default conveyor operation to easily pass (possibly nested) dependencies around.

  • Merge <operation merging> or nest <operation nesting> sub-pipelines.

  • Hierarchical dependencies <subdoc> may access data values deep in solution

    with json pointer path expressions.

  • Hierarchical dependencies annotated as implicit imply which subdoc dependency the function reads or writes in the parent-doc.

  • Denote and schedule sideffects on dependency values, to update them repeatedly (e.g. to add columns into pandas.DataFrames).

  • Deterministic pre-decided execution plan.

  • Early eviction of intermediate results from solution, to optimize memory footprint.

  • Solution tracks all intermediate overwritten <overwrite> values for the same dependency.

  • Parallel execution (but underdeveloped).

  • Elaborate plotting with configurable plot themes.

  • Integration with Sphinx sites with the new graphtik directive.

  • Authored with debugging in mind.

Quick start

Here’s how to install:

pip install graphtik

OR with various “extras” dependencies, such as, for plotting:

pip install graphtik[plot]
. Tip::

Supported extras:

plot

for plotting with Graphviz,

matplot

for plotting in maplotlib windows

sphinx

for embedding plots in sphinx-generated sites,

test

for running pytests,

dill

may help for pickling parallel tasks - see marshalling term and set_marshal_tasks() configuration.

all

all of the above, plus development libraries, eg black formatter.

dev

like all

Let’s build a graphtik computation graph that produces x3 outputs out of 2 inputs a and b:

  • a x b

  • a - a x b

  • |a - a x b| ^ 3

>>> from graphtik import compose, operation
>>> from operator import mul, sub
>>> @operation(name="abs qubed",
...            needs=["a_minus_ab"],
...            provides=["abs_a_minus_ab_cubed"])
... def abs_qubed(a):
...     return abs(a) ** 3

Compose the abspow function along the mul & sub built-ins into a computation graph:

>>> graphop = compose("graphop",
...     operation(needs=["a", "b"], provides=["ab"])(mul),
...     operation(needs=["a", "ab"], provides=["a_minus_ab"])(sub),
...     abs_qubed,
... )
>>> graphop
Pipeline('graphop', needs=['a', 'b', 'ab', 'a_minus_ab'],
                    provides=['ab', 'a_minus_ab', 'abs_a_minus_ab_cubed'],
                    x3 ops: mul, sub, abs qubed)

Run the graph and request all of the outputs:

>>> graphop(a=2, b=5)
{'a': 2, 'b': 5, 'ab': 10, 'a_minus_ab': -8, 'abs_a_minus_ab_cubed': 512}

… or request a subset of outputs:

>>> solution = graphop.compute({'a': 2, 'b': 5}, outputs=["a_minus_ab"])
>>> solution
{'a_minus_ab': -8}

… and plot the results (if in jupyter, no need to create the file):

>>> solution.plot('graphop.svg')    # doctest: +SKIP

sample graphtik plot graphtik legend

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

graphtik-10.0.0.tar.gz (161.2 kB view hashes)

Uploaded Source

Built Distribution

graphtik-10.0.0-py2.py3-none-any.whl (109.9 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page