Skip to main content

A Python lib for solving & executing graphs of functions, with `pandas` in mind

Project description

Latest version in PyPI Latest release in GitHub (build-version: v10.4.0, build-date: 2020-10-09T11:27:24.502636) Supported Python versions of latest release in PyPi Development Status Travis continuous integration testing ok? (Linux) ReadTheDocs ok? cover-status Code Style Apache License, version 2.0

Github watchers Github stargazers Github forks Issues count

It’s a DAG all the way down!

sample graphtik plot

Computation graphs for Python & Pandas

Graphtik is a library to compose, plot & execute graphs of python functions (a.k.a pipelines) that consume and populate (possibly nested) named data (a.k.a dependencies), based on whether values for those dependencies exist in the inputs or have been calculated earlier, with pandas in mind.

  • Its primary use case is building flexible algorithms for data science/machine learning projects.
  • It should be extendable to implement the following:
    • an IoC dependency resolver (e.g. Java Spring);
    • an executor of interdependent tasks based on files (e.g. GNU Make);
    • a custom ETL engine;
    • a spreadsheet calculation engine.

Graphtik sprang from Graphkit (summer 2019, v1.2.2) to experiment with Python 3.6+ features, but has diverged significantly with enhancements ever since.

Features

  • Deterministic pre-decided execution plan (excepting partial-outputs or endured operations, see below).
  • Can assemble existing functions without modifications into pipelines.
  • dependency resolution can bypass calculation cycles based on data given and asked.
  • Support functions with optional <optionals> input args and/or varargs <varargish>.
  • Support functions with partial outputs; keep working even if certain endured operations fail.
  • Facilitate trivial conveyor operations and alias on provides.
  • Support cycles, by annotating repeated updates of dependency values as sideffects, (e.g. to add columns into pandas.DataFrames).
  • Hierarchical dependencies <subdoc> may access data values deep in solution with json pointer path expressions.
  • Hierarchical dependencies annotated as implicit imply which subdoc dependency the function reads or writes in the parent-doc.
  • Merge <operation merging> or nest <operation nesting> sub-pipelines.
  • Early eviction of intermediate results from solution, to optimize memory footprint.
  • Solution tracks all intermediate overwritten <overwrite> values for the same dependency.
  • Parallel execution (but underdeveloped).
  • Elaborate Graphviz plotting with configurable plot themes.
  • Integration with Sphinx sites with the new graphtik directive.
  • Authored with debugging in mind.
  • Parallel execution (but underdeveloped & deprecated).

Anti-features

  • It’s not an orchestrator for long-running tasks, nor a calendar scheduler - Apache Airflow, Dagster or Luigi may help for that.
  • It’s not really a parallelizing optimizer, neither a map-reduce framework - look additionally at Dask, IpyParallel, Celery, Hive, Pig, Spark, Hadoop, etc.

Quick start

Here’s how to install:

pip install graphtik

OR with various “extras” dependencies, such as, for plotting:

pip install graphtik[plot]
. Tip::

Supported extras:

plot
for plotting with Graphviz,
matplot
for plotting in maplotlib windows
sphinx
for embedding plots in sphinx-generated sites,
test
for running pytests,
dill
may help for pickling parallel tasks - see marshalling term and set_marshal_tasks() configuration.
all
all of the above, plus development libraries, eg black formatter.
dev
like all

Let’s build a graphtik computation graph that produces x3 outputs out of 2 inputs α and β:

  • α x β
  • α - αxβ
  • |α - αxβ| ^ 3
>>> from graphtik import compose, operation
>>> from operator import mul, sub
>>> @operation(name="abs qubed",
...            needs=["α-α×β"],
...            provides=["|α-α×β|³"])
... def abs_qubed(a):
...     return abs(a) ** 3

Compose the abspow function along the mul & sub built-ins into a computation graph:

>>> graphop = compose("graphop",
...     operation(needs=["α", "β"], provides=["α×β"])(mul),
...     operation(needs=["α", "α×β"], provides=["α-α×β"])(sub),
...     abs_qubed,
... )
>>> graphop
Pipeline('graphop', needs=['α', 'β', 'α×β', 'α-α×β'],
                    provides=['α×β', 'α-α×β', '|α-α×β|³'],
                    x3 ops: mul, sub, abs qubed)

Run the graph and request all of the outputs (notice that unicode characters work also as Python identifiers):

>>> graphop(α=2, β=5)
{'α': 2, 'β': 5, 'α×β': 10, 'α-α×β': -8, '|α-α×β|³': 512}

… or request a subset of outputs:

>>> solution = graphop.compute({'α': 2, 'β': 5}, outputs=["α-α×β"])
>>> solution
{'α-α×β': -8}

… and plot the results (if in jupyter, no need to create the file):

>>> solution.plot('executed_3ops.svg')  # doctest: +SKIP

sample graphtik plot

graphtik legend

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for graphtik, version 10.4.0
Filename, size File type Python version Upload date Hashes
Filename, size graphtik-10.4.0-py2.py3-none-any.whl (123.2 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size graphtik-10.4.0.tar.gz (183.0 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page