DataJet

A Data Dependency Graph Framework and Executor

DataJet abstracts over function calls by mapping inputs through a graph of functions to desired outputs. As a programmer, you declare your data transformations (functions of inputs to outputs) once, and datajet will handle mapping any input to any output reachable by the graph of functions.

Key Features

• Lazy: Only Evaluate and return the data you need
• Declarative: Declare Data and functions on the data explicitly, using plain python
• Dependency-Free: Just Python.

Installation

Requirements:

• Python >=3.8

To Get Started, Install DataJet From pypi:

pip install datajet

Why would I use this?

• DataJet simplifies the codebase of dynamic systems with mutliple ways to calculate a datapoints from different inputs.
• DataJet de-couples downstream calculations from the mechanics of calculating upstream dependencies.

Quickstart

from datajet import execute

dollars = [7.98, 20.94, 37.9, 30.31]
units =  [1, 3, 5, 4,]

def prices(dollars, units):
    return [d/u for d, u in zip(dollars, units)]

def average_price(prices):
    return sum(prices) / len(prices) 

def average_price_rounded_down(average_price):
    return average_price * 1000 // 10 / 100


datajet_map = {
    "prices": prices,
    "average_price": average_price,
    "average_price_rounded_down": average_price_rounded_down,
}
execute(
        datajet_map,
        context={
            "dollars": dollars,
            "units": units,
        }, 
        fields=['average_price_rounded_down']
)
{'average_price_rounded_down': 7.52}

And, if you have prices, you can directly get what you need:

prices = [3.99, 4.49, 2.89, 2.79, 2.99]

execute(datajet_map,context={"prices": prices,}, fields=['average_price', 'average_price_rounded_down'])
{'average_price': 3.4299999999999997, 'average_price_rounded_down': 3.42}

Important Details

Keys can be any hashable. The value corresponding to each key can be a function or an object. The functions can have 0 or more parameters. The parameter names must correspond to other keys in the dict if no explicitly defined inputs to the callable are declared in the map. See Datamap reference for how to explicitly define inputs.

You can also define multiple ways of calculating a piece of data via defining a list of functions as the value to the key. Again, each function's parameters must correspond to other keys in the dict, or else you can define which other keys should be inputs to the function via explicitly defining inputs.

Full Documentation

https://bmritz.github.io/datajet/

Development

To create the development environment locally:

git clone
make install

This will start a poetry shell that has all the dev dependencies installed. You can run deactivate to exit the shell.

To run tests

make test

Development troubleshooting

If you see:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

Go to /Applications/Python3.x and run 'Install Certificates.command'

Built on ideas inspired by

• wilkerlucio/Pathom3
• stitchfix/hamilton