Skip to main content

Utilities for working with hierarchical data structures.

Project description

* Summary

This is a collection of functions I find I've been writing over and over again.

Often times I'm working with hierarchical data structures and attempting
to apply transformations on them to yield a similar-but-different output
hierarchical data structures.

There are lots of ways to do this... these are just utilities for reference to
do so.

Input could be nested json, xml, etc and there are [[][various ways]] to get this into
a python dictionary.

This python dictionary however still retains the complexity of the original

This module provides some utility functions (in the spirit of =toolz= / =functoolz=)
that can be used to grok hierarchies.

* Hierarchy Path Concepts

Many of the functions receiver a /hierarchy path/ (=hp=).

Given the example data such as:

#+BEGIN_SRC python
teams = {"compile-day": "monday",
"compile-secret": "SECRET!",
"nhl": [{"team": "stars" , "players": 10, "pos": ["l", "r", "c"]},
{"team": "bruins" , "players": 30 },
{"team": "preds" , "players": 90 }],
"nba": [{"team": "mavs" , "players": -9, "details": [{"who": "bill", "pos": ["r", "c", "l"]},
{"who": "ted", "pos": ["d"]},
{"who": "fred"}]},
{"team": "bucks" , "players": 3 , "details": [{"who": "ken" , "pos": ["c"]}]}],}

Some /hierarchy paths/ could look like this:

#+BEGIN_SRC python

Or any of these:
#+BEGIN_SRC python

Notice the use of numbers and/or =*= in the Hierarchy Path that acts as a
index-of-list identifier or a globing.

This idea of a hierarchy path wouldn't work with dictionaries that had numeric
keys like =0= and/or dictionaries that have keys such 's =*=. This hasn't been
an issue for me, as most of my dictionaries are generated from xml, and these
keys wouldn't be [[][valid tags in xml]].

* Function listing and Samples
** =get_in_hp=

Like toolz's =get_in= but works with a Hierarchy Path:

#+BEGIN_SRC python
get_in_hp("/compile-day", teams)
# Output:

get_in_hp("/nba/*/details/*/who", teams)
# Output:
[['bill', 'ted', 'fred'], ['ken']]

get_in_hp('/nhl/0/players' , teams)
# Output:

get_in_hp('/nba/*/details/*/' , teams)
# Output:
[[{'pos': ['r', 'c', 'l'], 'who': 'bill'},
{'pos': ['d'], 'who': 'ted'},
{'who': 'fred'}],
[{'pos': ['c'], 'who': 'ken'}]]

get_in_hp('/nba/*/details/*/who' , teams)
# Output:
[['bill', 'ted', 'fred'], ['ken']]

** =flatten_recur=

Flattens recursive lists.

#+BEGIN_SRC python

flatten_recur([['bill', 'ted', 'fred'], ['ken']])
#=> ['bill', 'ted', 'fred', 'ken']

flatten_recur(get_in_hp("/nba/*/details/*/who", teams) )
#=> =['bill', 'ted', 'fred', 'ken']


** =assoc_in_hp=
Like toolz's =assoc_in= but works with a hierarchy path:

#+BEGIN_SRC python
# change team 0's name:
assoc_in_hp({"nhl":[{"team": "stars" , "players": 10},
{"team": "preds" , "players": 90}]},

# Output:
{'nhl': [{'team': 'STARS!', 'players': 10},
{'team': 'preds', 'players': 90}]}

# Change team 1's players count:
assoc_in_hp({"nhl":[{"team": "stars" , "players": 10},
{"team": "preds" , "players": 90}]},

# Output:
{'nhl': [{'players': 10, 'team': 'stars'},
{'players': 40, 'team': 'preds'}]}

# Should work with deeply nested lists too;
# Change the second position on the first team to "coach!"
assoc_in_hp({"nhl":[{"team": "stars" , "players": 10, "pos": ["l", "r", "c"]},
{"team": "preds" , "players": 90}]},

# Output:
{'nhl': [{'players': 10, 'pos': ['l', 'r', 'coach!'], 'team': 'stars'},
{'players': 90, 'team': 'preds'}]}

** =update_in_hp=
Similar to toolz's =update_in=.

Updates an input dictionaries value with the result of calling a function on
that value.

#+BEGIN_SRC python

my_fn = lambda x: x + 1

update_in_hp({"nhl":[{"team": "stars" , "players": 10},
{"team": "preds" , "players": 90}]},
my_fn, default=0)

{"nhl":[{"team": "stars" , "players": 11},
{"team": "preds" , "players": 90}]},

It can gets interesting (hopefully useful?) with wildcards.

Lets pretend that we want to apply a 10% raise to all players across the board,
on all teams.

This would be our function:

#+BEGIN_SRC python
def ten_percent_raise(salary):
if salary:
return salary * 1.10
# If they don't have a salary then lets pretend they own us money.
return -40

Then just apply that function to the correct path for the input dictionary:
#+BEGIN_SRC python
update_in_hp({"payroll": [{"team": "stars", "players": [{"player": "bill" , "salary": 10, },
{"player": "ted" , "salary": 30, },
{"player": "ned" , "salary": 20, },
{"player": "fred" , },]},
{"team": "preds", "players": [{"player": "ken" , "salary": 5, },
{"player": "jen" , "salary": 8, },
{"player": "ben" , "salary": 9, },
{"player": "len" , },]}]},
"/payroll/*/players/*/salary", # path is payroll for all teams, all players, salary node.

{'payroll': [{'players': [{'player': 'bill', 'salary': 11.0},
{'player': 'ted', 'salary': 33.0},
{'player': 'ned', 'salary': 22.0},
{'player': 'fred', 'salary': -40}],
'team': 'stars'},
{'players': [{'player': 'ken', 'salary': 5.5},
{'player': 'jen', 'salary': 8.8},
{'player': 'ben', 'salary': 9.9},
{'player': 'len', 'salary': -40}],
'team': 'preds'}]}


Here's another (similar) example; Lets b64 encode all the players ssn's:

#+BEGIN_SRC python
import base64

def make_data_not_super_secure(ssn):
return base64.b64encode(ssn.encode()).decode("utf-8")

update_in_hp({"payroll": [{"team": "stars", "players": [{"player": "bill" , "salary": 10, "ssn": "001-01-0001"},
{"player": "ted" , "salary": 30, "ssn": "001-01-0002"},
{"player": "ned" , "salary": 20, "ssn": "001-01-0003"},
{"player": "fred" , "ssn": "001-01-0004"},]},
{"team": "preds", "players": [{"player": "ken" , "salary": 5, "ssn": "001-01-0005"},
{"player": "jen" , "salary": 8, "ssn": "001-01-0006"},
{"player": "ben" , "salary": 9, "ssn": "001-01-0007"},
{"player": "len" , "ssn": "001-01-0008"},]}]},
"/payroll/*/players/*/ssn", # path is payroll for all teams, all players, salary node.

{'payroll': [{'players': [{'player': 'bill', 'salary': 10, 'ssn': 'MDAxLTAxLTAwMDE='},
{'player': 'ted', 'salary': 30, 'ssn': 'MDAxLTAxLTAwMDI='},
{'player': 'ned', 'salary': 20, 'ssn': 'MDAxLTAxLTAwMDM='},
{'player': 'fred', 'ssn': 'MDAxLTAxLTAwMDQ='}],
'team': 'stars'},
{'players': [{'player': 'ken', 'salary': 5, 'ssn': 'MDAxLTAxLTAwMDU='},
{'player': 'jen', 'salary': 8, 'ssn': 'MDAxLTAxLTAwMDY='},
{'player': 'ben', 'salary': 9, 'ssn': 'MDAxLTAxLTAwMDc='},
{'player': 'len', 'ssn': 'MDAxLTAxLTAwMDg='}],
'team': 'preds'}]}


* TODO test results
- [ ] add this to jenkins/ci

#+BEGIN_SRC bash
cwd: /Users/me/git-repos/hierarchy_utils/
cmd: pytest --color=no

==================================================================================================================================================== test session starts =====================================================================================================================================================
platform darwin -- Python 3.7.3, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /usr/local/opt/python/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/me/git-repos/hierarchy_utils, inifile: pytest.ini
collected 10 items

tests/ PASSED
tests/ PASSED
tests/ PASSED
tests/ PASSED
tests/ PASSED
tests/ PASSED
tests/ PASSED
tests/ PASSED
tests/ PASSED
tests/ PASSED

================================================================================================================================================= 10 passed in 0.09 seconds ==================================================================================================================================================

* Ideas/References

Probably could do a lot of this with xlst... or a billion other ways.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for hierarchy-utils, version 0.1.3
Filename, size File type Python version Upload date Hashes
Filename, size hierarchy_utils-0.1.3-py3-none-any.whl (6.9 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size hierarchy_utils-0.1.3.tar.gz (8.5 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page