Skip to main content

Relation tools for Python.

Project description

https://travis-ci.org/ymoch/reltools.svg?branch=master https://codecov.io/gh/ymoch/reltools/branch/master/graph/badge.svg https://badge.fury.io/py/reltools.svg https://img.shields.io/badge/python-3.5+-blue.svg https://img.shields.io/lgtm/grade/python/g/ymoch/reltools.svg

Relation tools for Python. This relates two data (sorted by certain keys) like SQL joining.

Inspired by itertools.groupby, as long as input data are sorted, almost all processes are evaluated lazily, which results in the reduction of memory usage. This feature is for the big data joining without any SQL engines.

Installation

Install with pip.

pip install reltools

Features

One-To-Many

One-to-many relationing is provided by relate_one_to_many.

Here, input left-hand-side (lhs) and right-hand-side (rhs) are sorted in 1st (and also 2nd) keys.

>>> lhs = [
...     (1, 'a', 's'),
...     (2, 'a', 't'),
...     (3, 'b', 'u'),
... ]
>>> rhs = [
...     (1, 'a', 'v'),
...     (1, 'b', 'w'),
...     (3, 'b', 'x'),
... ]

relate_one_to_many relates rhs items to each lhs item using the first items as keys.

>>> from reltools import relate_one_to_many
>>> one_to_many_related = relate_one_to_many(lhs, rhs)
>>> for left, right in one_to_many_related:
...     left, list(right)
((1, 'a', 's'), [(1, 'a', 'v'), (1, 'b', 'w')])
((2, 'a', 't'), [])
((3, 'b', 'u'), [(3, 'b', 'x')])

You can use custom key functions for not only relate_one_to_many but also API functions.

>>> import operator
>>> custom_key = operator.itemgetter(0, 1)
>>> one_to_many_related = relate_one_to_many(
...     lhs, rhs, lhs_key=custom_key, rhs_key=custom_key)
>>> for left, right in one_to_many_related:
...     left, list(right)
((1, 'a', 's'), [(1, 'a', 'v')])
((2, 'a', 't'), [])
((3, 'b', 'u'), [(3, 'b', 'x')])

OneToManyChainer helps to relate many rhs iterables to lhs.

>>> another_rhs = [
...     ('s', 'f'),
...     ('t', 'g'),
...     ('t', 'h'),
... ]
>>> from reltools import OneToManyChainer
>>> chainer = OneToManyChainer(lhs)
>>> chainer.append(rhs)
>>> chainer.append(
...     another_rhs,
...     lhs_key=operator.itemgetter(2),
...     rhs_key=operator.itemgetter(0),
... )
>>> for left, right, another_right in chainer.chain():
...     left, list(right), list(another_right)
((1, 'a', 's'), [(1, 'a', 'v'), (1, 'b', 'w')], [('s', 'f')])
((2, 'a', 't'), [], [('t', 'g'), ('t', 'h')])
((3, 'b', 'u'), [(3, 'b', 'x')], [])

Left Outer Join

Left outer joining is provided by left_join. While SQL left outer joining returns all the combinations, this returns the pair of items. Note that the right can empty, like SQL left joining.

>>> from reltools import left_join
>>> lhs = [(1, 'a'), (1, 'b'), (2, 'c'), (4, 'd')]
>>> rhs = [(1, 's'), (1, 't'), (3, 'u'), (4, 'v')]
>>> relations = left_join(lhs, rhs)
>>> for left, right in relations:
...     list(left), list(right)
([(1, 'a'), (1, 'b')], [(1, 's'), (1, 't')])
([(2, 'c')], [])
([(4, 'd')], [(4, 'v')])

Right Outer Join

Right outer joining is not supported because it is left-and-right-opposite of left joining. Use left_join(rhs, lhs, rhs_key, lhs_key).

Full Outer Join

Full outer joining, which is an original feature of reltools, is provided by outer_join. In contrast to left_join, full outer joining preserve keys that are only in rhs.

>>> from reltools import outer_join
>>> lhs = [(1, 'a'), (1, 'b'), (2, 'c'), (4, 'd')]
>>> rhs = [(1, 's'), (1, 't'), (3, 'u'), (4, 'v')]
>>> relations = outer_join(lhs, rhs)
>>> for left, right in relations:
...     list(left), list(right)
([(1, 'a'), (1, 'b')], [(1, 's'), (1, 't')])
([(2, 'c')], [])
([], [(3, 'u')])
([(4, 'd')], [(4, 'v')])

Inner Join

Inner joining is provided by inner_join. In contrast to left_join, right cannot be empty, like SQL inner joining.

>>> from reltools import inner_join
>>> relations = inner_join(lhs, rhs)
>>> for left, right in relations:
...     list(left), list(right)
([(1, 'a'), (1, 'b')], [(1, 's'), (1, 't')])
([(4, 'd')], [(4, 'v')])

Many-To-Many

SQL-like many-to-many relationing using an internal table is not supported. This is because reltools supports only sorted data and does not prefer random accessing. To achieve many-to-many relationing, unnormalize data on preproceing and use outer joining or inner joining.

Memory Efficiency

Almost all processes are evaluated lazily, which results in the reduction of memory usage. (You can try the efficiency by commands like RELTOOLS_TRY_COUNT=10000000 python -m doctest README.rst)

>>> import os
>>> n = int(os.environ.get('RELTOOLS_TRY_COUNT', 1000))
>>> lhs = ((i, 'left') for i in range(n))
>>> rhs = ((i, 'right') for i in range(n))
>>> for left, right in relate_one_to_many(lhs, rhs):
...     assert len(list(right)) == 1

Development

This project’s structure is based on Poetry. All tests are written with doctest and run with pytest.

poetry install
poetry run pytest

For stability, following checks are also run when testing.

License

https://img.shields.io/badge/License-MIT-brightgreen.svg

Copyright (c) 2018 Yu MOCHIZUKI

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for reltools, version 0.9.4
Filename, size File type Python version Upload date Hashes
Filename, size reltools-0.9.4-py3-none-any.whl (6.8 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size reltools-0.9.4.tar.gz (7.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page