A project for feeding various nested data formats into pandas
Project description
<!---
Copyright (c) 2019 Michael Vilim
This file is part of the bamboo library. It is currently hosted at
https://github.com/mvilim/bamboo
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
## bamboo
[![PyPI Release](https://img.shields.io/pypi/v/bamboo-nested.svg)](https://pypi.org/project/bamboo-nested/)
[![Build Status](https://travis-ci.org/mvilim/bamboo.svg?branch=master)](https://travis-ci.org/mvilim/bamboo)
bamboo is a library for feeding nested data formats into pandas. The space of data representable in nested formats is larger than the space covered by pandas. pandas supports only data representable in a flat table (though things like multi-indexs allows certain types of tree formats to be efficiently projected into a table). Data which supports arbitrary nesting is not in general convertible to a pandas dataframe. In particular, data which contains multiple repetition structures (e.g. JSON arrays) that are not nested within each other will not be flattenable into a table.
As a simple example
The current data formats supported are:
* JSON
* Apache Avro
* Apache Arrow
* Profobuf (via [PBD](https://github.com/mvilim/pbd))
bamboo works by projecting a flattenable portion (a subset of the nested columns) of the data into a pandas dataframe. By projecting various combinations of columns, one can make use of all the relationships implied by the nested structure of the data.
### Installation
To install from PyPI:
```
pip install bamboo-nested
```
### Example
A minimal example of flattening a JSON string:
```
from bamboo import from_json
obj = [{'a': None, 'b': [1, 2], 'c': [5, 6]}, {'a': -1.0, 'b': [3, 4], 'c': [7, 8]}]
node = from_json(json.dumps(obj))
> - a float64
> - b []uint64
> - c []uint64
```
Flattening just the values of column `a`:
```
df_a = node.flatten(include=['a'])
> a
> 0 NaN
> 1 -1.0
```
Flattening columns `a` and `b` (note that column `a` is repeated to match the corresponding elements of column `b`):
```
df_ab = node.flatten(include=['a', 'b'])
> a b
> 0 NaN 1
> 1 NaN 2
> 2 -1.0 3
> 3 -1.0 4
```
Trying to flatten two repetition lists at the same level will lead to an error (as this structure is unflattenable without taking a Cartesian product):
```
df_bc = node.flatten(include=['b', 'c'])
> ValueError: Attempted to flatten conflicting lists
```
### Building
To build this project:
Building from source requires cmake (`pip install cmake`) and Boost.
```
python setup.py
```
### Unit tests
To run the unit tests:
```
python setup.py test
```
or use nose:
```
nosetests python/bamboo/tests
```
### Licensing
This project is licensed under the [Apache 2.0 License](https://github.com/mvilim/bamboo/blob/master/LICENSE). It uses the pybind11, Arrow, Avro, nlohmann JSON, and PBD projects. The licenses can be found in those [projects' directories](https://github.com/mvilim/bamboo/blob/master/cpp/thirdparty).
Copyright (c) 2019 Michael Vilim
This file is part of the bamboo library. It is currently hosted at
https://github.com/mvilim/bamboo
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
## bamboo
[![PyPI Release](https://img.shields.io/pypi/v/bamboo-nested.svg)](https://pypi.org/project/bamboo-nested/)
[![Build Status](https://travis-ci.org/mvilim/bamboo.svg?branch=master)](https://travis-ci.org/mvilim/bamboo)
bamboo is a library for feeding nested data formats into pandas. The space of data representable in nested formats is larger than the space covered by pandas. pandas supports only data representable in a flat table (though things like multi-indexs allows certain types of tree formats to be efficiently projected into a table). Data which supports arbitrary nesting is not in general convertible to a pandas dataframe. In particular, data which contains multiple repetition structures (e.g. JSON arrays) that are not nested within each other will not be flattenable into a table.
As a simple example
The current data formats supported are:
* JSON
* Apache Avro
* Apache Arrow
* Profobuf (via [PBD](https://github.com/mvilim/pbd))
bamboo works by projecting a flattenable portion (a subset of the nested columns) of the data into a pandas dataframe. By projecting various combinations of columns, one can make use of all the relationships implied by the nested structure of the data.
### Installation
To install from PyPI:
```
pip install bamboo-nested
```
### Example
A minimal example of flattening a JSON string:
```
from bamboo import from_json
obj = [{'a': None, 'b': [1, 2], 'c': [5, 6]}, {'a': -1.0, 'b': [3, 4], 'c': [7, 8]}]
node = from_json(json.dumps(obj))
> - a float64
> - b []uint64
> - c []uint64
```
Flattening just the values of column `a`:
```
df_a = node.flatten(include=['a'])
> a
> 0 NaN
> 1 -1.0
```
Flattening columns `a` and `b` (note that column `a` is repeated to match the corresponding elements of column `b`):
```
df_ab = node.flatten(include=['a', 'b'])
> a b
> 0 NaN 1
> 1 NaN 2
> 2 -1.0 3
> 3 -1.0 4
```
Trying to flatten two repetition lists at the same level will lead to an error (as this structure is unflattenable without taking a Cartesian product):
```
df_bc = node.flatten(include=['b', 'c'])
> ValueError: Attempted to flatten conflicting lists
```
### Building
To build this project:
Building from source requires cmake (`pip install cmake`) and Boost.
```
python setup.py
```
### Unit tests
To run the unit tests:
```
python setup.py test
```
or use nose:
```
nosetests python/bamboo/tests
```
### Licensing
This project is licensed under the [Apache 2.0 License](https://github.com/mvilim/bamboo/blob/master/LICENSE). It uses the pybind11, Arrow, Avro, nlohmann JSON, and PBD projects. The licenses can be found in those [projects' directories](https://github.com/mvilim/bamboo/blob/master/cpp/thirdparty).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
bamboo-nested-0.0.8.tar.gz
(18.7 kB
view hashes)
Built Distributions
Close
Hashes for bamboo_nested-0.0.8-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f80156103e33043a21f1eafbecf3f4f95863b11a50d18b2d21d84ca6e3fc6f9 |
|
MD5 | 4acb5875eea83eb28e04e45f15aaf9a6 |
|
BLAKE2b-256 | 2b542540b925631188f71d2de0df6ce90c7c926726efc36f8273fcd802b46760 |
Close
Hashes for bamboo_nested-0.0.8-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef6bac8936d3e4a642585a38eae566ee7ba1a98a94e440108d9496dd37e51034 |
|
MD5 | 84c101d883ed2d2477c4006621a49e2a |
|
BLAKE2b-256 | 71f78ecd68f3f690f7c673d84f80a1b9726979551516088d9dc6bf78b9c40717 |
Close
Hashes for bamboo_nested-0.0.8-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f4f3bc20c7ea9e957755071ec71c95fd7d699b8347b6f6ed9132a97ed8b0e1d |
|
MD5 | 91a473d813e0ebf2e740eb94828e3ce3 |
|
BLAKE2b-256 | 866903ac3e30cb007ec1cb6c18b77036d394b2ec2122635024547ad4a685bd75 |
Close
Hashes for bamboo_nested-0.0.8-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4abae6ad10b2128e3065e65742904147fc7968a1866604a769e7191b90b775d1 |
|
MD5 | 276a37903de1786ffa64c9199e0d82a5 |
|
BLAKE2b-256 | aa8fc776b1c18e6f1a182d677ec855f350df48b57cbb91b1099b49987a795b1c |
Close
Hashes for bamboo_nested-0.0.8-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d0d325057a5e55bb1c0817236b041cd1547e65aa4346720e12777769aaeca4d |
|
MD5 | cd35a83effcf566284a630d57e52749b |
|
BLAKE2b-256 | 3f185a0fcaa36e777e9641aca70f041806a6e9a8540189f13176952735b408b9 |