oamap 0.1.0

Toolset for computing directly on hierarchically nested, columnar data, such as Apache Arrow.

Large datasets can be more compact and faster to access when they are laid out in columns (see Apache Arrow). Even hierarchically nested data can be presented this way, though converting the data between the columnar form and the object form can degrade performance. Non-hierarchical data (rectangular tables) is often accessed without materializing rows (see Apache Drill), but this is more complex for data containing arbitrary-length lists of objects.

OAMap is a suite of tools for performing calculations in this way. The name stands for Object-Array-Map, in analogy with Object-Relational-Mapping (ORM) in relational databases. Pure Python calculations are considerably faster and more memory efficient when datasets are expressed in OAMaps, but the real power comes from compiling columized code. This toolset includes extensions to Numba that will compile your object-oriented code into native array manipulations. Generally, you’d use uncompiled Python for low latency exploration of the data and Numba-compiled functions for high throughput.

OAMap only strictly depends on Numpy, but Numba will accelerate it and pyarrow, h5py, etc. provide hooks for converting data among various formats.