oamap 0.1.3

Toolset for computing directly on hierarchically nested, columnar data, such as Apache Arrow.

Data analysts are often faced with a choice between speed and flexibility. Tabular data, such as an SQL table or CSV file, can be accessed quickly, which improves the question-and-answer nature of exploratory data analysis. Hierarchically nested data, such as JSON, expresses better the relationship between nested quantities. These relationships can be represented with separate, linked tables (i.e. database normalization), but at the cost of complexity for the data analyst and the introduction of expensive joins (see this question, which got me started on this project). Ideally, we want to perform calculations on JSON-like structures at the speed of SQL.

Tools that analyze tabular data get their performance primarily by laying out data in an intelligent way: computers can access contiguous data more quickly than separated data, whether loading from a disk to memory or from memory to the processor. Datasets with many attributes, of which only a few will be

OAMap, short for Object-Array Mapping and intended

Large datasets can be more compact and faster to access when they are laid out in columns (see Apache Arrow). Even hierarchically nested data can be presented this way, though converting the data between the columnar form and the object form can degrade performance. Non-hierarchical data (rectangular tables, such as an SQL table) can be accessed faster by not materializing rows (see Apache Drill), but this is more complex for data containing variable-length objects, such as arbitrary-length lists.

OAMap is a suite of tools for performing calculations in this way. The name stands for Object-Array-Map, in analogy with Object-Relational-Mapping (ORM) in relational databases. Pure Python calculations are considerably faster and more memory efficient when datasets are expressed in OAMaps, but the real power comes from compiling columized code. This toolset includes extensions to Numba that will compile your object-oriented code into native array manipulations. Generally, you’d use uncompiled Python for low latency exploration of the data and Numba-compiled functions for high throughput.

OAMap only strictly depends on Numpy, but Numba will accelerate it and pyarrow, h5py, etc. provide hooks for converting data among various formats.