darr 0.5.3

Memory-mapped numeric arrays, based on a format that is self-explanatory and tool-independent

Darr is a Python library that stores NumPy arrays on disk in a way that is simple and self-documented, which makes them easily accessible from a wide range of computing environments. Arrays are automatically kept up-to-date with a full explanation of how data is stored, including code to read itself in languages such as R, Julia, IDL, Matlab, Maple, and Mathematica, or in Python/Numpy without Darr (see example). Keeping data universally readable and documented is a pillar of good scientific practice, and a good idea in general. More rationale for a tool-independent approach to numeric array storage is provided here.

Under the hood, Darr uses NumPy memory-mapped arrays, which is a widely established and trusted way of working with disk-based numerical arrays, and which makes Darr fully NumPy compatible. This enables efficient out-of-core read/write access to potentially very large arrays. What Darr adds is that it automatically keeps your arrays fully documented, open, and thus widely readable. Further, Darr adds functionality to make your life easier in other ways, such as the support for ragged arrays, the ability to create arrays from iterators, append and truncate functionality, and the easy use of metadata.

Flat binary files and (JSON) text files are accompanied by a README text file that explains how the array and metadata are stored (see example arrays). It is trivially easy to share your arrays with others or with yourself when working in different computing environments because they always contains clear documentation of the specific data at hand, including code to read it. Does your colleague want to try out an interesting algorithm in R or Matlab on your arrays? No need to export anything or to provide elaborate explanation. No dependence on complicated formats or specialized libraries. No looking up things. A copy-paste of a few lines of code from the documentation stored with the data is sufficient. Self-documentation and code examples are automatically updated as you change your arrays when working with them.

See this tutorial for a brief introduction, or the documentation for more info.

Darr is currently pre-1.0, still undergoing development. It is open source and freely available under the New BSD License terms.

Features

• Data is stored purely based on flat binary and text files, maximizing universal readability.

• Automatic self-documention, including copy-paste ready code snippets for reading the array in a number of popular data analysis environments, such as Python (without Darr), R, Julia, Octave/Matlab, GDL/IDL, and Mathematica (see example array).

• Disk-persistent array data is directly accessible through NumPy indexing and may be larger than RAM and that is easily appendable.

• Supports ragged arrays.

• Easy use of metadata, stored in a widely readable separate JSON text file.

• Many numeric types are supported: (u)int8-(u)int64, float16-float64, complex64, complex128.

• Integrates easily with the Dask library for out-of-core computation on very large arrays.

• Minimal dependencies, only NumPy.

See the documentation for more information.