darr 0.3.1

Memory-mapped numeric arrays, based on a format that is self-explanatory and tool-independent

Darr is a Python science library that enables you to store and access disk-based numeric arrays, without depending on tool-specific data formats. This makes it easy to access the same data in many different languages and on different analysis platforms. No exporting required and, as the data is saved in a self-explanatory way, not much explanation required either when sharing or archiving your data. Tool-independent and easy access to data is in line with good scientific practice as it promotes wide and long-term availability, to others but also to yourself. More rationale for this approach is provided here.

Darr supports efficient read/write/append access and is based on universally readable flat binary files and automatically generated text files, containing human-readable explanation of precisely how your binary data is stored. It also provides specific code that reads the data in a variety of current scientific data tools such as Python, R, Julia, IDL, Matlab, Maple, and Mathematica (see example array).

Darr currently supports numerical N-dimensional arrays, and experimentally supports numerical ragged arrays, i.e. a series of arrays in which one dimension varies in length.

See this tutorial for a brief introduction, or the documentation for more info.

Darr is currently pre-1.0, still undergoing significant development. However we have been using it in practice in our lab for more than a year on both Linux and Windows machines. It is open source and freely available under the New BSD License terms.

Features

Pro’s:

• Data storage purely based on flat binary and text files, tool independence.

• Human-readable explanation of how the binary data is stored is saved in a README text file.

• README includes examples of how to read the particular array in popular analysis environments such as Python (without Darr), R, Julia, Octave/Matlab, GDL/IDL, and Mathematica.

• Supports very large data arrays, larger than RAM, through memory-mapping.

• Data read/write access is simple and powerful through NumPy indexing (see here).

• Data is easily appendable.

• Many numeric types are supported: (u)int8-(u)int64, float16-float64, complex64, complex128.

• Easy use of metadata, stored in a separate JSON text file.

• Minimal dependencies, only NumPy.

• Integrates easily with the Dask library for numeric computation on very large arrays.

• Supports ragged arrays (still experimental).

See the documentation for more information.