darr 0.2.0

Memory-mapped numeric arrays, based on a format that is self-explanatory and tool-independent

Darr is a Python science library that enables you to work efficiently with disk-based numeric arrays without depending on tool-specific data formats. This makes it easy to share your data with those who do not use Darr or even Python. No exporting required and, as the data is saved in a self-explanatory way, not much explanation required either. Tool-independent and easy access to data is in line with good scientific practice as it promotes wide and long-term availability, to others but also to yourself. More rationale for this approach is provided here.

Darr supports efficient read/write/append access and is based on universally readable flat binary files and automatically generated text files, containing human-readable explanation of precisely how your binary data is stored. It also provides specific code that reads the data in a variety of current scientific data tools such as Python, R, Julia, IDL, Matlab, Maple, and Mathematica (see example array).

Darr currently supports numerical N-dimensional arrays, and experimentally supports numerical ragged arrays, i.e. a series of arrays in which one dimension varies in length.

See this tutorial for a brief introduction, or the documentation for more info.

Darr is currently pre-1.0, still undergoing significant development. It is open source and freely available under the New BSD License terms.

Features

Pro’s:

• Purely based on flat binary and text files, tool independence.

• Human-readable explanation of how the binary data is stored is saved in a README text file.

• Includes examples of how to read the array in popular analysis environments such as Python (without Darr), R, Julia, Octave/Matlab, GDL/IDL, and Mathematica.

• Supports very large data arrays, larger than RAM.

• Data read/write access is simple through NumPy indexing (see here).

• Data is easily appendable.

• Many numeric types are supported: (u)int8-(u)int64, float16-float64, complex64, complex128.

• Easy use of metadata, stored in a separate JSON text file.

• Minimal dependencies, only NumPy.

• Integrates easily with the Dask library for numeric computation on very large Darr arrays.

• Supports ragged arrays (still experimental).

See the [documentation](http://darr.readthedocs.io/) for more information.