Memory-mapped numeric arrays, based on a format that is self-explanatory and tool-independent
Darr is a Python science library that enables you to store and access disk-based numeric arrays, without depending on tool-specific data formats. This makes it easy to access the same data in many different languages and on different analysis platforms. No exporting required and, as the data is saved in a self-explanatory way, not much explanation required either when sharing or archiving your data. Tool-independent and easy access to data is in line with good scientific practice as it promotes wide and long-term availability, to others but also to yourself. More rationale for this approach is provided here.
Darr supports efficient read/write/append access and is based on universally readable flat binary files and automatically generated text files, containing human-readable explanation of precisely how your binary data is stored. It also provides specific code that reads the data in a variety of current scientific data tools such as Python, R, Julia, IDL, Matlab, Maple, and Mathematica (see example array).
Darr currently supports numerical N-dimensional arrays, and experimentally supports numerical ragged arrays, i.e. a series of arrays in which one dimension varies in length.
Darr is currently pre-1.0, still undergoing significant development. However we have been using it in practice in our lab for more than a year on both Linux and Windows machines. It is open source and freely available under the New BSD License terms.
- Data storage purely based on flat binary and text files, tool independence.
- Human-readable explanation of how the binary data is stored is saved in a README text file.
- README includes examples of how to read the particular array in popular analysis environments such as Python (without Darr), R, Julia, Octave/Matlab, GDL/IDL, and Mathematica.
- Supports very large data arrays, larger than RAM, through memory-mapping.
- Data read/write access is simple and powerful through NumPy indexing (see here).
- Data is easily appendable.
- Many numeric types are supported: (u)int8-(u)int64, float16-float64, complex64, complex128.
- Easy use of metadata, stored in a separate JSON text file.
- Minimal dependencies, only NumPy.
- Integrates easily with the Dask library for numeric computation on very large arrays.
- Supports ragged arrays (still experimental).
See the documentation for more information.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size darr-0.3.3-py3-none-any.whl (45.9 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size darr-0.3.3.tar.gz (58.9 kB)||File type Source||Python version None||Upload date||Hashes View|