Skip to main content

Memory-mapped numeric arrays, based on a format that is self-explanatory and tool-independent

Project description

Darr is a Python science library that enables you to store and access disk-based numeric arrays, without depending on tool-specific data formats. This makes it easy to access the same data in many different languages and on different analysis platforms. No exporting required and, as the data is saved in a self-explanatory way, not much explanation required either when sharing or archiving your data. Tool-independent and easy access to data is in line with good scientific practice as it promotes wide and long-term availability, to others but also to yourself. More rationale for this approach is provided here.

Darr supports efficient read/write/append access and is based on universally readable flat binary files and automatically generated text files, containing human-readable explanation of precisely how your binary data is stored. It also provides specific code that reads the data in a variety of current scientific data tools such as Python, R, Julia, IDL, Matlab, Maple, and Mathematica (see example array).

Darr currently supports numerical N-dimensional arrays, and experimentally supports numerical ragged arrays, i.e. a series of arrays in which one dimension varies in length.

See this tutorial for a brief introduction, or the documentation for more info.

Darr is currently pre-1.0, still undergoing significant development. However we have been using it in practice in our lab for more than a year on both Linux and Windows machines. It is open source and freely available under the New BSD License terms.



  • Data storage purely based on flat binary and text files, tool independence.

  • Human-readable explanation of how the binary data is stored is saved in a README text file.

  • README includes examples of how to read the particular array in popular analysis environments such as Python (without Darr), R, Julia, Octave/Matlab, GDL/IDL, and Mathematica.

  • Supports very large data arrays, larger than RAM, through memory-mapping.

  • Data read/write access is simple and powerful through NumPy indexing (see here).

  • Data is easily appendable.

  • Many numeric types are supported: (u)int8-(u)int64, float16-float64, complex64, complex128.

  • Easy use of metadata, stored in a separate JSON text file.

  • Minimal dependencies, only NumPy.

  • Integrates easily with the Dask library for numeric computation on very large arrays.

  • Supports ragged arrays (still experimental).

See the documentation for more information.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

darr-0.3.3.tar.gz (58.9 kB view hashes)

Uploaded source

Built Distribution

darr-0.3.3-py3-none-any.whl (45.9 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page