Create Numpy NPY files that are larger than the main memory
Project description
NpyAppendArray
Create Numpy NPY files that are larger than the main memory of the machine by
appending on the zero axis. The file can then be read with mmap_mode="r"
.
Installation
pip install npy-append-array
Usage
from npy_append_array import NpyAppendArray
import numpy as np
arr1 = np.array([[1,2],[3,4]])
arr2 = np.array([[1,2],[3,4],[5,6]])
filename='out.npy'
# Appending to an array created by np.save is possible, but can fail in certain
# corner cases: e.g. a record as dtype, dim surpassing a critical threshold.
# Initialize the array with npaa.append directly (see below) so the
# header will be created with 64 byte of spare header space for growth.
# np.save(filename, arr1)
npaa = NpyAppendArray(filename)
npaa.append(arr1)
npaa.append(arr2)
npaa.append(arr2)
data = np.load(filename, mmap_mode="r")
print(data)
Limitations
- Only tested with Linux. For Windows, consider using WSL (version 2 or above).
- Exception thrown when Fortran order is used.
- NPY version 3 is unsupported because there is no
numpy.lib.format.read_array_header_3_0
function defined in https://numpy.org/devdocs/reference/generated/numpy.lib.format.html
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
npy-append-array-0.9.6.tar.gz
(3.2 kB
view hashes)
Built Distribution
Close
Hashes for npy_append_array-0.9.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 138ae43eb6cff8c2f6255572eb519764f772fd965e599497c973523a80cd0b75 |
|
MD5 | f9dd21feb4b5615de68b5bd3fd0c53ff |
|
BLAKE2b-256 | 0a86e0a6cce1ffcf02c7f19736fe97c136dc3c98202747a93f79326436d2fac2 |