A replacement for the np.load(), np.save() and np.savez_compressed() NumPy functions with support for streaming data on one dimension
Project description
np-stream
This Python package provides a replacement for the np.load()
, np.save()
and np.savez_compressed()
NumPy functions with support for streaming data on one dimension.
For uncompressed arrays with a known size, use np.memmap()
instead.
Installation
$ pip install np-stream
Usage
Writing uncompressed data
The dimension which lists entries is added first for C-contiguous arrays (the default when creating arrays) and last otherwise. In this example, the array is C-contiguous, therefore its final shape is (100, 6, 2)
.
from np_stream import NpyStreamWriter
import numpy as np
with NpyStreamWriter(file) as writer:
for index in range(100):
writer.write(np.random.rand(6, 2))
Alternatively, the writer can be used without a context manager. The call to flush()
at the end is mandatory.
writer = NpyStreamWriter(file, shape=entry_shape)
for index in range(100):
writer.write(np.random.rand(6, 2))
writer.flush()
Writing compressed data
To produced compressed data, just replace NpyStreamWriter
with NpzStreamWriter
and create an .npz
instead of an .npy
file. The writer uses Deflate compression.
- from np_stream import NpyStreamWriter
+ from np_stream import NpzStreamWriter
entry_shape = (6, 2)
- with NpyStreamWriter(file, shape=entry_shape) as writer:
+ with NpzStreamWriter(file, shape=entry_shape) as writer:
for index in range(100):
writer.write(np.random.rand(*entry_shape))
Reading uncompressed data
reader = NpyStreamReader(file)
for entry in reader:
print(entry)
Reading compressed data
reader = NpzStreamReader(file)["arr_0"]
for entry in reader:
print(entry)
Similar packages
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for np_stream-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 983913e70e7935f366c1ed6f5168fa554e1d76692204a6a51097649995a0adab |
|
MD5 | b803b6ff7802430f95d619f7e9f6e484 |
|
BLAKE2b-256 | 2986d44f00fb26f8f26280c48cd4cb593bc5febb50faa8026c33e761f3fbc9f3 |