Skip to main content

np-stream

This Python package provides a replacement for the np.load(), np.save() and np.savez_compressed() NumPy functions with support for streaming data on one dimension.

For uncompressed arrays with a known size, use np.memmap() instead.

Installation

$ pip install np-stream

Usage

Writing uncompressed data

The dimension which lists entries is added first for C-contiguous arrays (the default when creating arrays) and last otherwise. In this example, the array is C-contiguous, therefore its final shape is (100, 6, 2).

from np_stream import NpyStreamWriter
import numpy as np

with NpyStreamWriter(file) as writer:
  for index in range(100):
    writer.write(np.random.rand(6, 2))

Alternatively, the writer can be used without a context manager. The call to flush() at the end is mandatory.

writer = NpyStreamWriter(file, shape=entry_shape)

for index in range(100):
  writer.write(np.random.rand(6, 2))

writer.flush()

Writing compressed data

To produced compressed data, just replace NpyStreamWriter with NpzStreamWriter and create an .npz instead of an .npy file. The writer uses Deflate compression.

- from np_stream import NpyStreamWriter
+ from np_stream import NpzStreamWriter

  entry_shape = (6, 2)

- with NpyStreamWriter(file, shape=entry_shape) as writer:
+ with NpzStreamWriter(file, shape=entry_shape) as writer:
    for index in range(100):
      writer.write(np.random.rand(*entry_shape))

Reading uncompressed data

reader = NpyStreamReader(file)

for entry in reader:
  print(entry)

Reading compressed data

reader = NpzStreamReader(file)["arr_0"]

for entry in reader:
  print(entry)

Similar packages

Metadata

Release files for np-stream 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for np-stream 0.1.0
File Size Uploaded Provenance Actions
np-stream-0.1.0.tar.gz 8.0 kB Without provenance Details for np-stream-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for np-stream 0.1.0
File Interpreter ABI Platform Actions
np_stream-0.1.0-py3-none-any.whl Python 3 none any Details for np_stream-0.1.0-py3-none-any.whl

Total release size: 17.6 kB

Release files / np-stream-0.1.0.tar.gz

Download URL np-stream-0.1.0.tar.gz
Size 8.0 kB
Tags Source
SHA-256 checksum
How to use checksums
5f64d175877325a3ce3b51ebe976a828bfb0638590d64242a033f30f5086eaff
BLAKE2b-256 checksum
How to use checksums
7fd4542e445605da74edf85a0869a3b65dc4eaf351a192ae3fa7e53b32c127ed
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.2

Release files / np_stream-0.1.0-py3-none-any.whl

Download URL np_stream-0.1.0-py3-none-any.whl
Size 9.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
983913e70e7935f366c1ed6f5168fa554e1d76692204a6a51097649995a0adab
BLAKE2b-256 checksum
How to use checksums
2986d44f00fb26f8f26280c48cd4cb593bc5febb50faa8026c33e761f3fbc9f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.2

Security for np-stream 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page