Skip to main content

pyg-npy

pip install from https://pypi.org/project/pyg-npy/

conda install from https://anaconda.org/yoavgit/pyg-npy

A quick utility to save dataframes as npy files.

It supports append and checks lightly on column names matching and index.

For simple read/write/append, it is about 5-10 times faster than parquet writing or pystore.

import numpy as np; import pandas as pd
from pyg_npy import pd_to_npy, pd_read_npy
import pystore
import datetime

pystore.set_path("c:/temp/pystore")
store = pystore.store('mydatastore')
collection = store.collection('NASDAQ')
arr = np.random.normal(0,1,(100,10))
df = pd.DataFrame(arr, columns = list('abcdefghij'))
dates = [datetime.datetime(2020,1,1) + datetime.timedelta(i) for i in range(-10000,0)]
ts = pd.DataFrame(np.random.normal(0,1,(10000,10)), dates, columns = list('abcdefghij'))

### write
%timeit collection.write('TEST', df, overwrite = True)
19.5 ms ± 1.97 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit df.to_parquet('c:/temp/test.parquet')
9.53 ms ± 650 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit pd_to_npy(df, 'c:/temp/test.npy')
947 µs ± 38.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)


### read
%timeit collection.item('TEST').to_pandas()
7.7 ms ± 171 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit pd.read_parquet('c:/temp/test.parquet')
2.85 ms ± 54.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit pd_read_npy('c:/temp/test.npy')
847 µs ± 39.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

### append
# because we need to append data with increasing index value, we use this:

collection.write('TEST2', ts.iloc[:100], overwrite = True)
%time len([collection.append('TEST2', ts.iloc[i*100: i*100+100], npartitions=2) for i in range(1,100)])
Wall time: 12.1 s

pd_to_npy(ts.iloc[:100], 'c:/temp/test.npy')
%time len([pd_to_npy(ts.iloc[i*100: i*100+100], 'c:/temp/test.npy', 'a', True) for i in range(1,100)])
Wall time: 2.14 s

Metadata

Release files for pyg-npy 0.0.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyg-npy 0.0.9
File Size Uploaded Provenance Actions
pyg-npy-0.0.9.tar.gz 8.2 kB Without provenance Details for pyg-npy-0.0.9.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pyg-npy 0.0.9
File Interpreter ABI Platform Actions
pyg_npy-0.0.9-py3-none-any.whl Python 3 none any Details for pyg_npy-0.0.9-py3-none-any.whl

Total release size: 16.1 kB

Release files / pyg-npy-0.0.9.tar.gz

Download URL pyg-npy-0.0.9.tar.gz
Size 8.2 kB
Tags Source
SHA-256 checksum
How to use checksums
1b71643e78f291be92d0e7c52584f05172881fec9846739bc8e10aca4c115ff9
BLAKE2b-256 checksum
How to use checksums
67656c67e08b27a15c6d54cd352f316994d8a21aa1025b6f5a901fab8647f664
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.9.13

Release files / pyg_npy-0.0.9-py3-none-any.whl

Download URL pyg_npy-0.0.9-py3-none-any.whl
Size 7.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7d6f3944b8f3adb9a60499d6d8bdd5fa38ad054d884c5e86a1e066a1be4438d2
BLAKE2b-256 checksum
How to use checksums
503e0f5739e600c9b0968cc210ed63ba6aaf782055eb3cac5709671da87221cb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.9.13

Security for pyg-npy 0.0.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page