Read parquet data directly into numpy array
Project description
JollyJack
Features
- Reading parquet files directly into numpy arrays
- Compatible with PalletJack
Known limitations
- Works only with a single-precision floats
- Data cannot contain null values
- Only a local file system is supported
Required:
- pyarrow ~= 16.0
JollyJack operates on top of pyarrow, making it an essential requirement for both building and using JollyJack. While our source package is compatible with recent versions of pyarrow, the binary distribution package specifically requires the latest major version of pyarrow.
Installation
pip install jollyjack
How to use:
Generating a sample parquet file:
import jollyjack as jj
import pyarrow.parquet as pq
import pyarrow as pa
import numpy as np
chunk_size = 3
n_row_groups = 2
n_columns = 5
n_rows = n_row_groups * chunk_size
path = "my.parquet"
data = np.random.rand(n_rows, n_columns).astype(np.float32)
pa_arrays = [pa.array(data[:, i]) for i in range(n_columns)]
schema = pa.schema([(f'column_{i}', pa.float32()) for i in range(n_columns)])
table = pa.Table.from_arrays(pa_arrays, schema=schema)
pq.write_table(table, path, row_group_size=chunk_size, use_dictionary=False, write_statistics=True, store_schema=False, write_page_index=True)
Generating a numpy array to read into:
# Create an array of zeros
np_array = np.zeros((n_rows, n_columns), dtype='f', order='F')
Reading entire file into numpy array:
pr = pq.ParquetReader()
pr.open(path)
row_begin = 0
row_end = 0
for rg in range(pr.metadata.num_row_groups):
row_begin = row_end
row_end = row_begin + pr.metadata.row_group(rg).num_rows
# To define which subset of the numpy array we want read into,
# we need to create a view which shares underlying memory with the target numpy array
subset_view = np_array[row_begin:row_end, :]
jj.read_into_numpy_f32(metadata = pr.metadata
, parquet_path = path
, np_array = subset_view
, row_group_idx = rg
, column_indices = range(pr.metadata.num_columns))
print(np_array)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
jollyjack-0.3.0.tar.gz
(118.1 kB
view hashes)
Built Distributions
Close
Hashes for jollyjack-0.3.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3db363a9563f9c9415d5108226eaa83a67a585fd654ad8314f52fe210d85ce5e |
|
MD5 | 3182fe7d1e2c6e1220fa992f27c02007 |
|
BLAKE2b-256 | 0fda20caab15de7e4788643add9c2eee09339b8e3293fc8af615a982911d1c3c |
Close
Hashes for jollyjack-0.3.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 656d44349f0ff6c6d93edb844a8d1cf36e9e16900128a288ac27193b025d5103 |
|
MD5 | 081c3a6d1b8279294e5e2259ab59d418 |
|
BLAKE2b-256 | d8455b53aadd9779c7de7b0f3dab76192de0ac8db6fe9b818c90d21f3e998249 |
Close
Hashes for jollyjack-0.3.0-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | adf51139951d59d68b0c5bf9bbc65229c50da8857742cb10d43eea54105f2fa3 |
|
MD5 | e8d610e3665eef70891b6c76c75cd34f |
|
BLAKE2b-256 | d90573fe0f762cd57f3458c6d762b0dfb8e214e0a18b9b74de531f2e2c9cddda |
Close
Hashes for jollyjack-0.3.0-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e440b893fac15dd95252b8ea4bc0036ee60b4b041e01bee4da688a7a1e2229a |
|
MD5 | 77ce78f5f53b2bb00c7e33a8b1bc7280 |
|
BLAKE2b-256 | 760eed8237edf499cd4caf91283ec62d1239de73583d3f242d924313e6dd6631 |