kerchunk
Cloud-friendly access to archival data
Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files. It does this by extracting the byte ranges, compression information and other information about the data and storing this metadata in a new, separate object. This means that you can create a virtual aggregate dataset over potentially many source files, for efficient, parallel and cloud-friendly in-situ access without having to copy or translate the originals. It is a gateway to in-the-cloud massive data processing while the data providers still insist on using legacy formats for archival storage.
Why Kerchunk:
We provide the following things:
- completely serverless architecture
- metadata consolidation, so you can understand a many-file dataset (metadata plus physical storage) in a single read
- read from all of the storage backends supported by fsspec, including object storage (s3, gcs, abfs, alibaba), http, cloud user storage (dropbox, gdrive) and network protocols (ftp, ssh, hdfs, smb...)
- loading of various file types (currently netcdf4/HDF, grib2, tiff, fits, zarr), potentially heterogeneous within a single dataset, without a need to go via the specific driver (e.g., no need for h5py)
- asynchronous concurrent fetch of many data chunks in one go, amortizing the cost of latency
- parallel access with a library like zarr without any locks
- logical datasets viewing many (>~millions) data files, and direct access/subselection to them via coordinate indexing across an arbitrary number of dimensions
For further information, please see the documentation.
1 maintainer
Release files for kerchunk 0.2.10
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| kerchunk-0.2.10.tar.gz | 716.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| kerchunk-0.2.10-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 785.0 kB
Release files / kerchunk-0.2.10.tar.gz
| Download URL | kerchunk-0.2.10.tar.gz |
|---|---|
| Size | 716.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
aae63c0fe4ca2e97025f026578a0577545011fe6679751392c815e0d1d6bf954
|
|
BLAKE2b-256 checksum How to use checksums |
5ca0ebeca522912e68f360117404a6b2f740d4b0a343d98e9586377a2fd21567
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.11
|
Release files / kerchunk-0.2.10-py3-none-any.whl
| Download URL | kerchunk-0.2.10-py3-none-any.whl |
|---|---|
| Size | 68.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7fdaa77dae25c75d3ec9402c49208f37ae51d346ab082724e3e32608438f8c66
|
|
BLAKE2b-256 checksum How to use checksums |
64e43c356a9ea448a48caa5e44cd51293f7e896cd606f2ef86da96f5d61cc427
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.11
|
