File System Interface for CDF Files
Project description
cdffs
A file system interface (cdffs
) to allow users to work with CDF Files using the fsspec supported/compatible python packages (pandas
, xarray
etc).
fsspec
provides an abstract file system interface to work with local/cloud storages and based on the protocol name (example, s3
or abfs
) provided in the path, fsspec
translates the incoming requests to storage specific implementations and send the responses back to the upstream package to work with the desired data.
Refer fsspec documentation to get the list of all supported/compatible python packages.
Installation
cdffs
is available on PyPI. Install using,
pip install cognite-cdffs
Usage
Three important steps to follow when working with CDF Files using the fsspec supported python packages.
- Import
cdffs
package
from cognite import cdffs
- Create a client config to connect with CDF. Refer ClientConfig from Cognite Python SDK documentation on how to create a client config.
# Get TOKEN_URL, CLIENT_ID, CLIENT_SECRET, COGNITE_PROJECT,
# CDF_CLUSTER, SCOPES from environment variables.
oauth_creds = OAuthClientCredentials(
token_url=TOKEN_URL, client_id=CLIENT_ID, client_secret=CLIENT_SECRET, scopes=SCOPES
)
client_cnf = ClientConfig(
client_name="cdf-client",
base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
project=COGNITE_PROJECT,
credentials=oauth_creds
)
-
Pass the client config as
connection_config
instorage_options
when reading/writing the files.- Read
zarr
files using usingxarray
.
ds = xarray.open_zarr("cdffs://sample_data/test.zarr", storage_options={"connection_config": client_cnf})
- Write
zarr
files usingxarray
.
ds.to_zarr("cdffs://sample_data/test.zarr", storage_options={"connection_config": client_cnf, "file_metadata": metadata})
- Read
Refer cdffs.readthedocs.io for more details.
Contributing
Want to contribute? Check out CONTRIBUTING.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cognite_cdffs-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 96fbb05e9ce1c16d218a282b9adda0938fb0ba776acb07a3307c05b83cd8ca91 |
|
MD5 | 1ebe04c6719d4fa82ce879352a07040b |
|
BLAKE2b-256 | 38fb2d3464d51397b7a853e7af784ab926bc6631818d8d579a8ac09ba81acb6b |