A modern C++ header only cdf library
Project description
Python packages
Linux x86_64 | Windows x86_64 | MacOs x86_64 | MacOs ARM64 |
---|---|---|---|
Unit Tests
Linux x86_64 | Windows x86_64 | MacOs x86_64 |
---|---|---|
CDFpp (CDF++)
A NASA's CDF modern C++ library. This is not a C++ wrapper but a full C++ implementation. Why? CDF files are still used for space physics missions but few implementations are available. The main one is NASA's C implementation available here but it lacks multi-threads support (global shared state), has an old C interface and has a license which isn't compatible with most Linux distributions policy. There are also Java and Python implementations which are not usable in C++.
List of features and roadmap:
- CDF reading
- read uncompressed file headers
- read uncompressed attributes
- read uncompressed variables
- read variable attributes
- loads cdf files from memory (std::vector or char*)
- handles both row and column major files
- read variables with nested VXRs
- read compressed files (GZip, RLE)
- read compressed variables (GZip, RLE)
- read UTF-8 encoded files
- read ISO 8859-1(Latin-1) encoded files (converts to UTF-8 on the fly)
- variables values lazy loading
- decode DEC's floating point encoding (Itanium, ALPHA and VAX)
- CDF writing
- write uncompressed headers
- write uncompressed attributes
- write uncompressed variables
- write compressed variables
- write compressed files
- General features
- uses libdeflate for faster GZip decompression
- highly optimized CDF reads (up to ~4GB/s read speed from disk)
- handle leap seconds
- Python wrappers
- Documentation
- Examples (see below)
- Benchmarks
If you want to understand how it works, how to use the code or what works, you may have to read tests.
Installing
From PyPi
pip3 install --user pycdfpp
From sources
meson build
cd build
ninja
sudo ninja install
Basic usage
Python
Reading CDF files
Basic example from a local file:
import pycdfpp
cdf = pycdfpp.load("some_cdf.cdf")
cdf_var_data = cdf["var_name"].values #builds a numpy view or a list of strings
attribute_name_first_value = cdf.attributes['attribute_name'][0]
Note that you can also load in memory files:
import pycdfpp
import requests
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
plt.plot(tha_l2_fgm["tha_fgl_gsm"])
plt.show()
Buffer protocol support:
import pycdfpp
import requests
import xarray as xr
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
xr.DataArray(tha_l2_fgm['tha_fgl_gsm'], dims=['time', 'components'], coords={'time':tha_l2_fgm['tha_fgl_time'].values, 'components':['x', 'y', 'z']}).plot.line(x='time')
plt.show()
# Works with matplotlib directly too
plt.plot(tha_l2_fgm['tha_fgl_time'], tha_l2_fgm['tha_fgl_gsm'])
plt.show()
Datetimes handling:
import pycdfpp
import os
# Due to an issue with pybind11 you have to force your timezone to UTC for
# datetime conversion (not necessary for numpy datetime64)
os.environ['TZ']='UTC'
mms2_fgm_srvy = pycdfpp.load("mms2_fgm_srvy_l2_20200201_v5.230.0.cdf")
# to convert any CDF variable holding any time type to python datetime:
epoch_dt = pycdfpp.to_datetime(mms2_fgm_srvy["Epoch"])
# same with numpy datetime64:
epoch_dt64 = pycdfpp.to_datetime64(mms2_fgm_srvy["Epoch"])
# note that using datetime64 is ~100x faster than datetime (~2ns/element on an average laptop)
Writing CDF files
Creating a basic CDF file:
import pycdfpp
import numpy as np
from datetime import datetime
cdf = pycdfpp.CDF()
cdf.add_attribute("some attribute", [[1,2,3], [datetime(2018,1,1), datetime(2018,1,2)], "hello\nworld"])
cdf.add_variable(f"some variable", values=np.ones((10),dtype=np.float64))
pycdfpp.save(cdf, "some_cdf.cdf")
C++
#include "cdf-io/cdf-io.hpp"
#include <iostream>
std::ostream& operator<<(std::ostream& os, const cdf::Variable::shape_t& shape)
{
os << "(";
for (auto i = 0; i < static_cast<int>(std::size(shape)) - 1; i++)
os << shape[i] << ',';
if (std::size(shape) >= 1)
os << shape[std::size(shape) - 1];
os << ")";
return os;
}
int main(int argc, char** argv)
{
auto path = std::string(DATA_PATH) + "/a_cdf.cdf";
// cdf::io::load returns a optional<CDF>
if (const auto my_cdf = cdf::io::load(path); my_cdf)
{
std::cout << "Attribute list:" << std::endl;
for (const auto& [name, attribute] : my_cdf->attributes)
{
std::cout << "\t" << name << std::endl;
}
std::cout << "Variable list:" << std::endl;
for (const auto& [name, variable] : my_cdf->variables)
{
std::cout << "\t" << name << " shape:" << variable.shape() << std::endl;
}
return 0;
}
return -1;
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pycdfpp-0.5.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a63db14ccf66a7b40fd40f4d1016b11a0f389b320eb5a703576873c178e1a45 |
|
MD5 | fe7ffe6e92be3cf27b676caa0f1ef1bf |
|
BLAKE2b-256 | 09d99b853bc56171187fd1f49a2770569f7538bb37b462615c5e2781e998c386 |
Hashes for pycdfpp-0.5.0-cp311-cp311-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9bee818ac71c12e043e309221e774625f8fc9e306b1f46418102d6d05de07f2c |
|
MD5 | dcced4280969ac49845ebd5182e8ea30 |
|
BLAKE2b-256 | b64053ba6eef9964db80d2cc91c5a9502f62d89bd4585fca37acb0e7aef3d5d8 |
Hashes for pycdfpp-0.5.0-cp311-cp311-macosx_12_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe523cc1a5cc9ada2f187d4cd0d8ab9c9c88f506b6c4af3c93b167847858e39d |
|
MD5 | c6c8093cdfce95b04a7d78af34bb435a |
|
BLAKE2b-256 | 1d94b60cbbaa96aff91c36f627bbc43ea49042ce1f2013e0dabdd53fd40d23b5 |
Hashes for pycdfpp-0.5.0-cp311-cp311-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a41d10583b3e321b16dcc8831c03ef9bd81810b4c1dab4cc17a1735a5887cbf |
|
MD5 | 19e03fb30d52ff893035b949355b280b |
|
BLAKE2b-256 | 91eef75ea9eb7ebef4f41bfbde46169202e30eef22052638ed11f80247733ca7 |
Hashes for pycdfpp-0.5.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e9468c66572a7684ff3319afd7332ecb5751abf508371defdbb6e3f561af5ac1 |
|
MD5 | 41346aec4f90205573109e11c46ef821 |
|
BLAKE2b-256 | 0af2fc6f9bc149daa4f7346ee16486c997a75722308c90b2665c26d4bf2781e0 |
Hashes for pycdfpp-0.5.0-cp310-cp310-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41acec11001eb4f183913abe395499d97fd997de8f1c636067249168d713581d |
|
MD5 | 57d23752de820706c4e199cbc01c2f68 |
|
BLAKE2b-256 | 7a6d4e63bf363cdd90b1fe734e5f66925f6a8e05907af1753e1645d4a164504e |
Hashes for pycdfpp-0.5.0-cp310-cp310-macosx_12_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c850e3f16d52d347be02b11aa9bd7c2999faff2be20019ed5769bcc0f712d8ef |
|
MD5 | 66b34983f866c45a31ef654ab84b8396 |
|
BLAKE2b-256 | e72b66fe98e239bdd9f4ee77e89b174610c0a0b566a3043e4c4e57ff3b6a540e |
Hashes for pycdfpp-0.5.0-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4017064f456c2e1f39c21ee13f6069de669273efe731fa51144bf1810b0869e6 |
|
MD5 | 34288af667c5da7fb3352f3b65ef0d0b |
|
BLAKE2b-256 | b7ab1f06a5020f94c7e7ea0e5af7e234ed95d0eab0d3547ef22b1c8e7af93afc |
Hashes for pycdfpp-0.5.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da013096f1e0aebfd8bc2f31580300f37a78a9fae2eef51993d7782e5e736d91 |
|
MD5 | 174857578fb4a18c13a26b14408071e4 |
|
BLAKE2b-256 | 6c0a4da1b8831fb9a43824e3589043798cf9e2742ef12b1e95738f8c427b2e97 |
Hashes for pycdfpp-0.5.0-cp39-cp39-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2fe6c56d69aca79d34a303dfcd4367b70a027bb05189a5aef0e056576e9d0950 |
|
MD5 | fca9259c236263aaed609939c2c4acf5 |
|
BLAKE2b-256 | 2f5f4a79e01f72bffe4bbfa815c3460a331fed7796dd8bcd07003ec34b1fd153 |
Hashes for pycdfpp-0.5.0-cp39-cp39-macosx_12_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26fe3ab6936b40a44d42a80dd8573c99e37525c0938be6783ec4e6be6faa8094 |
|
MD5 | 890f04e87a2dacffed9c1c2877bf6212 |
|
BLAKE2b-256 | 3c344de017a5b5b134a6e024570a32d03a7279b4ef105755ffa6e4acf9c1fdf5 |
Hashes for pycdfpp-0.5.0-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c442a528baa383ee6495336f2db21e5b8bb2480215cfd5a8f458c78386069db4 |
|
MD5 | 9e7f8cf183e35d107fd8f60e93136278 |
|
BLAKE2b-256 | bcdb536f7c5e2fc9281dfd1703716d14104abb525d2716bcb593b0f7a5822235 |
Hashes for pycdfpp-0.5.0-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b5b77dcc7de1bc245d819e66855dbb9677caed6e4ee1cbd6f57e54cd54d93d4c |
|
MD5 | 2e315a87506cb5c0d3d2e449094360b8 |
|
BLAKE2b-256 | c78a4299ca6254efb629379d54ecafd661da51c81431e01cb230437d9cc70f8a |
Hashes for pycdfpp-0.5.0-cp38-cp38-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43bd54848088fdbfa2a9619d7e29890bb32d95c164340ca9102d5b29201d67d1 |
|
MD5 | a7ea0564a92bf48c3e0aa000ba6fe93f |
|
BLAKE2b-256 | b55383a8abb6862b522ff3f964dbc723a0cef2cc6604cf74dd761de973f4404e |
Hashes for pycdfpp-0.5.0-cp38-cp38-macosx_12_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a82ace0e7235158585fdca2512c60abc60489bca2d3da55282d43d37df78f9f8 |
|
MD5 | 659a9739c565cdb829c6f1981936af71 |
|
BLAKE2b-256 | 1ce4b1c4fad35ef645cdcfaa4e0c430800840c8cada510661577eeeb79f63393 |
Hashes for pycdfpp-0.5.0-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d32d5072b290996eeb64996ee76879728637fb15cd1bab9786738a7b266d60b8 |
|
MD5 | 4a91275121f0da7809844ef0db87ba2e |
|
BLAKE2b-256 | 6e884752183a0c4134d746a475ab0a4dbcac9a2dc822c7b08271d1e071a693d4 |