A modern C++ header only cdf library
Project description
Python packages
Linux x86_64 | Windows x86_64 | MacOs x86_64 | MacOs ARM64 |
---|---|---|---|
Unit Tests
Linux x86_64 | Windows x86_64 | MacOs x86_64 |
---|---|---|
CDFpp (CDF++)
A NASA's CDF modern C++ library. This is not a C++ wrapper but a full C++ implementation. Why? CDF files are still used for space physics missions but few implementations are available. The main one is NASA's C implementation available here but it lacks multi-threads support (global shared state), has an old C interface and has a license which isn't compatible with most Linux distributions policy. There are also Java and Python implementations which are not usable in C++.
List of features and roadmap:
- CDF reading
- read uncompressed file headers
- read uncompressed attributes
- read uncompressed variables
- read variable attributes
- loads cdf files from memory (std::vector or char*)
- handles both row and column major files
- read variables with nested VXRs
- read compressed files (GZip, RLE)
- read compressed variables (GZip, RLE)
- read UTF-8 encoded files
- read ISO 8859-1(Latin-1) encoded files (converts to UTF-8 on the fly)
- variables values lazy loading
- decode DEC's floating point encoding (Itanium, ALPHA and VAX)
- pad values
- CDF writing
- write uncompressed headers
- write uncompressed attributes
- write uncompressed variables
- write compressed variables
- write compressed files
- pad values
- General features
- uses libdeflate for faster GZip decompression
- highly optimized CDF reads (up to ~4GB/s read speed from disk)
- handle leap seconds
- Python wrappers
- Documentation
- Examples (see below)
- Benchmarks
If you want to understand how it works, how to use the code or what works, you may have to read tests.
Installing
From PyPi
python3 -m pip install --user pycdfpp
From sources
meson build
cd build
ninja
sudo ninja install
Or if youl want to build a Python wheel:
python -m build .
# resulting wheel will be located into dist folder
Basic usage
Python
Reading CDF files
Basic example from a local file:
import pycdfpp
cdf = pycdfpp.load("some_cdf.cdf")
cdf_var_data = cdf["var_name"].values #builds a numpy view or a list of strings
attribute_name_first_value = cdf.attributes['attribute_name'][0]
Note that you can also load in memory files:
import pycdfpp
import requests
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
plt.plot(tha_l2_fgm["tha_fgl_gsm"])
plt.show()
Buffer protocol support:
import pycdfpp
import requests
import xarray as xr
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
xr.DataArray(tha_l2_fgm['tha_fgl_gsm'], dims=['time', 'components'], coords={'time':tha_l2_fgm['tha_fgl_time'].values, 'components':['x', 'y', 'z']}).plot.line(x='time')
plt.show()
# Works with matplotlib directly too
plt.plot(tha_l2_fgm['tha_fgl_time'], tha_l2_fgm['tha_fgl_gsm'])
plt.show()
Datetimes handling:
import pycdfpp
import os
# Due to an issue with pybind11 you have to force your timezone to UTC for
# datetime conversion (not necessary for numpy datetime64)
os.environ['TZ']='UTC'
mms2_fgm_srvy = pycdfpp.load("mms2_fgm_srvy_l2_20200201_v5.230.0.cdf")
# to convert any CDF variable holding any time type to python datetime:
epoch_dt = pycdfpp.to_datetime(mms2_fgm_srvy["Epoch"])
# same with numpy datetime64:
epoch_dt64 = pycdfpp.to_datetime64(mms2_fgm_srvy["Epoch"])
# note that using datetime64 is ~100x faster than datetime (~2ns/element on an average laptop)
Writing CDF files
Creating a basic CDF file:
import pycdfpp
import numpy as np
from datetime import datetime
cdf = pycdfpp.CDF()
cdf.add_attribute("some attribute", [[1,2,3], [datetime(2018,1,1), datetime(2018,1,2)], "hello\nworld"])
cdf.add_variable(f"some variable", values=np.ones((10),dtype=np.float64))
pycdfpp.save(cdf, "some_cdf.cdf")
C++
#include "cdf-io/cdf-io.hpp"
#include <iostream>
std::ostream& operator<<(std::ostream& os, const cdf::Variable::shape_t& shape)
{
os << "(";
for (auto i = 0; i < static_cast<int>(std::size(shape)) - 1; i++)
os << shape[i] << ',';
if (std::size(shape) >= 1)
os << shape[std::size(shape) - 1];
os << ")";
return os;
}
int main(int argc, char** argv)
{
auto path = std::string(DATA_PATH) + "/a_cdf.cdf";
// cdf::io::load returns a optional<CDF>
if (const auto my_cdf = cdf::io::load(path); my_cdf)
{
std::cout << "Attribute list:" << std::endl;
for (const auto& [name, attribute] : my_cdf->attributes)
{
std::cout << "\t" << name << std::endl;
}
std::cout << "Variable list:" << std::endl;
for (const auto& [name, variable] : my_cdf->variables)
{
std::cout << "\t" << name << " shape:" << variable.shape() << std::endl;
}
return 0;
}
return -1;
}
caveats
- NRV variables shape, in order to expose a consistent shape, PyCDFpp exposes the reccord count as first dimension and thus its value will be either 0 or 1 (0 mean empty variable).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pycdfpp-0.6.1-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bbab0be11dbd79954a3eac7bfb9265d63507f0ee844047473a26d77756fa5916 |
|
MD5 | c797b9a0270efad422f51c3f91e439c7 |
|
BLAKE2b-256 | 45f1ea8a9f0af456131f62af97fe959cedb540f97dbef5402cd5e6daa7efe739 |
Hashes for pycdfpp-0.6.1-cp312-cp312-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d18c7c15efbe10195bd25feda04a1192faabd4b43ae6e6fbac043788d728185 |
|
MD5 | 39b6dfff76a1c0e2035942219851f0cf |
|
BLAKE2b-256 | a4c76aa880567d8c995c6c69797113bccb72ce7554bbf4fd45123ae31807daf2 |
Hashes for pycdfpp-0.6.1-cp312-cp312-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2354039c695d90cd0519572d1f72f6d2fe29fa6bbab806a0a9d4e058f69b3c67 |
|
MD5 | 419fb8441843a4b24e08d47864a1c04c |
|
BLAKE2b-256 | ab63d0c056a68cd11de189ffdcae5bf68827857efcb1a00d31d5929f95f1116d |
Hashes for pycdfpp-0.6.1-cp312-cp312-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ec93c1c9e68669bf149f09278b0138a800fefbc69e581c4b3c7aeb59b203bf3 |
|
MD5 | 8e5f17a94de5ede92f9772ee27edd51a |
|
BLAKE2b-256 | fa9e6b4b800fa0c3d17a2d4941d44e45a958dbcdca8a0cf4e1c3e59742e51a93 |
Hashes for pycdfpp-0.6.1-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 160f693833c3b0846403dd5ad88f3f2ecb52a38f9df9a8d03723413b005d6f5e |
|
MD5 | 007568c0d402d872d9e0f7ccb9fdfb21 |
|
BLAKE2b-256 | 56af4db4b195fe4d1738d4dc07a99153863cf5ba39549a64ac62fa9291402e8d |
Hashes for pycdfpp-0.6.1-cp311-cp311-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76d872598b64ab9fbdceadd9809bbc7e67413dcbf5945617adcee02f1e2b6e85 |
|
MD5 | 5e0967136f1f7836d98683e3920b1287 |
|
BLAKE2b-256 | ace46a3c3a5a207e6fc86c23c5626daf3cf0bf5e6ac1ec178fefc3907327bdfb |
Hashes for pycdfpp-0.6.1-cp311-cp311-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 70d04c753c51fcc8ed5f75b3b1ff3934676e4483ee624021b9a5f77ad2deec23 |
|
MD5 | 04af728e30e321ad0a63f45a90319beb |
|
BLAKE2b-256 | aa35e5836db1de5b2803b7be59a824d0a5d4e3409aa24006799ad10c3ff99448 |
Hashes for pycdfpp-0.6.1-cp311-cp311-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c13b637e2520904f80777799f08f07c3ca1e461c0ac8d629b65976739228dc0 |
|
MD5 | cd34294525748fffc65244d4a093495c |
|
BLAKE2b-256 | 43c49dba9428a65e780aa397ed65f8cdc36e57089a30a7b65202bb09b068d02a |
Hashes for pycdfpp-0.6.1-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e67c4c316a87b105e18301605202acc4b64678d0f76281c4542be2ba79a30fe |
|
MD5 | 26279df5857a186a36a3fb0d3693c3eb |
|
BLAKE2b-256 | 5159d48225e9d78c72fbfe9f80a95c43ddade767856e80cc4f4c4a696ef157b5 |
Hashes for pycdfpp-0.6.1-cp310-cp310-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b02c8f1e5e52dc70f76fcb1d4723f251cbc99f3f764347b84b5a86b83131be0e |
|
MD5 | 871aeeabb8d19ef22ce18127defbeaa2 |
|
BLAKE2b-256 | 784489172b7f83fc3a34f56130e269ff528a5f737fc3ef82050a1c7866313d7b |
Hashes for pycdfpp-0.6.1-cp310-cp310-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9210fa23d95e5f6bd018efcf286b6af4e78b9018417ca53079f5c4ec802ef989 |
|
MD5 | 0efae9dfb0d8f83be630e540f9c22793 |
|
BLAKE2b-256 | a6c80174f4466a23432f8a20194ca451fe448cfd7702c95442af25ca5101920b |
Hashes for pycdfpp-0.6.1-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b87d4fdb53e3e5a11bbd183fb9e4b87dc224bfa9f31adc0aca7b53502f7f34f7 |
|
MD5 | 1f3ca71e52999575a83e4326ed7d9700 |
|
BLAKE2b-256 | 1818135f96e4335580ab7547b1c00299495355b2759092df98397b9c23ded0f8 |
Hashes for pycdfpp-0.6.1-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21b894e5ee9f322c43523ef83224be92047be2d5862d727dbdf834c41a427e2a |
|
MD5 | 39e5b9436eb1005c490319a2fe4292d5 |
|
BLAKE2b-256 | 60535b4ed7c2a7dd977988a356acb64b3669f09e6372e974112c404b67083062 |
Hashes for pycdfpp-0.6.1-cp39-cp39-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c9134a90e9c1dc85c85a40f3f6db3c663915eeaf38e10175310e298618f10ac |
|
MD5 | fe03ac1cb1beb39d258cdd2cdac8fef6 |
|
BLAKE2b-256 | 92fd71c6f08c1b1f76ebcc9e40a777ea0ccebf47a281386c2375cad00e15b302 |
Hashes for pycdfpp-0.6.1-cp39-cp39-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c8d4d4d4eccb9cd252b5959eb55af56c3500b31d9e1414443f90704e33eb0d30 |
|
MD5 | 82da67ed94d1ebb9a974ec58f4d0ffbb |
|
BLAKE2b-256 | 589161ac19ebd4cf28c637feb0ceb5a3cf7f72ba048fff9af9ca3c9d2d191da2 |
Hashes for pycdfpp-0.6.1-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87ede886367ee24d1f7fcc5d244958b6836079faccb7d5debba41344420ba421 |
|
MD5 | 8b31c59d349e8b39791cf8c8973992a7 |
|
BLAKE2b-256 | b46e06e15d80b5b37faede52ffc9c474f004ba84ae9f4430e262e3356d0356a5 |
Hashes for pycdfpp-0.6.1-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66f18b3bca46c94a7b21fc666607740ed7275c912b046787dfc02028735eeadb |
|
MD5 | 9696a7c026d46fecac0680df2dbea22c |
|
BLAKE2b-256 | 41efceb56d9c6814722e99e1398546d20ae977440a8b5f8f622b6c0b157d97fc |
Hashes for pycdfpp-0.6.1-cp38-cp38-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d33d2339780e87ed25f221f04b51e6483e2ea856b525a7e3ac62aca3f18876dc |
|
MD5 | a66b0ddd9b830d43b89d43931b001288 |
|
BLAKE2b-256 | 5ab7507954584615c812a0904451c433facd4c96812bfaf6921eac6e506ee023 |
Hashes for pycdfpp-0.6.1-cp38-cp38-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a53628f8e914b4de7716e717d169deae12fc2f5915007594ada002feedadc76b |
|
MD5 | c34e91b99901a26f66f9d5edbb991d1c |
|
BLAKE2b-256 | 7302bef7f9e35769bd1eb640406fbd4d0450ba49b0ab3b2f7ddbee0125941a11 |
Hashes for pycdfpp-0.6.1-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 91414c60eefef17a09384e051fafabae1a4bb410c583efaade8e269bd95fc6b1 |
|
MD5 | 3afd62c5b3b37c0dd3b9054d6d63e60d |
|
BLAKE2b-256 | 165aa33b6c0b16f20016ef126c39e088ca3f544f822302752fd30155e11a161f |