A modern C++ header only cdf library
Project description
Python packages
Linux x86_64 | Windows x86_64 | MacOs x86_64 | MacOs ARM64 |
---|---|---|---|
Unit Tests
Linux x86_64 | Windows x86_64 | MacOs x86_64 |
---|---|---|
CDFpp (CDF++)
A NASA's CDF modern C++ library. This is not a C++ wrapper but a full C++ implementation. Why? CDF files are still used for space physics missions but few implementations are available. The main one is NASA's C implementation available here but it lacks multi-threads support (global shared state), has an old C interface and has a license which isn't compatible with most Linux distributions policy. There are also Java and Python implementations which are not usable in C++.
List of features and roadmap:
- CDF reading
- read uncompressed file headers
- read uncompressed attributes
- read uncompressed variables
- read variable attributes
- loads cdf files from memory (std::vector or char*)
- handles both row and column major files
- read variables with nested VXRs
- read compressed files (GZip, RLE)
- read compressed variables (GZip, RLE)
- read UTF-8 encoded files
- read ISO 8859-1(Latin-1) encoded files (converts to UTF-8 on the fly)
- variables values lazy loading
- decode DEC's floating point encoding (Itanium, ALPHA and VAX)
- pad values
- CDF writing
- write uncompressed headers
- write uncompressed attributes
- write uncompressed variables
- write compressed variables
- write compressed files
- pad values
- General features
- uses libdeflate for faster GZip decompression
- highly optimized CDF reads (up to ~4GB/s read speed from disk)
- handle leap seconds
- Python wrappers
- Documentation
- Examples (see below)
- Benchmarks
If you want to understand how it works, how to use the code or what works, you may have to read tests.
Installing
From PyPi
python3 -m pip install --user pycdfpp
From sources
meson build
cd build
ninja
sudo ninja install
Or if youl want to build a Python wheel:
python -m build .
# resulting wheel will be located into dist folder
Basic usage
Python
Reading CDF files
Basic example from a local file:
import pycdfpp
cdf = pycdfpp.load("some_cdf.cdf")
cdf_var_data = cdf["var_name"].values #builds a numpy view or a list of strings
attribute_name_first_value = cdf.attributes['attribute_name'][0]
Note that you can also load in memory files:
import pycdfpp
import requests
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
plt.plot(tha_l2_fgm["tha_fgl_gsm"])
plt.show()
Buffer protocol support:
import pycdfpp
import requests
import xarray as xr
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
xr.DataArray(tha_l2_fgm['tha_fgl_gsm'], dims=['time', 'components'], coords={'time':tha_l2_fgm['tha_fgl_time'].values, 'components':['x', 'y', 'z']}).plot.line(x='time')
plt.show()
# Works with matplotlib directly too
plt.plot(tha_l2_fgm['tha_fgl_time'], tha_l2_fgm['tha_fgl_gsm'])
plt.show()
Datetimes handling:
import pycdfpp
import os
# Due to an issue with pybind11 you have to force your timezone to UTC for
# datetime conversion (not necessary for numpy datetime64)
os.environ['TZ']='UTC'
mms2_fgm_srvy = pycdfpp.load("mms2_fgm_srvy_l2_20200201_v5.230.0.cdf")
# to convert any CDF variable holding any time type to python datetime:
epoch_dt = pycdfpp.to_datetime(mms2_fgm_srvy["Epoch"])
# same with numpy datetime64:
epoch_dt64 = pycdfpp.to_datetime64(mms2_fgm_srvy["Epoch"])
# note that using datetime64 is ~100x faster than datetime (~2ns/element on an average laptop)
Writing CDF files
Creating a basic CDF file:
import pycdfpp
import numpy as np
from datetime import datetime
cdf = pycdfpp.CDF()
cdf.add_attribute("some attribute", [[1,2,3], [datetime(2018,1,1), datetime(2018,1,2)], "hello\nworld"])
cdf.add_variable(f"some variable", values=np.ones((10),dtype=np.float64))
pycdfpp.save(cdf, "some_cdf.cdf")
C++
#include "cdf-io/cdf-io.hpp"
#include <iostream>
std::ostream& operator<<(std::ostream& os, const cdf::Variable::shape_t& shape)
{
os << "(";
for (auto i = 0; i < static_cast<int>(std::size(shape)) - 1; i++)
os << shape[i] << ',';
if (std::size(shape) >= 1)
os << shape[std::size(shape) - 1];
os << ")";
return os;
}
int main(int argc, char** argv)
{
auto path = std::string(DATA_PATH) + "/a_cdf.cdf";
// cdf::io::load returns a optional<CDF>
if (const auto my_cdf = cdf::io::load(path); my_cdf)
{
std::cout << "Attribute list:" << std::endl;
for (const auto& [name, attribute] : my_cdf->attributes)
{
std::cout << "\t" << name << std::endl;
}
std::cout << "Variable list:" << std::endl;
for (const auto& [name, variable] : my_cdf->variables)
{
std::cout << "\t" << name << " shape:" << variable.shape() << std::endl;
}
return 0;
}
return -1;
}
caveats
- NRV variables shape, in order to expose a consistent shape, PyCDFpp exposes the reccord count as first dimension and thus its value will be either 0 or 1 (0 mean empty variable).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pycdfpp-0.6.3-pp310-pypy310_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c88eaccd48d82afe54a44777063711a9aaa147e92d67802860a67d14b5adc38 |
|
MD5 | 9a51c320ad1f616c6bfb8cbbca475c48 |
|
BLAKE2b-256 | b8ef82859323b880c1b461858fb1b354cb4e41e9351fa8fdd2b6314f0d41d76e |
Hashes for pycdfpp-0.6.3-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eca8538c48ecb5411c9cc48eb663edc86e5fb933441e06fb7c9ab2d4920e1529 |
|
MD5 | 37413b728b66a024d49db8e4798ec5e6 |
|
BLAKE2b-256 | 38d46c61371e9e4d65d6720e62318672f12753a6b02d6618d18bae4886238b1c |
Hashes for pycdfpp-0.6.3-pp310-pypy310_pp73-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 417d4831ca4311f74c2ba6a5ca30cd4aba8be9fdca663290e9df72c41ed75f0b |
|
MD5 | e1537a8e52d66ea4093830cad35d6f44 |
|
BLAKE2b-256 | 13b6b5b9ccf218fd54da473087a10577626627e374f42188f788fa8537ed41b2 |
Hashes for pycdfpp-0.6.3-pp310-pypy310_pp73-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da288fdd1fff79307039440228f535a60b679bd314902dfac62228ca0e3c1939 |
|
MD5 | 3c8d4b26bdc381ae6ffea8b865c8fb01 |
|
BLAKE2b-256 | f0861092730971fe2a8182226dd821be83ded632effe486c7bbae2e076504542 |
Hashes for pycdfpp-0.6.3-pp39-pypy39_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f558945d17d0048360ef360a475e26982466455a30f32c284ba0b2bf4299ee3 |
|
MD5 | 1d66056526e608c18b756dc79ebc00da |
|
BLAKE2b-256 | 0e07659d85108efa272456850decd82b96ee1d2c32c2d36ad3d4f13383c20036 |
Hashes for pycdfpp-0.6.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba8dbfb70953b69ea06565e44205784b58e80fc752639a74ffae0acecf41e7d4 |
|
MD5 | 35673b2af4a3a72f641cc139093ffdbb |
|
BLAKE2b-256 | 24541e71a25bc51516e647abfabef4d5428f6832c5bf6bbfb788551c2e85906b |
Hashes for pycdfpp-0.6.3-pp39-pypy39_pp73-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c559ea44ae7a2306b1fd57054f14b1d49419f3d7a4e0416e6c3de7f982eb2f72 |
|
MD5 | 7c443ba0209e3755b6499845a3005559 |
|
BLAKE2b-256 | 818e0310373ea25829dd33442a4e66096190d52bcd7c8f36cf6c565a1cdc285b |
Hashes for pycdfpp-0.6.3-pp39-pypy39_pp73-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84404f0443de522267dbf905a08eee0d4f45a1b6c276ce853364c42598a0944b |
|
MD5 | f4d88a5905da361ed517bee048575260 |
|
BLAKE2b-256 | 9a6ade46fe67da236b3b6d04ed7e8739aab575f8597ccf308c29794f4437861e |
Hashes for pycdfpp-0.6.3-pp38-pypy38_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3703d46f0d24e386256583c5074a1c86495726101095b007ea061dd3d983bf5c |
|
MD5 | 0943322d1b25bc2b634f4840018dbc98 |
|
BLAKE2b-256 | 1e05af505b072b5578bcd72697007275c3c771522e0efb0782d294ad14faaa9d |
Hashes for pycdfpp-0.6.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 967e00ef15446e3385260955e8ee3b9b2e5a7ebf97b36e8ed215d69f23e3a260 |
|
MD5 | 3185962628b07fcc125312154026495e |
|
BLAKE2b-256 | cc9f59e58e386e6dcc98bf069945099e369eb301c5b1bb9cc758370b9e7c9d84 |
Hashes for pycdfpp-0.6.3-pp38-pypy38_pp73-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ac65d78dea73f3836590166db73f7ad86fe1f7b78cde16e1d5dba8caa2174f5 |
|
MD5 | 011603fc663fb21bef4e2c4abc02f3de |
|
BLAKE2b-256 | 580386ddb66c700ee5a942c9af9ece325ebcb4c0ab7f56943a2b42ad006ab3c5 |
Hashes for pycdfpp-0.6.3-pp38-pypy38_pp73-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8cd5327cd19da12efed5c3d0c233284aa5dfa6f1a2e7da9cc87b42eadd311b5 |
|
MD5 | b96cd3b928c4077e43dce85d14e1dbe2 |
|
BLAKE2b-256 | d60b544967b6220c615c7e7ee9c94d093d7ede10b72d2619ec4beadfa6883d8b |
Hashes for pycdfpp-0.6.3-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c53bda5743148a7cca4b3e84d36fb7db0c12a5feab3b567513a95369fa91d78c |
|
MD5 | a8a28f4259eaac26feece96576395dbc |
|
BLAKE2b-256 | fdf5e0c8a5c3d5cc1665b342a394fb5a4db9233fb7942f980f0c7d053743d4a3 |
Hashes for pycdfpp-0.6.3-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60a10b8554f52516f406074b69a7501b1a733f81e38a565cb4ba7b2ca616e389 |
|
MD5 | 22b759cfcef3861ae3f4a0aa2b4900db |
|
BLAKE2b-256 | 7c1c41596fb89ad23994592bbb3e320937e502da3a753a2df9630bb9a9fc31f1 |
Hashes for pycdfpp-0.6.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dde707fb0b0d1274310b3450834372da15f57ec7a8890640cbbb7c211adcbe6d |
|
MD5 | d5be3c7221d1ec1c7a356dd23719f9be |
|
BLAKE2b-256 | a6afebd69a2b03988118f89727b32f24768e61c703c73908f2bbe9fa3087189e |
Hashes for pycdfpp-0.6.3-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d3e1c26e0fba71029dc1dbc5287069be9d75732fee6ea9d29c99f10d3537af7c |
|
MD5 | 72b97f504252caa29698bd33edfabf60 |
|
BLAKE2b-256 | d50f4a46f2b777b912ec3680cb2bc14638667b9f0049c9e8eb633df17821f3b8 |
Hashes for pycdfpp-0.6.3-cp312-cp312-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b45b8469d1a4cde66f71610f1a028299805484f4de60e16c5c44e0758d37764 |
|
MD5 | 478b0fa3d2db72b67930fa740fc12be8 |
|
BLAKE2b-256 | 55e5d4e51a4b2b39ff6dc9b29f4f6d0132946e28925657e5a4cf39e0a8531860 |
Hashes for pycdfpp-0.6.3-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a11f887da34f7c1a83f15fdcf9e68e23a65cda654115327ead5dc890119bf0a8 |
|
MD5 | 3e075b6cab9a0f038246d8a2d6b38eea |
|
BLAKE2b-256 | a728bcf57e6b977360bd6c8534409decd2065f4efdf5cdf22698852be5aa3813 |
Hashes for pycdfpp-0.6.3-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 55e955d7a6173247500d68e7dcaf44df31d2baacde8eff9aa75a5e7194b37432 |
|
MD5 | 0cfb57e321dfef90eec3ca61afcf19de |
|
BLAKE2b-256 | 074edb88ab42043e6c91dc7df4473fbddf441da0659288f59ffb5afb351c9829 |
Hashes for pycdfpp-0.6.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be06f6a84b870b4b097741a366a5f2fa35a55dfeb45900a1bc3afc1a5a5ac6dc |
|
MD5 | 3fc54c3d380349182495497d9339f288 |
|
BLAKE2b-256 | 789a5045bf16d11305bda1bb3f82530da8cc0eb67c71632f4b17ecae127f13ac |
Hashes for pycdfpp-0.6.3-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | db37aae0b041247083b164d56b312034387a94aae34ef1dec21b54486370e018 |
|
MD5 | faeecec8c319cfb7bb2865464e4accae |
|
BLAKE2b-256 | 2e2c5a2b37d82ce11e51e09f0f545fb8d258282312b7e1979c5f9b5fe22da0ba |
Hashes for pycdfpp-0.6.3-cp311-cp311-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6f21dbb59ff305adc432ae0eec17579f1302af3fb1b13552bd4ef077b02e965 |
|
MD5 | 9f06eba6eefc7e344e5d265f888fbd0b |
|
BLAKE2b-256 | acff2bbd94c8edbd0f391ad2f573dc99d56002211e0ab706d9b958b34f63a0ee |
Hashes for pycdfpp-0.6.3-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3d6327ac055b1e6421b57ee19b7ab313faefee3e9a923701d7cace4fe86942d2 |
|
MD5 | e0d8fcf31ffb355f608e6930e07a11a4 |
|
BLAKE2b-256 | b28ec2c624ac7134b1fd13bab861be161ab1151e39d34724fc07671fa72c133e |
Hashes for pycdfpp-0.6.3-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 015ce5707cbbc0174d2b956f57d36393a662a4241089377693232e54b04ad254 |
|
MD5 | 600e97ccb642ff67b1a8b5f0c7c8410d |
|
BLAKE2b-256 | d69abb95a5df8df2d0076ae4de6932df4e2144b2838876db69384f9539a4dd5c |
Hashes for pycdfpp-0.6.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22ca6b8d44717fb1eb18fc6c08d6412aacf2c205325ca44c830bc99d6ea6bdfe |
|
MD5 | 983505e9d6262b8f99d7c9c609fa05d1 |
|
BLAKE2b-256 | 8fabbf6e0660bc48dc86b8051afa855c9b6a4856f0a39ce2a059f9afd41f0de9 |
Hashes for pycdfpp-0.6.3-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cebd6d41a487da7843bc0e11d18f2962c24dfe9bbd3f63030498cde840a3cc7d |
|
MD5 | 904b6eb55c6edc95b330d312086cf765 |
|
BLAKE2b-256 | 543e11fdbd98755ba4041a41f50d0501475b16c20fd5bf36d6a696f7c59798c6 |
Hashes for pycdfpp-0.6.3-cp310-cp310-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba6f790053779b3bba018ed5ece41299dfb1508ead5bb4f2a914c190e9a1563d |
|
MD5 | 01a1e001940450ec8e0e8fcd7c946cbd |
|
BLAKE2b-256 | 6a20f30564789e9cbc80f950058c2be069c13a55c24576538803a9647925339a |
Hashes for pycdfpp-0.6.3-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 341d9a19edb1b1e8040fe8562619045a55b0e3acac3a6a89260f89447c893040 |
|
MD5 | 7e30d283825bde5f56e63e0e57c34668 |
|
BLAKE2b-256 | 9da70d6ec8b8ec6e21d4a59ce1b09145030b57980b666f22af0bb0ef443765cf |
Hashes for pycdfpp-0.6.3-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 132e420b9a5c655c3cbffd3f7aa57e31ce18a2ec8f851ad1333b812f85c9a8a1 |
|
MD5 | d21ef45b8ffcd8db333ce83b9b83e00e |
|
BLAKE2b-256 | bd3e95c37a2a14acd9cc4fe7d86ea3c22cec7d65407eac3f092deea4ef3028d3 |
Hashes for pycdfpp-0.6.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9cf521de95de82234ccaf34b5763dd5eaf5d90c92aeea5caf487a7e9433725d6 |
|
MD5 | e57efd03be2b572caf627981678d284d |
|
BLAKE2b-256 | 1dd453a26a674599dab80538a4f18701bf43a831bbefa6278f2db65b19bc2ebe |
Hashes for pycdfpp-0.6.3-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dcfae0e665890513212e1220aa6f1cb170462df126d705dbbce91f3e68658dab |
|
MD5 | c22d1f5ad5f1e0ad5681e26d48815990 |
|
BLAKE2b-256 | 5bef18ea699946589d616c37f4e78798af4949de81fc2098457a88f90e0bde5d |
Hashes for pycdfpp-0.6.3-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce34ea1c31e6fff359d6799af6378355dcdaf990703b9345550c796a79656aae |
|
MD5 | 8622e2b0c3447fe1ae4615854d9fdddd |
|
BLAKE2b-256 | 1ad3f779e0eb9f02bc5853179f49e1400d8a27049cac427c71f22b772b8455c8 |
Hashes for pycdfpp-0.6.3-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e9e62e7a9eab41b6f537495cd83f08224c7ca1221670537b8a2c3aa1ab8b32a8 |
|
MD5 | 08f1c348334052b59e1dab6fb360b73b |
|
BLAKE2b-256 | 96de8db86cb99a0da891459bbc7656ccf50bfd339c5dc18041417c47902c46e8 |
Hashes for pycdfpp-0.6.3-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb5a409a1200ac7a713eac11e8439fb4b9ed6b96eb419198884b3049f115627c |
|
MD5 | 9efe07ea5b51af3218cfb51ed257c751 |
|
BLAKE2b-256 | a1a1b3b1e25f1e6e5bf6a0ac6a154dac8d92bf49cd3e6a567eede2b50e78c7e4 |
Hashes for pycdfpp-0.6.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a69229fdc175124782e5eb6ca7f414e02d2ad304ae8a2cfd2cd62c791f8c947 |
|
MD5 | b7b6728e217faac8653ea136f68c0c63 |
|
BLAKE2b-256 | 4d976307251adccd335f191dbcacbbb6cc8c5bf1ac9ab8bbe595f91c135b9676 |
Hashes for pycdfpp-0.6.3-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa579e4e1e22d6b56dae879ef4b5b200019d84f2efc9e11421e0d5269d0e633b |
|
MD5 | 97766da415ccd4d86f1f92292087adc9 |
|
BLAKE2b-256 | 51189ea487791f2151f6544524c1ccec7ce80c7fbbaa38a41298bc0395187062 |
Hashes for pycdfpp-0.6.3-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3dc9dbb6b88f2731f5bb26991c9a682499f9aa4ffb96b7ec02a9c757f385d081 |
|
MD5 | 95326f729499721695f372ac90a220b8 |
|
BLAKE2b-256 | c66887163dd177e1ea553e17b043e4f9f2f36cfb62f9604ac1638713a6a45538 |