A modern C++ header only cdf library
Project description
Python packages
Linux x86_64 | Windows x86_64 | MacOs x86_64 | MacOs ARM64 |
---|---|---|---|
Unit Tests
Linux x86_64 | Windows x86_64 | MacOs x86_64 |
---|---|---|
CDFpp (CDF++)
A NASA's CDF modern C++ library. This is not a C++ wrapper but a full C++ implementation. Why? CDF files are still used for space physics missions but few implementations are available. The main one is NASA's C implementation available here but it lacks multi-threads support (global shared state), has an old C interface and has a license which isn't compatible with most Linux distributions policy. There are also Java and Python implementations which are not usable in C++.
List of features and roadmap:
- CDF reading
- read uncompressed file headers
- read uncompressed attributes
- read uncompressed variables
- read variable attributes
- loads cdf files from memory (std::vector or char*)
- handles both row and column major files
- read variables with nested VXRs
- read compressed files (GZip, RLE)
- read compressed variables (GZip, RLE)
- read UTF-8 encoded files
- read ISO 8859-1(Latin-1) encoded files (converts to UTF-8 on the fly)
- variables values lazy loading
- decode DEC's floating point encoding (Itanium, ALPHA and VAX)
- pad values
- CDF writing
- write uncompressed headers
- write uncompressed attributes
- write uncompressed variables
- write compressed variables
- write compressed files
- pad values
- General features
- uses libdeflate for faster GZip decompression
- highly optimized CDF reads (up to ~4GB/s read speed from disk)
- handle leap seconds
- Python wrappers
- Documentation
- Examples (see below)
- Benchmarks
If you want to understand how it works, how to use the code or what works, you may have to read tests.
Installing
From PyPi
python3 -m pip install --user pycdfpp
From sources
meson build
cd build
ninja
sudo ninja install
Or if youl want to build a Python wheel:
python -m build .
# resulting wheel will be located into dist folder
Basic usage
Python
Reading CDF files
Basic example from a local file:
import pycdfpp
cdf = pycdfpp.load("some_cdf.cdf")
cdf_var_data = cdf["var_name"].values #builds a numpy view or a list of strings
attribute_name_first_value = cdf.attributes['attribute_name'][0]
Note that you can also load in memory files:
import pycdfpp
import requests
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
plt.plot(tha_l2_fgm["tha_fgl_gsm"])
plt.show()
Buffer protocol support:
import pycdfpp
import requests
import xarray as xr
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
xr.DataArray(tha_l2_fgm['tha_fgl_gsm'], dims=['time', 'components'], coords={'time':tha_l2_fgm['tha_fgl_time'].values, 'components':['x', 'y', 'z']}).plot.line(x='time')
plt.show()
# Works with matplotlib directly too
plt.plot(tha_l2_fgm['tha_fgl_time'], tha_l2_fgm['tha_fgl_gsm'])
plt.show()
Datetimes handling:
import pycdfpp
import os
# Due to an issue with pybind11 you have to force your timezone to UTC for
# datetime conversion (not necessary for numpy datetime64)
os.environ['TZ']='UTC'
mms2_fgm_srvy = pycdfpp.load("mms2_fgm_srvy_l2_20200201_v5.230.0.cdf")
# to convert any CDF variable holding any time type to python datetime:
epoch_dt = pycdfpp.to_datetime(mms2_fgm_srvy["Epoch"])
# same with numpy datetime64:
epoch_dt64 = pycdfpp.to_datetime64(mms2_fgm_srvy["Epoch"])
# note that using datetime64 is ~100x faster than datetime (~2ns/element on an average laptop)
Writing CDF files
Creating a basic CDF file:
import pycdfpp
import numpy as np
from datetime import datetime
cdf = pycdfpp.CDF()
cdf.add_attribute("some attribute", [[1,2,3], [datetime(2018,1,1), datetime(2018,1,2)], "hello\nworld"])
cdf.add_variable(f"some variable", values=np.ones((10),dtype=np.float64))
pycdfpp.save(cdf, "some_cdf.cdf")
C++
#include "cdf-io/cdf-io.hpp"
#include <iostream>
std::ostream& operator<<(std::ostream& os, const cdf::Variable::shape_t& shape)
{
os << "(";
for (auto i = 0; i < static_cast<int>(std::size(shape)) - 1; i++)
os << shape[i] << ',';
if (std::size(shape) >= 1)
os << shape[std::size(shape) - 1];
os << ")";
return os;
}
int main(int argc, char** argv)
{
auto path = std::string(DATA_PATH) + "/a_cdf.cdf";
// cdf::io::load returns a optional<CDF>
if (const auto my_cdf = cdf::io::load(path); my_cdf)
{
std::cout << "Attribute list:" << std::endl;
for (const auto& [name, attribute] : my_cdf->attributes)
{
std::cout << "\t" << name << std::endl;
}
std::cout << "Variable list:" << std::endl;
for (const auto& [name, variable] : my_cdf->variables)
{
std::cout << "\t" << name << " shape:" << variable.shape() << std::endl;
}
return 0;
}
return -1;
}
caveats
- NRZ variables shape, in order to expose a consistent shape, PyCDFpp exposes the reccord count as first dimension and thus its value will be either 0 or 1 (0 mean empty variable).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pycdfpp-0.6.0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d72648fc71166f07cc48a1c8028933598531472fb4b716820ad763978dfb877e |
|
MD5 | c69ddcfcea3027b7a493c95d834aa8d5 |
|
BLAKE2b-256 | 2682c04553cd2ff5b62a0ec019fa0b6bd777c2fdc730cdfb209e9e2b0ba82768 |
Hashes for pycdfpp-0.6.0-cp312-cp312-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 63744fad6c46beb176b341fed29b454926a8c4e4cbe8b3f814eefbbdcd489a89 |
|
MD5 | 71f3b64097212c475b9f20e81ec6cef9 |
|
BLAKE2b-256 | cc362960bdc8ce0987f49e7482bad43f6e89623535b766a6a36bb8afffde3d9e |
Hashes for pycdfpp-0.6.0-cp312-cp312-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f8675661ce4b4cb6a8e8e3a0993801cfcf6cc9e071c1acc970996ab12dcf682 |
|
MD5 | f332445e1a61031e9e1b987b5006d9c7 |
|
BLAKE2b-256 | d819b3c2f88677ac1a864da461b72bfc583cd86a1f5ab5dff9d238dd93598189 |
Hashes for pycdfpp-0.6.0-cp312-cp312-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2059c4ae3262ba600f647599bd0e7cd5e4078b91a1b3f73c3b6a3da6bf90e1f0 |
|
MD5 | a95a3bbf25487241bd7c35e405f691ee |
|
BLAKE2b-256 | cfd99c2604341767686e8bc1706a99e270ca41a24469fea06db3cac59d900670 |
Hashes for pycdfpp-0.6.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f3e61ff6800cb0124b6ae4f53c3a4ca359c2b0d3ea066618fd58c5a9ef6b05d |
|
MD5 | cf75f5c27c29e9a8a47d2fe42d4b5fd2 |
|
BLAKE2b-256 | 576aa8bb8b8e91781f00a97fb1b43bc99903d1d4c3a464aedcd2848a4af3654d |
Hashes for pycdfpp-0.6.0-cp311-cp311-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2e49567a5e9e25824f86b670f5c12ddc6fd4623fa24bebddeb09593fed982cf |
|
MD5 | b80ef8545ba124c798733f252b9400db |
|
BLAKE2b-256 | fb74b0d2c393dc2dfbcf33a1e59a0e86db3d5f6d0b0f56ab13f74737140b7312 |
Hashes for pycdfpp-0.6.0-cp311-cp311-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85a33d0783f67f1d4c569949728fc36ddc80da0988bb266d43cabed416ce4c38 |
|
MD5 | fc0f3923849383bd9eb3ef6d05aff461 |
|
BLAKE2b-256 | 081a7cd89021ee104533ac67baee0ebf387dbb10febbb49b14cdf3295f8a296b |
Hashes for pycdfpp-0.6.0-cp311-cp311-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e7036a2fdc0db74b93167b6fc3e0303af9b6c4682ee74171bf1c4739c1cd05e |
|
MD5 | 06ecabcf076afc4e378a44361fcbc5c3 |
|
BLAKE2b-256 | da3b5a417daa33340709815cf9a394f1f55ff22974bf36f1172cdf4a8154f6bd |
Hashes for pycdfpp-0.6.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 08892c07223a2c557b3e32abb141c92795b20467d672fb2aba2fee4c3260a3c9 |
|
MD5 | 9355de0a10646e8418e2ad779d08554f |
|
BLAKE2b-256 | 42c0e054f77518b900251af3defbccca1309e571bedd545f79f911c11f714177 |
Hashes for pycdfpp-0.6.0-cp310-cp310-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f75866b95c080e00d8dd6c72eed74f9646dc4e1d8ef307b34615b1487a35f8b |
|
MD5 | 7dd5fa88ae5d7b245ce6fa1ce46d3fb5 |
|
BLAKE2b-256 | 0b94ab843c62677000955226ad63f5b26fcc53845282441fe4e94069b6febcfd |
Hashes for pycdfpp-0.6.0-cp310-cp310-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c883b1e8a465d731212ff0cc0cb9e980ede92cc160cf0a616455c368efcc181b |
|
MD5 | 8986a5d3705265b5eaf8b718599f4600 |
|
BLAKE2b-256 | b798db6a4ee478c46c4f36a62ecc7999f86c7ceb922f6011db562dd146459ece |
Hashes for pycdfpp-0.6.0-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d956ac841166001b4049066f6cd913645ccf83ac3322eaccba69cd627a27b22c |
|
MD5 | 8d615589038737ffd96fe3a249ce4173 |
|
BLAKE2b-256 | 57f8a4d1a51c018aec78ac40e1c6e04023cd6e795a62fd1dac039ef05002f778 |
Hashes for pycdfpp-0.6.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 760b2e4817e712013d4a3620d2b042744660e67dfd42c731d7d6122ad78c72a4 |
|
MD5 | cca969446cac632fd7dc86b9af0bc0c8 |
|
BLAKE2b-256 | 5ffd404ee2d8e59570849873f6af713cbc0a8a231b9e4a0df34be2fdac1e1766 |
Hashes for pycdfpp-0.6.0-cp39-cp39-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4dd954236880a9901db5728b5536dc1c01e9b314fcd04cccebf8899e53c449e2 |
|
MD5 | 9cc11bd5da2563bb4298d27f7f024303 |
|
BLAKE2b-256 | 3c35453aa85d53e9dcd5782a427aab95a7efad9600253aa398a4920cd2e03202 |
Hashes for pycdfpp-0.6.0-cp39-cp39-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7526df8bdeb95f6bfe2cd4deb8aca3344db689150eef7f6002460558f2ef1f8c |
|
MD5 | 1f4ca44c30219eebad41f45804754f1f |
|
BLAKE2b-256 | f860d3348d31ea50b22ff4df5203e0798b0ed97d3780509d0569f0fcd36006a4 |
Hashes for pycdfpp-0.6.0-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81b5752427a077abe88b764729fbe44b2c1889f9856ebcd1970a5ca545f44615 |
|
MD5 | 4cb54357246909f6a37d54b9540598af |
|
BLAKE2b-256 | 800cc398e4e0718f2f5a9e50f5fdeb24bb7916e4b330efe3ad9a1ab29af3d2b6 |
Hashes for pycdfpp-0.6.0-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab357dc1e7afbb29d084fbdc8c3270a3c3efaa77843b182e60f9b41f701c94eb |
|
MD5 | 8fead067e0c95e50162c48252ade3584 |
|
BLAKE2b-256 | f026588c8c28f3dab36011762bd268f4b4af6ebdd7d59b7550792569848d5e2e |
Hashes for pycdfpp-0.6.0-cp38-cp38-manylinux_2_28_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f26f6323044ecedf92550550db6e2e2f1d304b4cd2daff0c88f563866a59f56f |
|
MD5 | 7fd3e8bd7f832666219371deb0beee32 |
|
BLAKE2b-256 | 68daeade1b11975478dd98d3b25b2d4d80212e5784f8cae546fabbb453dd16cb |
Hashes for pycdfpp-0.6.0-cp38-cp38-macosx_13_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9368530db7b7ddd9d253093638ea77eb597c4a9936ea164bf531931d5891dcdb |
|
MD5 | 1de17d5cc5deee39d3db5935872f82e4 |
|
BLAKE2b-256 | 805af1843d5a54d5676df64d38b3097791ac61227d2c820a2a658cd0c96f3587 |
Hashes for pycdfpp-0.6.0-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c23beb08874d5ab528c825a42742cba8eae0083d752b32c07adfe67a6829755 |
|
MD5 | 8ed22582d82b79cb892937a63d3a2a40 |
|
BLAKE2b-256 | 0fe7220bdcade61f5cc4cdf9485cee176a0a756e6f4af3a7341bb730803ed949 |