CDFpp (CDF++)

A modern, from-scratch C++20 implementation of NASA's CDF (Common Data Format) with full Python bindings.

📖 Documentation: quickstart, guides for reading and for producing ISTP-compliant files, cookbook, and C++ guide. The examples run live in your browser.

Why another CDF library? NASA's official C implementation has no multi-thread support (global shared state), an aging C89 interface, and a license incompatible with most Linux distribution policies. CDFpp solves all three: it is thread-safe, idiomatic C++20, and MIT-licensed.

Highlights

• C++20 library, mostly headers — see the C++ guide for the compiler flags
• Complete read/write support — CDF versions 2.2 through 3.x, row and column major, compressed files and variables (GZip, RLE)
• Python bindings (pycdfpp) via pybind11 — zero-copy NumPy integration, GIL-free I/O
• SIMD-accelerated time conversions — AVX512/AVX2/SSE2 runtime dispatch for TT2000, EPOCH, EPOCH16
• Lazy loading — variable data is read on first access, not at file open
• Read files where they live — pycdfpp.load("s3://…") reads only the byte ranges it needs, through fsspec or your own byte-range reader, instead of downloading the file
• Drop-in cdflib replacement — replace import cdflib with import pycdfpp.cdflib as cdflib and your cdflib code runs on pycdfpp, unchanged. For a library you can't edit, such as SunPy, call pycdfpp.cdflib.install() before importing it: its own import cdflib then gets pycdfpp's version too
• Fast — 5.6× to several hundred times faster than spacepy and cdflib at reading, 3.7× to 14× at writing compressed files (see below); SIMD time conversions at up to 8 billion epochs/s (TT2000, AVX-512)
• Runs everywhere — Linux, Windows, macOS (x86_64 + ARM64), and WebAssembly (Pyodide / emscripten-forge)
• In-browser app — CDFpp Explorer inspects, plots, and ISTP-validates CDF files entirely client-side, no install

Packages & CI

	Linux x86_64	Linux aarch64	Windows x86_64	Windows ARM64	macOS x86_64	macOS ARM64	WASM (Pyodide)
Wheels
Tests

Also available on emscripten-forge for use in JupyterLite and other Emscripten-based environments.

CDFpp Explorer — CDF files in your browser

CDFpp compiles to WebAssembly, so the whole library runs client-side — no install, no server, and your files never leave your machine. CDFpp Explorer is a small web app built on it:

• Inspect — browse variables and attributes grouped by ISTP VAR_TYPE, with live search and a value preview
• Plot — line plots and spectrograms, ISTP-aware (DEPEND_0 time axis, DISPLAY_TYPE, SCALETYP, fill/valid masking), with CSV / JSON export
• Validate — hand the file to AstraLint for ISTP conformance checking

👉 Open CDFpp Explorer

The WebAssembly wrapper lives in wacdfpp/ and ships TypeScript declarations (wacdfpp/cdfpp.d.ts) for embedding CDF read/write — with zero-copy typed arrays — in your own JavaScript app.

Installing

From PyPI

pip install pycdfpp

# optional: read files on S3, or on any store fsspec knows, without downloading them
pip install pycdfpp[s3]

# optional: the cdflib-compatible API
pip install pycdfpp[cdflib]

C++ library

See Adding CDFpp to your project in the C++ guide.

From source (Python wheel)

python -m build .
# wheel is in dist/

Quick start (Python)

Reading a CDF file

import pycdfpp

cdf = pycdfpp.load("my_data.cdf")

# Variables expose numpy arrays (zero-copy when possible)
data = cdf["variable_name"].values

# Global attributes
print(cdf.attributes["Project"][0])

# Variable attributes
print(cdf["variable_name"].attributes["UNITS"].value)

# Iterate over all variables
for name, var in cdf.items():
    print(f"{name}: shape={var.shape}, type={var.type}")

Loading from memory

import pycdfpp

# Load from any bytes-like object (useful with HTTP responses, S3, etc.)
with open("my_data.cdf", "rb") as f:
    cdf = pycdfpp.load(f.read())

Reading a file on S3

Only the byte ranges a read needs are fetched, each at most once. On NASA's HelioCloud, opening a 53.6 MB file fetched 2.3 MB, and reading one variable 8.6 MB. Nothing is downloaded whole by default.

import pycdfpp

url = ("s3://gov-nasa-hdrl-data1/spdf/cdaweb/data/ace/mag/level_2_cdaweb/"
       "mfi_h0/2020/ac_h0_mfi_20200101_v07.cdf")
cdf = pycdfpp.load(url)
cdf["BGSEc"].values   # fetches only this variable

Any fsspec URL works (s3://, gs://, https://…), and so does any object with size() and read_ranges(). Public buckets are read anonymously; AWS credentials found on the machine are used when there are some, and storage_options goes to fsspec as is. See Reading files.

Time conversions

CDFpp handles all three CDF time types (EPOCH, EPOCH16, TT2000) and converts them to numpy datetime64[ns] or Python datetime:

import pycdfpp
import numpy as np

cdf = pycdfpp.load("my_data.cdf")

# Convert any CDF time variable to numpy datetime64 (fast, ~2ns/element)
times = pycdfpp.to_datetime64(cdf["Epoch"])

# Or to Python datetime objects
times_dt = pycdfpp.to_datetime(cdf["Epoch"])

# Format as strings (e.g. for PDS4 compliance)
time_strings = pycdfpp.to_time_string(cdf["Epoch"], "%Y-%m-%dT%H:%M:%SZ")
# array([b'2020-02-01T00:00:00.000000000Z', ...])

# Convert Python/numpy times to CDF types
tt2000_values = pycdfpp.to_tt2000(np.array(['2020-01-01', '2020-06-15'], dtype='datetime64[ns]'))
epoch_values = pycdfpp.to_epoch(np.array(['2020-01-01', '2020-06-15'], dtype='datetime64[ns]'))

Creating and writing CDF files

import pycdfpp
import numpy as np
from datetime import datetime

cdf = pycdfpp.CDF()

# Add global attributes (each entry is a list of values)
cdf.add_attribute("Project", ["MyMission"])
cdf.add_attribute("PI_name", ["Jane Doe"])

# Add a time variable with TT2000 encoding
times = np.arange('2020-01-01', '2020-01-02', dtype='datetime64[h]').astype('datetime64[ns]')
cdf.add_variable("Epoch", values=times, data_type=pycdfpp.DataType.CDF_TIME_TT2000)

# Add a data variable with variable attributes
cdf.add_variable("B_GSM",
    values=np.random.randn(24, 3).astype(np.float32),
    attributes={
        "FIELDNAM": "Magnetic Field",
        "UNITS": "nT",
        "DEPEND_0": "Epoch",
    })

# Save to disk
pycdfpp.save(cdf, "output.cdf")

# Or save to memory (returns bytes)
data = pycdfpp.save(cdf)

Compressed CDF files

import pycdfpp
import numpy as np

cdf = pycdfpp.CDF()
cdf.add_variable("data", values=np.zeros(10000, dtype=np.float64))

# Whole-file GZip compression
cdf.compression = pycdfpp.CompressionType.gzip_compression
pycdfpp.save(cdf, "compressed.cdf")

# Or per-variable compression
cdf2 = pycdfpp.CDF()
cdf2.add_variable("data",
    values=np.zeros(10000, dtype=np.float64),
    compression=pycdfpp.CompressionType.gzip_compression)

Filtering variables

import pycdfpp

cdf = pycdfpp.load("large_file.cdf")

# Keep only specific variables and attributes (returns a new CDF)
filtered = cdf.filter(variables=["Epoch", "B_GSM"], attributes=["Project"])

# Filter with a regex pattern
filtered = cdf.filter(variables="B_.*", attributes=".*")

# Filter with a callable
filtered = cdf.filter(variables=lambda var: var.name.startswith("B_"))

# Drop one variable attribute, here Frame of B
filtered = cdf.filter(variable_attributes=lambda var, attr: not (var.name == "B" and attr.name == "Frame"))

Cloning variables between CDF files

import pycdfpp

src = pycdfpp.load("source.cdf")
dst = pycdfpp.CDF()

# Clone a variable (deep copy, including its attributes)
dst.add_variable(src["Epoch"])
dst.add_variable(src["B_GSM"])

NumPy buffer protocol

Variables implement the Python buffer protocol, so they work directly with NumPy and any library that accepts array-like objects:

import pycdfpp
import numpy as np

cdf = pycdfpp.load("my_data.cdf")

# Direct numpy array construction (zero-copy for numeric types)
arr = np.array(cdf["B_GSM"])

Quick start (C++)

Reading

#include "cdfpp/cdf-io/cdf-io.hpp"
#include <iostream>

int main()
{
    // cdf::io::load returns std::optional<CDF>
    if (auto cdf = cdf::io::load("my_data.cdf"))
    {
        for (const auto& [name, variable] : cdf->variables)
            std::cout << name << " shape: " << variable.shape() << "\n";

        for (const auto& [name, attribute] : cdf->attributes)
            std::cout << name << "\n";
    }
}

Writing

#include "cdfpp/cdf-io/cdf-io.hpp"

int main()
{
    cdf::CDF my_cdf;

    // Save to file (returns bool)
    cdf::io::save(my_cdf, "output.cdf");

    // Or save to memory (returns a vector<char>)
    auto data = cdf::io::save(my_cdf);
}

Loading from memory

#include "cdfpp/cdf-io/cdf-io.hpp"
#include <vector>

void process(const std::vector<char>& buffer)
{
    if (auto cdf = cdf::io::load(buffer.data(), buffer.size()))
    {
        // ...
    }
}

Benchmarks

Compared with spacepy and cdflib

Everyday tasks on real CDAWeb files, with each library used the way its documentation shows, including its fastest time conversion. All three return the same values.

(N×) = N times longer than pycdfpp. Median of 5 runs, files in the page cache. AMD Ryzen 7 5800X (AVX2, no AVX-512), Python 3.13, pycdfpp 0.15.0, spacepy 0.7.0 (NASA CDF 3.9.0), cdflib 1.3.14.

On an Apple M2 (MacBook Air, macOS, NEON), pycdfpp is faster on every task, writing without compression included:

Same script, on AC power. Python 3.14, pycdfpp 0.16.0 built from source like the wheels, spacepy 0.7.0 (NASA CDF 3.9.1), cdflib 1.3.14. See what was done for Apple Silicon.

Why is pycdfpp faster?

• Opening a file only parses the headers, in C++. NASA's library, used by spacepy, hashes the whole file to check its MD5 checksum on every open. pycdfpp skips that check.
• Time conversion runs in C++ with SIMD, straight to datetime64[ns]. For TT2000, spacepy creates one Python datetime per value. cdflib converts CDF_EPOCH in a Python loop.
• Gzip blocks are decompressed on all cores at once (an MMS FPI distribution variable has 640 of them), with libdeflate, itself 1.5–1.7× faster than zlib on these files. On Linux, big buffers use 2 MB huge pages.
• Threads work: pycdfpp releases the GIL while reading and decompressing. cdflib is mostly Python, so it holds the GIL. NASA's library keeps global state.
• Writing compresses 256 KB blocks on all cores, with libdeflate. The other two compress with zlib, on one thread. Without compression there is little to gain. With copy=False, pycdfpp writes straight from the arrays, like spacepy. The rest of the gap is the time axis: pycdfpp converts it from datetime64 (1.3 ms on the Ryzen, 0.2 ms on the M2), spacepy is given TT2000 integers.

Reading scales to about 2× with threads, and to 3.5 GB/s on big files; writing reaches 710 MB/s with gzip, where spacepy stays at 42 MB/s. See the scaling results.

Details and caveats in the performance page. Reproduce with benchmarks/python_libs/compare.py.

C++ micro-benchmarks

Release builds (-O3). Source code in benchmarks/.

SIMD time conversions

Converting CDF time types to nanoseconds since 1970 (epochs/s, higher is better, one thread). On x86, CDFpp picks the best instruction set the CPU has at run time (AVX-512, AVX2 or SSE2). On ARM (aarch64), it uses NEON code of its own.

AMD Ryzen 7 7840U/HS laptop CPU (Zen 4, 5.1 GHz boost, 16 MB L3), AVX-512, measured with pycdfpp 0.13, before CDF_EPOCH conversions became exact (see below):

AMD Ryzen 7 5800X desktop CPU (Zen 3, 32 MB L3), AVX2 (median of 3 runs):

With AVX-512, TT2000 conversion peaked at ~8 billion epochs/s and EPOCH at ~14 billion epochs/s for L1/L2-resident data. With AVX2, SIMD runs 2.5–3× faster than scalar code. CDF_EPOCH conversions are exact since 0.14.0: they used to round to 256 ns. That made scalar EPOCH conversion about 2× slower (it was 2.3e+09 epochs/s on the 5800X), while the SIMD version, reworked to need no 64-bit integer conversions, is faster than the old scalar one. At 64M values (1 GB of data), every SIMD row drops to 1.2–1.5 billion epochs/s: the data no longer fits in cache.

Apple M2 (MacBook Air, 4+4 cores, 16 MB L2), NEON (median of 3 runs):

These TT2000 values are sorted from 1972 to 2036. NEON converts sorted values between two leap seconds 32 at a time, with one addition each, so it needs runs longer than a few dozen values: with 1K values, there is a leap second every 38 of them. The M2 keeps 4.2 billion epochs/s at 64M values, 1 GB of data, where the x86 CPUs drop to 1.3–1.5 billion.

Leap-second lookup

Epochs/s, one row per CPU:

RLE compression (bytes/s)

RLE inflate sustains ~1.8 GB/s on the 7840U/HS for data that fits in cache; on the 5800X it drops from 1.3 GB/s at 1 KB to 0.4 GB/s at 64 KB. At 1 MB both CPUs run at the same speed.

Features & roadmap

• Reading
  • CDF versions 2.2 through 3.x
  • Compressed files and variables (GZip, RLE)
  • Row and column major
  • Nested VXRs
  • Lazy variable loading
  • UTF-8 and ISO 8859-1 (Latin-1, auto-converted to UTF-8)
  • In-memory loading (std::vector<char>, char*, Python bytes)
  • DEC floating-point encoding (VAX, Alpha, Itanium)
  • Pad values and sparse records
• Writing
  • Uncompressed and compressed files/variables
  • All numeric types, strings, datetime types
  • Pad values and sparse records
• General
  • libdeflate for faster GZip
  • SIMD time conversions (AVX512/AVX2/SSE2 with runtime dispatch, NEON on ARM)
  • Leap-second handling
  • Python bindings with GIL-free I/O
  • Documentation

Caveats

• NRV variables shape: PyCDFpp exposes the record count as the first dimension, so NRV variables will have shape (0, ...) or (1, ...).
• Removed entries: Python wrappers hold references into C++ containers. Adding or removing other variables or attributes is safe, but a wrapper of an entry that is itself removed, e.g. by pycdfpp.filter_cdf(..., inplace=True), dangles:

  var = cdf["B_GSM"]
  cdf.add_variable("new_var", values=np.zeros(10))
  var.values  # fine
  
  pycdfpp.filter_cdf(cdf, variables=["new_var"], inplace=True)  # removes B_GSM
  var.values  # dangling: don't
  

See the full documentation for more details.