Skip to main content

pypi ci mit

Seamlessly compress VTK datasets using Zstandard.

Read in VTK datasets 37x faster, write 14x faster, all while using 28% less space versus VTK’s modern XML format.

Read/Write Speedup and Compression Ratios

Read/Write Speedup and Compression Ratios

File Type / Method

Write Speed

Compression Ratio

Notes

Legacy VTK (.vtk)

465 MB/s

0.88

Significant overhead

VTK XML, none

256 MB/s

0.70

Significant overhead

VTK XML, zlib

105 MB/s

2.52

VTK Default

VTK XML, lz4

401 MB/s

1.47

VTK XML, lzma

9.93 MB/s

3.10

VTK HDF (.vtkhdf), lvl0

1733 MB/s

0.93

No compression

VTK HDF (.vtkhdf), lvl4

137 MB/s

2.37

Default compression

pyvista-zstd (.pv), lvl3

711 MB/s

3.02

Threads = 0

pyvista-zstd (.pv), lvl3

1845 MB/s

3.02

Threads = 4

pyvista-zstd (.pv), lvl22

15.8 MB/s

3.79

All threads (-1)

Usage

Install with:

pip install pyvista-zstd

The package is a C++ library with a Python wrapper around it, so an install is either a prebuilt wheel or a compile. Wheels are published for:

Linux

x86_64 (manylinux2014 and manylinux_2_28)

macOS

x86_64 and arm64

Windows

AMD64

Everywhere else – musl-based Linux, Linux on aarch64, Windows on ARM – pip falls back to the source distribution and builds the core during the install, which needs a C++17 compiler on the machine. CMake does not: it comes from the build requirements. A machine without a compiler fails the install rather than producing a package that imports and cannot read anything.

Compatible with all VTK dataset types. Uses PyVista under the hood.

import pyvista_zstd

# create and write out
ds = pv.Sphere()
pyvista_zstd.write(ds, "dataset.pv")

# read in and show these are identical
ds_in = pyvista_zstd.read("dataset.pv")
assert ds == ds_in

For cell arrays where every cell contains the same number of points, pyvista-zstd stores the common cell width in the dataset metadata and omits the redundant offsets array. This applies to triangles, quads, tetrahedra, and any other fixed-width cell topology. Mixed-width cell arrays retain their explicit offsets.

Alternative VTK example

import vtk
import pyvista_zstd

# create dataset using VTK source
sphere_source = vtk.vtkSphereSource()
sphere_source.SetRadius(1.0)
sphere_source.SetThetaResolution(32)
sphere_source.SetPhiResolution(32)
sphere_source.Update()

vtk_ds = sphere_source.GetOutput()

# read back
pyvista_zstd.write(vtk_ds, "sphere.pv")
ds_in = pyvista_zstd.read("sphere.pv")

PyVista Integration

When pyvista-zstd is installed, it automatically registers with PyVista’s reader registry. This means pv.read() handles .pv files directly:

import pyvista as pv

mesh = pv.read("dataset.pv")

No additional imports needed. This works via PyVista’s pyvista.readers entry point group, so the registration happens at install time.

The C library

The reader and writer are a C++ core with a pure C ABI, pvzstd, declared in cpp/include/pvzstd/pvzstd.h. The Python package is one consumer of it, bound with ctypes; a C or C++ consumer – or a WebAssembly build – can link the same library directly, and it depends on nothing but zstd.

find_package(pvzstd CONFIG REQUIRED)
target_link_libraries(my_app PRIVATE pvzstd::pvzstd)

See The C library for the build options, both ways of consuming the CMake package, the ABI-version contract, and a worked example. The on-disk format is specified in doc/format/container-v2.md.

Rational

VTK’s XML writer is flexible and supports most datasets, but its compression is limited to a single thread, has only a subset of compression algorithms, and the XML format adds significant overhead.

To demonstrate this, the following example writes out a single file without compression. This example requires pyvista>=0.47.0 for the compression parameter.

>>> import numpy as np
>>> import pyvista as pv
>>> ugrid = pv.ImageData(dimensions=(200, 200, 200)).to_tetrahedra()
>>> ugrid["pdata"] = np.random.random(ugrid.n_points)
>>> ugrid["cdata"] = np.random.random(ugrid.n_cells)
>>> nbytes = (
...     ugrid.points.nbytes
...     + ugrid.cell_connectivity.nbytes
...     + ugrid.offset.nbytes
...     + ugrid.celltypes.nbytes
...     + ugrid["pdata"].nbytes
...     + ugrid["cdata"].nbytes
... )
>>> print(f"Size in memory: {nbytes / 1024**2:.2f} MB")

Size in memory: 1993.89 MB
Save using VTK XML format

>>> from pathlib import Path
>>> import time
>>> tmp_path = Path("/tmp/ds.vtu")
>>> tstart = time.time()
>>> ugrid.save(tmp_path, compression=None)
>>> print(f"Written without compression in {time.time() - tstart:.2f} seconds")
>>> nbytes_disk = tmp_path.stat().st_size
>>> print(f"  File size:            {nbytes_disk / 1024**2:.2f} MB")
>>> print(f"  Compression Ratio:    {nbytes / nbytes_disk}")
>>> print()

Written without compression in 7.93 seconds
File size:            2858.94 MB
Compression Ratio:    0.6974239255525742

This amounts to around a 43% overhead using VTK’s XML writer. Using the default compression we can get the file size down to 791 MB, but it takes 19 seconds to compress.

>>> tstart = time.time()
>>> ugrid.save(tmp_path, compression='zlib')  # default
>>> print(f"Compressed in {time.time() - tstart:.2f} seconds")
>>> nbytes_disk = tmp_path.stat().st_size
>>> print(f"  File size:            {nbytes_disk / 1024**2:.2f} MB")
>>> print(f"  Compression Ratio:    {nbytes / nbytes_disk}")
>>> print()

Compressed in 18.83 seconds
File size:            791.05 MB
Compression Ratio:    2.5205590295735663

Clearly there’s room for improvement here as this amounts to a compression rate of 105.89 MB/s.

VTK Compression with Zstandard: pyvista-zstd

This library, pyvista-zstd, writes out VTK datasets with minimal overhead and uses Zstandard for compression. Moreover, it’s been implemented with multi-threading support for both read and write operations.

Let’s compress that file again but this time using pyvista-zstd:

>>> import pyvista_zstd
>>> tmp_path = Path("/tmp/ds.pv")
>>> tstart = time.time()
>>> pyvista_zstd.write(ugrid, tmp_path)
>>> print(f"Compressed pyvista_zstd in {time.time() - tstart:.2f} seconds")
>>> nbytes_disk = tmp_path.stat().st_size
>>> print(f"  File size:            {nbytes_disk / 1024**2:.2f} MB")
>>> print(f"  Compression Ratio:    {nbytes / nbytes_disk}")

Compressed pyvista_zstd in 0.92 seconds
Threads:              -1
File size:            660.41 MB
Compression Ratio:    3.019175309922273

This gives us a write performance of 2167 MB/s using the default number of threads and compression level, resulting in a 20x speedup in write performance versus VTK’s XML writer. This speedup is most noticeable for larger files:

Speedup versus VTK’s XML

Speedup versus VTK’s XML

Even when disabling multi-threading we can still achieve excellent performance:

>>> tstart = time.time()
>>> pyvista_zstd.write(ugrid, tmp_path, n_threads=0)
>>> print(f"Compressed pyvista_zstd in {time.time() - tstart:.2f} seconds")
>>> nbytes_disk = tmp_path.stat().st_size
>>> print(f"  File size:            {nbytes_disk / 1024**2:.2f} MB")
>>> print(f"  Compression Ratio:    {nbytes / nbytes_disk}")

Compressed pyvista_zstd in 2.91 seconds
Threads:              0
File size:            660.47 MB
Compression Ratio:    3.0188911592355683

This amounts to a single-core compression rate of 685.18 MB/s, which is in agreement with Zstandard’s benchmarks.

Note that the benefit of threading drops off rapidly past 8 threads, though part of this is due to the performance versus efficiency cores of the CPU used for benchmarking (see below).

Read/Write Speed versus Number of Threads

Read/Write Speed versus Number of Threads


Reading in the dataset is also fast. Comparing with VTK’s XML reader using defaults:

Read VTK XML

>>> print(f"Read VTK XML:")
>>> timeit pv.read("/tmp/ds.vtu")
6.22 s ± 9.21 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Read zstd

>>> print(f"Read zstd:")
>>> timeit pyvista_zstd.read("/tmp/ds.pv")
563 ms ± 7.96 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

This is an 11x speedup for this dataset versus VTK’s XML, and it’s still fast even with multi-threading disabled:

>>> timeit pyvista_zstd.read("/tmp/ds.pv", n_threads=0)
1.11 s ± 4.51 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

This amounts to 1796 MB/s for a single core, which is also in agreement with Zstandard’s benchmarks.

Additionally, you can control Zstandard’s compression level by setting level=. A quick benchmark for this dataset indicates the defaults give a reasonable performance versus size tradeoff:

Read/Write Speed versus Compression Level

Read/Write Speed versus Compression Level

Note that both pyvista-zstd and VTK’s XML default compression give relatively constant compression ratios for this dataset across varying file sizes:

Compression Ratio versus VTK’s XML

Compression Ratio versus VTK’s XML

These benchmarks were performed on an i9-14900KF running the Linux kernel 6.12.41 using zstandard==0.24.0 from PyPI. Storage was a 2TB Samsung 990 Pro without LUKS mounted at /tmp.

Additional Information

The benchmarks/ directory contains additional benchmarks using many datasets, including all applicable datasets in pyvista.examples (see PyVista Dataset Gallery).

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyvista_zstd-0.4.0.tar.gz (1.2 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

pyvista_zstd-0.4.0-py3-none-win_amd64.whl (263.4 kB view details)

Uploaded Python 3Windows x86-64

pyvista_zstd-0.4.0-py3-none-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (431.2 kB view details)

Uploaded Python 3manylinux: glibc 2.24+ x86-64manylinux: glibc 2.28+ x86-64

pyvista_zstd-0.4.0-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (447.7 kB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

pyvista_zstd-0.4.0-py3-none-macosx_11_0_arm64.whl (300.3 kB view details)

Uploaded Python 3macOS 11.0+ ARM64

pyvista_zstd-0.4.0-py3-none-macosx_10_15_x86_64.whl (366.4 kB view details)

Uploaded Python 3macOS 10.15+ x86-64

File details

Details for the file pyvista_zstd-0.4.0.tar.gz.

File metadata

  • Download URL: pyvista_zstd-0.4.0.tar.gz
  • Upload date:
  • Size: 1.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pyvista_zstd-0.4.0.tar.gz
Algorithm Hash digest
SHA256 63295528c70ccf869e04b2d716b0d28650418a122daf08b5dd139afb8b6498dd
MD5 4a385c2eef446de30b4998d9365b1bb3
BLAKE2b-256 ba061c8da93710ec76f4f48c0b73d45a095091984bfc6a167728708802af4ecc

See more details on using hashes here.

Provenance

The following attestation bundles were made for pyvista_zstd-0.4.0.tar.gz:

Publisher: ci_cd.yml on pyvista/pyvista-zstd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pyvista_zstd-0.4.0-py3-none-win_amd64.whl.

File metadata

  • Download URL: pyvista_zstd-0.4.0-py3-none-win_amd64.whl
  • Upload date:
  • Size: 263.4 kB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pyvista_zstd-0.4.0-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 95dafc6cea595a17426c7a9e78021fedd0749f96b197847a937c3712b269c298
MD5 1b798284e9969ab617e266895b45fcf7
BLAKE2b-256 1b69f84654080831baee8209c89bd60161c8588f9d8e20ab1393b130fa2b312b

See more details on using hashes here.

Provenance

The following attestation bundles were made for pyvista_zstd-0.4.0-py3-none-win_amd64.whl:

Publisher: ci_cd.yml on pyvista/pyvista-zstd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pyvista_zstd-0.4.0-py3-none-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for pyvista_zstd-0.4.0-py3-none-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 b4868341c458660f4954b78b1c77b50abd8ff4242871e59bd60797d8019c74d9
MD5 ecf7d25a9b0e3300d65ea3e2063d16cb
BLAKE2b-256 1e99ae751ea6a4e4ad2d133fe835b4993f4c40181ac1e9516588cea6fbf036c9

See more details on using hashes here.

Provenance

The following attestation bundles were made for pyvista_zstd-0.4.0-py3-none-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl:

Publisher: ci_cd.yml on pyvista/pyvista-zstd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pyvista_zstd-0.4.0-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for pyvista_zstd-0.4.0-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 e9333f55072f8167b8bb66300377ff220cb0a2d7f1307bce292a89317af032ce
MD5 d2a0f1644fbe761b0ff8308a97be5ff7
BLAKE2b-256 9e55903f8f8079d124d6d08f2ffa80b1d39fdd4474841adef7d73b3001a813b0

See more details on using hashes here.

Provenance

The following attestation bundles were made for pyvista_zstd-0.4.0-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: ci_cd.yml on pyvista/pyvista-zstd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pyvista_zstd-0.4.0-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for pyvista_zstd-0.4.0-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 cd30fb559f1a31c50d9141af41c1b6d1e4d8b351866b7dad511d5c6ed6964665
MD5 65769abca5371f4888bdde54b63bb082
BLAKE2b-256 dd125a848fe5b29cdd3d16da510f571e109fa99cfbd10824fff95bcd3ed92046

See more details on using hashes here.

Provenance

The following attestation bundles were made for pyvista_zstd-0.4.0-py3-none-macosx_11_0_arm64.whl:

Publisher: ci_cd.yml on pyvista/pyvista-zstd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pyvista_zstd-0.4.0-py3-none-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for pyvista_zstd-0.4.0-py3-none-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 695e7c0ab2a4d14a9cee3905774febe2eccc0e3e78eb71540dfb73a7b8d8da22
MD5 da6425dc5b580059d0e75617eee17296
BLAKE2b-256 bf4273fe96e9bac8beed7a8d92b41cbd49ffc2ad261c0cc40098ed7c49cf5c31

See more details on using hashes here.

Provenance

The following attestation bundles were made for pyvista_zstd-0.4.0-py3-none-macosx_10_15_x86_64.whl:

Publisher: ci_cd.yml on pyvista/pyvista-zstd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.4.1

8 files

This release

0.4.0 This release

6 files

0.3.1

2 files

0.3.0

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page