Skip to main content

This repository contains utility functions written in Python for reading and writing files in different formats. They provide high level convenience wrappers for low level functions and libraries. Currently supported formats include: direct access binary, PETSc and HDF5. The HDF5 functions support both serial and parallel I/O and allow conveniently and efficiently saving/loading a wide variety of native Python datatypes as well as NumPy arrays.

IMPORTANT: Please do NOT post this code on your own github or other website. See LICENSE.txt for licensing information.

Feel free to email if you have any questions: samkat6@gmail.com


Installation

pip install pyioutils

For reading/writing HDF5 files in parallel, you will additionally need to install my pympiutils package:

pip install pympiutils

Note that parallelization depends on an MPI-enabled HDF5 installation and the h5py module built against it (instructions coming soon ...).

Usage

For more usage examples, see here. For parallel HDF5 I/O see here.

# Example 1: raw binary I/O
import numpy as np
from pyioutils.binaryio import write_binary, read_binary

# Write array as 64 bit float:
x=np.linspace(0., 7., num=8)
write_binary('x.bin', x, prec='>f8')

# Read data back in (default is as big endian 64 bit float)
xr=read_binary('x.bin', prec='f8', machineformat='>')

# Check
np.allclose(x,xr)

# Write multi-dim array to big endian binary file
# Data are written in Fortran (column major) order
y=np.reshape(x, (4,2), order='F')
write_binary('y.bin', y, prec='>f8')

# Read entire file and reshape using Fortran (column major) order
yr=read_binary('y.bin', dims=(4,2))

# Check
np.allclose(y,yr)

# Assume data are in blocks of size recSize; read the second record (iRec=1). This 
# assumes data are in the default big endian 64 bit float format.
y2=read_binary('y.bin', iRec=1, recSize=4)

# Write array to big endian binary file with header
z=np.array([280.,281.,282.,283.])
# Header as 32 bit int
write_binary('z.bin', len(z), prec='>i4')
# Append data as 64 bit float
write_binary('z.bin', z,prec='>f8', append=True)

# Read data back in skipping header
Tr=read_binary('z.bin', offsetBytes=4)
# Example 2: PETSc I/O
import numpy as np
from pyioutils.petscio import readPetscBinVec, writePetscBinVec, getPetscBinVecFileStats

# Write array to a PETSc Vec file
x=np.linspace(0.,7.,num=8)
writePetscBinVec("xy.petsc", x)

# Append another Vec
y=np.linspace(5.,10.,num=8)
writePetscBinVec("xy.petsc", y, append=True)

# Read both back in (multiple Vecs are returned as a list)
xy=readPetscBinVec("xy.petsc",nRec=-1)

# Check
np.allclose(x,xy[0])
np.allclose(y,xy[1])

# Read back in second Vec in file
yr=readPetscBinVec("xy.petsc", nRec=1, startRec=1)

# Check
np.allclose(y,yr)

# Multiple arrays can be written by passing them as a list
writePetscBinVec("xy2.petsc", [x, y])

# Query contents of a PETSc Vec file
vecLength, numVecs, numBytes=getPetscBinVecFileStats("xy.petsc")
print(f"{vecLength}, {numVecs}, {numBytes}")
# Example 3: HDF5 I/O
# For parallel I/O see the examples at: 
# https://github.com/samarkhatiwala/pympiutils/examples
import numpy as np
from pyioutils.hdfio import load, save

# Create some data
# NumPy array
x=np.arange(100,dtype=np.float64)
# List
y=[np.arange(n,dtype=np.float64) for n in range(1,5)]
# String
z='This is a test'
# Dictionary
d=dict({'v1': [1,2,3],'v2': 'some text', 'v3': np.ones(4)})

# Save the first three variables to HDF5 file passing them as a dictionary
save({'x': x, 'y': y, 'z': z}, 'data.h5')

# Add (append) the fourth one to the same file
save({'d': d}, "data.h5", append=True)

# Read it all back in
s=load("data.h5")

# Inspect
s.keys()
s.d.keys()

Metadata

Release files for pyioutils 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyioutils 1.0.0
File Size Uploaded Provenance Actions
pyioutils-1.0.0.tar.gz 12.6 kB Without provenance Details for pyioutils-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pyioutils 1.0.0
File Interpreter ABI Platform Actions
pyioutils-1.0.0-py3-none-any.whl Python 3 none any Details for pyioutils-1.0.0-py3-none-any.whl

Total release size: 24.8 kB

Release files / pyioutils-1.0.0.tar.gz

Download URL pyioutils-1.0.0.tar.gz
Size 12.6 kB
Tags Source
SHA-256 checksum
How to use checksums
5b2c057de323534c48ae27b5d3123cc3f27e7720d3e7942c9f4ef6fa0010fad7
BLAKE2b-256 checksum
How to use checksums
4f0dc2f6d2d15d9cf67c417ffe3a09660c1a5ebd5b20a2f874f7071380acbfab
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.0.1 CPython/3.9.6

Release files / pyioutils-1.0.0-py3-none-any.whl

Download URL pyioutils-1.0.0-py3-none-any.whl
Size 12.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0d981e01f72749b592f91ef7b3cc7f26b85f34db8ade1d78aa24d033b6623c6c
BLAKE2b-256 checksum
How to use checksums
f21eafd2501331e1328b6612d652d3689a2652122f626ffba3b722cb34871579
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.0.1 CPython/3.9.6

Security for pyioutils 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page