How it works

Define a class with attributes that are backed by disk storage.

from filebacked import FileBacked

class MyClass(FileBacked):
    myint: int

myobj = MyClass()
myobj.myint = 1

The type and name of the attribute will influence the format of the resulting HDF5 file. Let us save the object.

import h5py

with h5py.File('myfile.hdf5', 'w') as f:
    myobj.write(f)

The resulting file should have a root dataset named ‘myint’, a scalar with value 1. And now let us read it again.

with h5py.File('myfile.hdf5', 'r') as f:
    newobj = MyClass.read(f)
assert newobj.myint == 1

Supported types

The following types are supported:

• Scalar numbers (int, float and numpy scalar types)

• Strings (str)

• Numpy arrays (numpy.ndarray)

• Scipy sparse matrices (scipy.sparse.spmatrix) of CSR, CSC and COO type (although you are free to give the type hint as the general spmatrix superclass)

• Homogeneous tuples (Tuple[eltype, ...]) and lists (List[eltype]) where the element type is supported

• Dictionaries (Dict[keytype, valuetype]) where the key and value types are supported

• Option types (Option[valuetype]) where the value type is supported

• Union types with arguments that support type checking with isinstance(..., arg)

• Subclasses of FileBacked and FileBackedDict[keytype, valuetype]

Arbitrary Python objects are stored as pickled strings if the allow_pickle keyword argument is passed to the write and read methods, respectively, or if the type is object.

Types can be specified using standard builtins or type hint objects from the typing module, as above.

To add support for a custom type, create a new Filter subclass:

from filebacked import Filter, register_filter

class MyFilter(Filter):

    def applicable(self, tp):
        # Return true if the filter can be used for objects of the
        # given type.

    def write(self, group, name, obj, tp, **kwargs):
        # Write the object to the given group as a subgroup or
        # dataset with the given name.

    def read(self, group, tp, **kwargs):
        # Read the object from the given group or dataset and
        # return it.

register_filter(MyFilter())

Newly registered filters will take priority over existing filters.

Interface

For writing subclasses of FileBacked or FileBackedDict, it is most useful to use the following pattern. In this case, you cannot write more than one object to a file, or you risk overlapping attributes.

with h5py.File('myfile.hdf5', 'w') as f:
    myobj.write(f)

Alternatively, use the write function for arbitrary objects of supported type. In this case you must specify a name and optionally a type for the object. It is recommended to always specify the type, because element types of generic objects cannot be deduced from the object alone.

with h5py.File('myfile.hdf5', 'w') as f:
    filebacked.write(f, 'somename', 3, int)

The write function will detect subclasses of FileBacked or FileBackedDict and delegate writing accordingly, and the write method of those two classes will delegate writing of attributes to the write function.

All the write functions take an arbitrary amount of keyword arguments that are passed throughout the object reference tree. You can use this to customize writing behaviour. For example, the FileBacked.write and FileBackedDict.write methods accept the keyword arguments only and skip, to avoid writing some attributes if necessary:

class MyClass(FileBacked):
    small: int
    large: np.ndarray

    def write(self, group, sparse=False, **kwargs):
        if sparse:
            super().write(group, skip=('small',), **kwargs)
        else:
            super().write(group, **kwargs)

class MyClass(FileBacked):

    __filebacked_ignore__ = ('will_not_be_saved',)

    will_be_saved: int
    will_not_be_saved: str

Caution

Unlike pickle, FileBacked will not maintain reference equality between objects. If the same (mutable) object is referenced more than once in the reference graph, it will instantiate as two different mutable objects upon reading. For the same reason, circular references will cause problems.

FileBacked uses type hints to determine the structure of the resulting HDF5 file. It does not prevent you from assigning objects with incorrect types.