Skip to main content

h5t

PyPI Python versions CI License: MIT

h5t loads HDF5 groups into detached, typed Python records, and writes them back with the same schema. Group attributes and ordinary NumPy-array fields are materialized while the file is open. Typed datasets keep snapshot metadata and can read their payload lazily without retaining an open file descriptor.

Installation

pip install h5t     # or: uv add h5t

Requires Python 3.11+. h5py, numpy, and Pydantic v2 are installed automatically.

Example

A schema is a plain class -- @h5t.dataset for a child dataset, @h5t.group for a group (including the root) -- loaded with h5t.load:

import dataclasses
import json
from pathlib import Path
from typing import Annotated, Any

import numpy as np

import h5t


@h5t.dataset(data="data")
@dataclasses.dataclass
class Measurement:
    unit: str
    data: h5t.LazyArray                       # lazy detached payload


@h5t.group()
@dataclasses.dataclass
class Nested:
    label: str


@h5t.group(extras="ignore")
@dataclasses.dataclass
class Result:
    version: int                              # implicit HDF5 attribute
    title: Annotated[str, h5t.Name("name")] # renamed attribute
    config: Annotated[
        dict[str, Any],
        h5t.Attr(converter=json.loads),
    ]
    array_attr: Annotated[np.ndarray, h5t.Attr()]
    values: np.ndarray                        # eager dataset payload
    samples: h5t.LazyArray                    # child dataset, no attributes declared
    measurement: Measurement                  # loaded via @h5t.dataset
    nested: Nested                            # recursively loaded group record
    note: str | None                          # absent becomes None
    revision: int = 1                         # absent uses a validated default


result = h5t.load(Result, Path("result.h5"), root="/")

Bind a Mapping-annotated field with attrs= (see below) to receive the node's full attribute snapshot: an immutable mapping keyed by on-disk HDF5 name, where declared attributes hold their Pydantic-processed values and undeclared ones retained under extras="ignore" hold the raw h5py values.

A LazyArray payload exposes snapshot metadata and explicit data access:

lazy = result.measurement.data
lazy.path, lazy.shape, lazy.dtype, lazy.ndim

complete = lazy.data         # first access reads and caches an ndarray
assert lazy.read() is complete

with lazy.open() as live:
    first_hundred = live[:100]  # fresh file view, useful for slices

h5t.load() closes every handle on normal and exceptional exits, and so does LazyArray.open(). Loading builds a record by calling the record type's own constructor with the loaded values; h5t.save (below) is the inverse.

Writing

h5t.save(record, path, root="/", mode="w-", overwrite=False) validates a @h5t.group record against its own schema and writes it, so the code that produces files is checked by the same declarations that read them:

h5t.save(result, "out.h5")                          # fails if out.h5 exists
h5t.save(result, "out.h5", root="/run2", mode="a")  # add another group to the file
  • mode: "w-"/"x" (default) fail if the file exists, "w" truncates, "a" writes into an existing file or creates one.
  • root: missing intermediate groups are created. A root group that already has members or attributes raises ValidationError unless overwrite=True, which replaces its contents (destructive, and not undone if the write then fails).
  • Attr(serializer=...) is the inverse of converter (e.g. json.dumps). An Enum attribute is stored as its .value; an attribute h5py cannot store (such as a dict) is a ValidationError telling you to add a serializer. A None optional is omitted.
  • A field bound with attrs= writes its undeclared entries back as raw attributes, so load -> save keeps them. Declared names always come from their typed fields.
  • Collection keys must be non-empty, contain no /, and match the field's Glob; otherwise the item would not load back.

Validation-first: the whole record is validated, jaxtyping shapes are checked, and every LazyArray payload is read before the file is opened, so a bad record never touches the target (and saving into the file a LazyArray came from is safe). If the write itself fails, the new file is removed; with mode="a" the group that save created is removed on a best-effort basis.

Field rules

Annotation HDF5 representation Loading behavior
@h5t.group()-decorated class child group (or the root) recursively materialized via record_type(**fields)
@h5t.dataset(data=...)-decorated class child dataset attributes loaded, record_type(**fields) constructed
Annotated[T, Payload("field")] child dataset attributes loaded, T(**fields) constructed
LazyArray (or a subclass) child dataset metadata snapshot, payload read on first access
Annotated[<any dataset field>, Eager()] child dataset complete payload cached during loading
np.ndarray child dataset complete payload loaded as an ndarray
Float[np.ndarray, "n n"], Float[LazyArray, "n _"] (jaxtyping) child dataset dtype and shape checked against the file's metadata before any read; otherwise as np.ndarray / LazyArray
dict[str, T] / Mapping[str, T] child group named by the field one item per child of that group, as a plain dict; T is any group/dataset record, Payload type, LazyArray, np.ndarray or jaxtyping array; filter with Glob
Annotated[T, Attr(...)] attribute converter, then Pydantic validation
scalar or Literal[...] attribute Pydantic validation

Name("stored-name") renames any field kind. Attr is valid only for attributes and never combines with Payload. Eager is valid only for dataset fields; combined with Payload it is allowed only when the payload field is LazyArray (it prefetches the array), since an np.ndarray payload is already eager. A parameterized alias such as numpy.typing.NDArray[np.floating] is accepted wherever np.ndarray is; for NDArray the dtype parameter is not validated (use jaxtyping, below, to check it). Unsupported collection-shaped child annotations raise SchemaError.

Dynamic collections (dict[str, T], Glob)

Real files often hold one group or dataset per sample, with names a schema cannot list. A dict[str, T] or Mapping[str, T] field (no Attr) mirrors the HDF5 tree: it names one child group (Name is honoured), and its value is a plain dict with one item per child of that group, in HDF5 iteration order. Annotated[..., h5t.Glob("chain_?")] keeps only children whose name matches the shell-style pattern (case-sensitive; it may not be empty or contain /). Attributes on the collection's group are ignored.

@h5t.group()
@dataclass
class Run:
    snapshots: dict[str, Snapshot]
    chains: Annotated[dict[str, h5t.LazyArray], h5t.Glob("chain_?")]
    raw: Annotated[dict[str, Snapshot], h5t.Name("Raw Data")] = field(default_factory=dict)

A missing group follows the usual rules (| None, a default or default_factory, else a ValidationError). Keys must be str; items cannot be scalars, optional, nested collections, or carry Name/Attr/Glob. Bare array items share the owner record's jaxtyping dimension bindings; record items bind independently. Layouts with per-sample groups at the file root are not covered yet. A jaxtyping annotation over np.ndarray or LazyArray has its dtype and shape checked by jaxtyping itself, from the dataset's HDF5 metadata and without reading the data or casting it. Float accepts any float width and Float64 only float64 (a file's byte order is ignored); Shaped accepts any dtype, and a custom AbstractDtype subclass works too. The whole shape grammar is jaxtyping's (..., *batch, #n, n+1, ...). A named dimension binds on first use and must agree across every array field of the same record (a nested record starts fresh, and a @h5t.dataset record is its own scope), so fisher: Float[np.ndarray, "n n"] with fiducials: Float[np.ndarray, "n"] rejects a mismatched file as /fiducials: the size of dimension n is 4 which does not equal the existing value of 3. A @h5t.dataset record may declare only attributes plus its one payload field; a @h5t.group record has no such restriction and may reference other schema types, including itself. Declarations compile at the decorator call, so a SchemaError usually surfaces right there. A @h5t.group record that names a record defined later in its module, or references itself, instead compiles on first use; @h5t.dataset has no such deferral, and does not need one -- being attributes-only, it can never forward-reference another schema type.

A record declared inside a function sees that function's locals when the decorator runs, so ordinary function-local records and self-references just work. A deferred @h5t.group record is retried later without them: if it both forward-references a name and uses a function-local type, pass the type explicitly with @h5t.group(localns={"Child": Child}) (@h5t.dataset accepts localns= too).

A record type has no reserved field names: data, path, shape and attrs are all free, because h5t contributes no base class for a field name to shadow.

Every @h5t.dataset/@h5t.group record accepts extras="ignore" (the default) or extras="forbid". A group policy applies to its immediate child and attribute names. A dataset policy applies to its attributes. Nested schemas keep their own policy, while a plain np.ndarray field never checks the dataset's attributes.

Foreign record types (Payload)

@h5t.dataset/@h5t.group are themselves sugar over Payload, h5t's lower-level mechanism for loading a child dataset into a plain record type without decorating it -- useful for a third-party type you cannot add a decorator to:

from dataclasses import dataclass

@dataclass
class Measurement:
    unit: str
    data: np.ndarray                          # eager payload

@h5t.group()
@dataclass
class Result:
    measurement: Annotated[Measurement, h5t.Payload("data")]

Payload("data") names the field holding the payload: annotate it np.ndarray for an eager array, or h5t.LazyArray for h5t's usual detached, lazily-read payload. Every other field of the record type is loaded as a dataset attribute: scalars, Literal[...], Attr(converter=...), defaults, and T | None all behave as they do on a group record. Payload's own extras= (default "ignore") governs the dataset's attributes, since a type you did not decorate cannot carry its own policy. h5t calls the record type's real constructor (__post_init__ runs; an invariant it raises surfaces as ValidationError).

Pass attrs="field" to bind a Mapping-annotated field to the dataset's attrs snapshot -- validated values for declared names, raw for undeclared. It lists every declared name, including ones absent from the file that fell back to None or to a default:

from collections.abc import Mapping
from typing import Any

@dataclass
class Measurement:
    unit: str
    data: np.ndarray
    attrs: Mapping[str, Any]

@h5t.group()
@dataclass
class Result:
    measurement: Annotated[Measurement, h5t.Payload("data", attrs="attrs")]

A foreign record has no path/shape/dtype of its own unless its Payload field is a LazyArray (whose own .path/.shape/.dtype you read through it). An Annotated[T, Payload(...)] use site resolves T's annotations against its module globals and class dict only; @h5t.dataset/@h5t.group additionally capture their own call site's frame, so a function-local record's annotations resolve against function locals too. A Payload field can also name a @h5t.group-decorated type directly.

Snapshot consistency

A loaded model is a snapshot, with one deliberate exception:

  • Group and dataset attributes, dataset shape/dtype/path, eager datasets, and plain arrays reflect the file during loading (h5t.load()).
  • A lazy dataset's first .data/.read() observes the file at that later moment and caches the resulting array permanently.
  • .open() always opens the current file and current dataset. It may therefore observe replacements or fail if the source was changed or deleted.
  • h5t.save copies lazy payloads: an uncached one is read from its source file at save time, without caching it on your record.

CLI

h5t check result.h5 --schema mypackage.schemas:Result --root /results/latest

Exit status is 0 on success, 1 for the first file/schema mismatch, and 2 for import, declaration, usage, or I/O errors.

Development

uv sync
uv run pytest
uv run ty check
uv run ruff check .

Metadata

Release files for h5t 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for h5t 0.4.0
File Size Uploaded Provenance Actions
h5t-0.4.0.tar.gz 27.6 kB With provenance Details for h5t-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for h5t 0.4.0
File Interpreter ABI Platform Actions
h5t-0.4.0-py3-none-any.whl Python 3 none any Details for h5t-0.4.0-py3-none-any.whl

Total release size: 57.6 kB

Release files / h5t-0.4.0.tar.gz

Download URL h5t-0.4.0.tar.gz
Size 27.6 kB
Tags Source
SHA-256 checksum
How to use checksums
24e6f5289f22e6345ea7aeb01e01745327eed7f49f9d1a9f77ce64e4e57d82c5
BLAKE2b-256 checksum
How to use checksums
a605b6996cc6331ae093c29969d573915065dae66d7fad743164963fa7d3fc6c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / h5t-0.4.0-py3-none-any.whl

Download URL h5t-0.4.0-py3-none-any.whl
Size 30.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fec53afe600897ddf42546014467e7094afd5da7358d986d1f0568de0e2f9ba9
BLAKE2b-256 checksum
How to use checksums
b18834361669ef3a959f15c276f1f998aabbaa68bdac3609fc95556e270ae8ad
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Security for h5t 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for h5t-0.4.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page