rowver (Python)

Read .rvr files in Python, through the same Rust reader as the rvr command, which this package also installs: pip install rowver (Python 3.11 or later).

import rowver

f = rowver.open("higgs_tautau.rvr")
f                         # <rowver.File 'higgs_tautau.rvr': 2248 rows, 11 columns, 1 derived, 2 samples>
f.samples, f.columns      # names, in the file's order
f["MET_pt"]               # NumPy array, one value per row (no copy, read-only: .copy() to change it)
f["Muon_pt"]              # Awkward array, a list per row
f["sample"], f["entry"]   # each row's sample index, and its entry in its file's tree
f["MET_pt"][f.rows_of("GluGluToHToTauTau")]  # one sample's rows: a slice, as a sample's rows come together
len(f), "MET_pt" in f     # rows; whether it has a column of that name (sample and entry too)
f.keys()                  # every name f[...] reads: sample, entry, the columns, the derived ones, the sample columns
f.show()                  # the same, a line each with its kind (f.kinds) and a derived column's expression
f["xsec"]                 # a sample column (one value per sample), as its sample's value in every row
f.path, f.header          # where it is; its JSON header, as a dict
f.derived                 # the derived columns: kept as expressions, worked out when first asked for
f["mean_jet_pt"]          # a derived column, read as any other, checked against what the file records
f.slice("MET_pt", 100, 200)  # rows 100 to 199 alone: only the chunks of the file that hold them read
f.slice("MET_pt", -100)      # the last 100; f.slice(name, slice(a, b, step)) reads as a list's slice
f.labels                  # each sample's label, what a plot shows
f.sample_columns          # per-sample values: events_read, events_total, metadata
f.recipe                  # the recipe.toml that made it
f.study                   # {"recipe.toml": …, "samples.toml": …, "samples.lock": …}
f.signed_by               # the public key that signed it, or None
f.complete                # whether it holds every entry its press chose (no holes, not empty)
f.empty                   # whether rvr empty let its columns go (rvr refill presses them again)
f.holes                   # what it could not read: [{"sample", "file", "entries", "kind", "why"}, …],
                          # entries (first, last) as the file records them, last not among them

f.arrays()                # every column as one Awkward record array, sample and entry first
f.arrays(["MET_pt"])      # sample, entry, then the columns named
f.arrays(["MET_pt"], sample="GluGluToHToTauTau")  # one sample's rows alone, only its chunks read (to_* too)
f.to_arrow()              # a pyarrow Table (list columns as Arrow lists), sample and entry first
f.to_pandas()             # a pandas DataFrame, sample (a categorical) and entry first
f.to_parquet("out.parquet")

with rowver.open("higgs_tautau.rvr") as f:  # f.close() at the end: what it read stays,
    met = f["MET_pt"]                        # its header still reads, a column then raises Error

The examples work with the columns of a file, and plot them with matplotlib, hist and ROOT.

A column is the same array each time it is asked for while anything holds it; the File keeps none in memory itself, so memory is what your own variables hold. A File pickles as its path (for multiprocessing, dask and the like), and a name it has no column of raises KeyError naming the nearest it has. Every column asked for at once (f.read([...]), arrays, to_*) is read in parallel. Only NumPy is required; Awkward, pyarrow and pandas are used when installed (pip install "rowver[all]"). A file whose header does not check out raises rowver.Error, a path that is not there FileNotFoundError; a file changed while it is read raises rowver.Error saying so. rowver.validate(path) checks every array and works out every derived column, as rvr validate does, and gives (rows, columns). Reading a column checks its chunk table and every compressed frame's own checksum, and leaves each chunk's SHA-256 to rowver.validate. An open File holds one open file, through which it reads, so a file renamed over it meanwhile is read as it was; dropping the File (f.close(), or del f) lets it go (a system allows a few hundred open files: 256 in a macOS shell). These, rowver.open and rowver.File are the whole package; every other name in it begins with _.

Building

From this folder, with Rust 1.91 or later: maturin develop --release installs it into the active Python; maturin build --release writes a wheel (one for every Python from 3.11). It builds on Linux and macOS.

MIT licensed.