Xarray-binfile

Custom Xarray file engine to handle raw binary files

• QA:

• Docs:

• Package:

• Meta:

Overview

Xarray-binfile is a Python package that integrates raw binary files with xarray. You provide the metadata that describes each binary file, and the package exposes those files through xarray datasets and data arrays with optional lazy chunking through Dask.

The package was first designed around binary outputs from the Fortran framework 2DECOMP&FFT and the CFD solver Xcompact3d, which is built on top of it. It is not limited to those ecosystems and can be used with any raw binary convention once the corresponding metadata mapping is provided.

It is aimed at workflows where raw binary files are part of an existing convention, such as simulation outputs or NumPy .tofile() dumps. Once opened, the data can be indexed, reduced, plotted, and computed with the usual xarray and Dask APIs. See why not just numpy.fromfile? in the documentation for what that buys you over numpy.fromfile and numpy.memmap.

Installation

pip install xarray-binfile

Python 3.10 or later is required; xarray, Dask, and NumPy are installed as dependencies. See the installation guide for version pinning and for pyproject.toml, requirements.txt, and conda environment examples.

Quick start

Describe the file convention once, then let xarray and Dask do the rest:

import numpy as np

from xarray_binfile.conventions import Layout, StepIndexedFiles

# Describe the convention once: coordinates, dtype, and filename pattern
convention = StepIndexedFiles(
    layout=Layout(
        {"x": np.linspace(0, 1, 8), "y": np.linspace(0, 1, 4)},
        dtype=np.float32,
    ),
    pattern="{name}-{step:04d}.bin",  # ux-0000.bin, ux-0001.bin, ...
)

# Open every matching file in a folder as one lazy, Dask-backed dataset
ds = convention.open("path/to/data", chunks={"time": 1})

# From here on it is plain xarray
mean = ds["ux"].mean(dim="time").compute()

# Write a derived result back to the same raw layout
mean.rename("ux_mean").expand_dims(time=[0]).binary_engine.to_file(
    convention.writer, "path/to/data"
)

The same files also open through the standard xarray entry points, xr.open_dataset(path, engine="binfile", read_specs_getter=convention.reader) and xr.open_mfdataset(...), so they fit into existing xarray code.

The documentation covers the rest:

• Spec getter protocols: the shipped conventions (step-indexed, time-stamped, static files, folder layouts, stacked variables) and how to describe your own.
• Read lazily with Dask and write derived results.
• Parallel and larger-than-memory workflows, benchmarking on your own data, and practical examples.
• API reference.

Contributing

Bug reports, feature requests, and pull requests are welcome. Start with CONTRIBUTING.md for the development setup with Hatch and the pull request checklist, or read the full contributor guide in the documentation. This project follows the Contributor Covenant.

Copyright and License

© 2025 Felipe N. Schuch. All content is under MIT License.