Skip to main content

F9 Columnar

Ruff python pytorch License pypi DOI

A PyTorch-based library for processing event data in high energy physics for ML applications.

Project description

F9Columnar is a Python library designed to streamline the processing of physics datasets (ROOT, HDF5, parquet) for machine learning. It is built on top of PyTorch, Uproot and Awkward Arrays. It provides a modular framework for efficient data loading, transformation, and analysis in high-energy physics research.

The library bridges the gap between traditional high-energy physics data formats and modern machine learning workflows. It enables physicists and data scientists to:

  • Process large datasets in many formats using PyTorch DataLoaders
  • Apply event selections, calculate variables, and create histograms
  • Convert data to piles, HDF5 files with events prepared for fast random-access reads during ML training

The library is designed to be modular and extensible, allowing users to easily add new processors and functionality as needed. A common workflow is illustrated in the diagram below:

F9Columnar

Check out the PyHEP 2025 talk for a good overview of what the library can do.

Setup

The library comes without any PyTorch dependencies by default to allow flexible installations. You can install it with or without PyTorch GPU support depending on your needs.

Install with PyTorch for GPU

pip install f9columnar
pip install torch

Install with PyTorch for CPU

pip install f9columnar
pip install torch --index-url https://download.pytorch.org/whl/cpu

Install for development

git clone https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar.git
cd F9Columnar
uv sync

Run the tests with uv run pytest.

Getting started

The examples folder walks through the library step by step. The examples are self-contained, you do not need real data. Example 1 generates dummy ROOT files that the other examples use. The last example is the exception, it shows the grid workflow and needs a grid environment to run.

  1. Making dummy ROOT inputs
  2. Basic data loading and histogramming with awkward arrays
  3. Data loading and processing with a processor graph
  4. Converting ROOT files into HDF5 piles used for ML training, in one set of piles and in k folds
  5. One ML training loader from two sources
  6. A Lightning training loop over partitioned ROOT files and over HDF5 piles
  7. Inference on the trained model with the inference server
  8. Feature scaling with fitted scalers as torch modules
  9. Rucio ntuples and batch processing on the Vega HPC with aCT

Projects using F9Columnar

  • SeeSawML: multi-dataset, multi-modal and multi-task ML training framework.

Citation

If you found F9Columnar useful, please consider citing its Zenodo record 10.5281/zenodo.19312298.

You can find an example bibtex entry below.

@software{F9Columnar,
  author       = {Gavranovic, Jan},
  title        = {F9Columnar: A PyTorch-Based Library for High Energy Physics Data Processing and Analysis},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.19312298},
  url          = {https://doi.org/10.5281/zenodo.19312298},
}

Metadata

Release files for f9columnar 0.9.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for f9columnar 0.9.1
File Size Uploaded Provenance Actions
f9columnar-0.9.1.tar.gz 244.9 kB Without provenance Details for f9columnar-0.9.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for f9columnar 0.9.1
File Interpreter ABI Platform Actions
f9columnar-0.9.1-py3-none-any.whl Python 3 none any Details for f9columnar-0.9.1-py3-none-any.whl

Total release size: 519.1 kB

Release files / f9columnar-0.9.1.tar.gz

Download URL f9columnar-0.9.1.tar.gz
Size 244.9 kB
Tags Source
SHA-256 checksum
How to use checksums
a0413271037005ac9b14546457f08f70b0f66280f106cf53f8801c76b518813c
BLAKE2b-256 checksum
How to use checksums
49eea3d59375013be2e2a30a6c0c8c09a0c773cf26ef243ffd1c96e38e9e9273
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / f9columnar-0.9.1-py3-none-any.whl

Download URL f9columnar-0.9.1-py3-none-any.whl
Size 274.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2f2d0ef813fb8c173017cd8d9823dd1e63e38633ec8cfbcdcefbf1ff3c7d80f1
BLAKE2b-256 checksum
How to use checksums
a06ce22b35049273fa21ee9badeff5a41f3ba1400f5e66ab84c1c46ef756c404
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for f9columnar 0.9.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.9.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page