F9 Columnar
A PyTorch-based library for processing event data in high energy physics for ML applications.
Project description
F9Columnar is a Python library designed to streamline the processing of physics datasets (ROOT, HDF5, parquet) for machine learning. It is built on top of PyTorch, Uproot and Awkward Arrays. It provides a modular framework for efficient data loading, transformation, and analysis in high-energy physics research.
The library bridges the gap between traditional high-energy physics data formats and modern machine learning workflows. It enables physicists and data scientists to:
- Process large datasets in many formats using PyTorch DataLoaders
- Apply event selections, calculate variables, and create histograms
- Convert data to piles, HDF5 files with events prepared for fast random-access reads during ML training
The library is designed to be modular and extensible, allowing users to easily add new processors and functionality as needed. A common workflow is illustrated in the diagram below:
Check out the PyHEP 2025 talk for a good overview of what the library can do.
Setup
The library comes without any PyTorch dependencies by default to allow flexible installations. You can install it with or without PyTorch GPU support depending on your needs.
Install with PyTorch for GPU
pip install f9columnar
pip install torch
Install with PyTorch for CPU
pip install f9columnar
pip install torch --index-url https://download.pytorch.org/whl/cpu
Install for development
git clone https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar.git
cd F9Columnar
uv sync
Run the tests with uv run pytest.
Getting started
The examples folder walks through the library step by step. The examples are self-contained, you do not need real data. Example 1 generates dummy ROOT files that the other examples use. The last example is the exception, it shows the grid workflow and needs a grid environment to run.
- Making dummy ROOT inputs
- Basic data loading and histogramming with awkward arrays
- Data loading and processing with a processor graph
- Converting ROOT files into HDF5 piles used for ML training
- One ML training loader from two sources
- A Lightning training loop over partitioned ROOT files and over HDF5 piles
- Inference on the trained model with the inference server
- Feature scaling with fitted scalers as torch modules
- Rucio ntuples and batch processing on the Vega HPC with aCT
Projects using F9Columnar
- SeeSawML: multi-dataset, multi-modal and multi-task ML training framework.
Citation
If you found F9Columnar useful, please consider citing its Zenodo record 10.5281/zenodo.19312298.
You can find an example bibtex entry below.
@software{F9Columnar,
author = {Gavranovic, Jan},
title = {F9Columnar: A PyTorch-Based Library for High Energy Physics Data Processing and Analysis},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.19312298},
url = {https://doi.org/10.5281/zenodo.19312298},
}
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file f9columnar-0.8.0.tar.gz.
File metadata
- Download URL: f9columnar-0.8.0.tar.gz
- Upload date:
- Size: 233.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.12.6 {"installer":{"name":"uv","version":"0.12.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
be51a40bd59eb71092a4b3302ed4ed7d399fb3f04e1cc2b1fb89b74cb5247fd3
|
|
| MD5 |
565855fb33345ae064138a6696ac6aef
|
|
| BLAKE2b-256 |
0e0c326776ef9e60d177f1c2f1c9eddd0edfb85736ff804c3c0d8ef6c684f64c
|
File details
Details for the file f9columnar-0.8.0-py3-none-any.whl.
File metadata
- Download URL: f9columnar-0.8.0-py3-none-any.whl
- Upload date:
- Size: 262.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.12.6 {"installer":{"name":"uv","version":"0.12.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e56c0aa7b6c0b437315965620663157a9ae8f9cc8c9ff4cc0cde75214ccea9bd
|
|
| MD5 |
1b5b00a4493e9db67b233300f0a4431f
|
|
| BLAKE2b-256 |
8b4340b0b7ef627b0b4ec70ef2a604c31e38818c06c64c919323375096d97b89
|
