digout 0.1.3

Pipeline framework to dump analysis-ready data from LHCb grid-based files

digout is a Python library purpose-built to execute the multi-stage workflow of converting raw LHCb DIGI files into analysis-ready parquet dataframes of particles and hits.

To manage this process in a scalable and reproducible manner, it implements a workflow framework organized around configurable steps (e.g., digi2root, root2df). The framework operates on a two-phase execution model: a stream phase runs once to prepare the dataset from a bookkeeping path, and a chunk phase processes each input file in parallel. This parallel execution is managed by swappable schedulers (such as local for local processing or htcondor for cluster submission), with the entire workflow being defined through YAML configuration files to ensure complete reproducibility.

Resources

Link	Description
📖 Full Documentation	The complete guide to installation, configuration, and concepts.
🚀 Quickstart Guide	The fastest way to get a working example running.
💡 Contributing Guide	Learn how to set up a development environment and contribute to the project.
🐛 Report a Bug	Found an issue? Let us know by creating a bug report.
📜 Changelog	See the latest changes from the release page

Core Features

• Automated Metadata Discovery: Automatically queries the LHCb bookkeeping system to retrieve necessary metadata (dddb_tag, conddb_tag, etc.), eliminating manual lookup.
• Scalable Parallel Processing: Built-in support for processing large datasets in parallel on a local machine or on a distributed cluster like HTCondor.
• Configuration-Driven and Reproducible: Define your entire workflow in YAML files. digout saves the final, resolved configuration for every run, ensuring any result can be reproduced.
• Idempotent Execution: Automatically detects and skips steps that have already been completed.
• Extensible Architecture: Easily define new steps or schedulers.

Main Workflows

• DIGI to DataFrame Conversion: Produce analysis-ready parquet dataframes from LHCb DIGI files. The available output dataframes are detailed on the DataFrames Page.

• DIGI to MDF Conversion: Convert LHCb DIGI files into the .mdf format required as input for the Allen framework.