LbAPCommon

Common utilities for LHCb DPA WP2 related software, including Analysis Productions workflow parsing, validation, and conversion tools.

Features

Workflow Parsing: Parse and render Analysis Productions YAML configuration files
DIRAC Conversion: Convert workflow definitions to DIRAC production requests
CWL Conversion: Convert production requests to Common Workflow Language (CWL) format
Validation & Linting: Validate workflow configurations and check for common issues
Data Models: Pydantic v2 models for workflow and production data structures

Installation

pip install LbAPCommon

For development:

# Using pixi (recommended)
pixi install
pixi run test

# Using conda/mamba
mamba env create --file environment.yaml
conda activate lbaplocal-dev
pip install -e '.[testing]'

Usage

DIRAC Production Request Generation

Convert Analysis Productions info.yaml files to DIRAC production requests. This is the primary CLI used by the Analysis Productions system:

python -m LbAPCommon <production_name> --input info.yaml --output output.json

# With specific AP package version
python -m LbAPCommon my_production --input info.yaml --output result.json --ap-pkg-version v1r0

# Process only a specific job from the workflow
python -m LbAPCommon my_production --input info.yaml --output result.json --only-include job_name

# Dump individual YAML request files for each production
python -m LbAPCommon my_production --input info.yaml --output result.json --dump-requests

# With DIRAC server credentials
python -m LbAPCommon my_production --input info.yaml --output result.json --server-credentials user password

Arguments:

production_name - Name of the production (alphanumeric + underscore, 2-200 chars)
--input - Path to the info.yaml workflow definition file
--output - Path for the output JSON file containing production requests
--ap-pkg-version - AnalysisProductions package version (default: v999999999999)
--only-include - Only process the workflow chain containing this job name
--dump-requests - Also write individual .yaml files for each production request
--server-credentials - DIRAC server credentials (username and password)

The output JSON contains:

rendered_yaml - The rendered workflow YAML after Jinja2 processing
productions - Dictionary of production requests, each containing:
- request - The DIRAC production request in LbAPI format
- input-dataset - Input dataset specification from Bookkeeping query
- dynamic_files - Auto-generated configuration files
- raw-yaml - Original YAML for the jobs in this production

CWL Conversion CLI

Convert DIRAC production request YAML files to CWL workflows:

# List productions in a file
lhcb-production-yaml-to-cwl list_productions production.yaml

# Generate CWL workflow files
lhcb-production-yaml-to-cwl generate production.yaml --output-dir ./cwl-output

# Convert a specific production
lhcb-production-yaml-to-cwl generate production.yaml --production "MyProduction"

Python API

from LbAPCommon import parse_yaml, render_yaml, validate_yaml
from LbAPCommon.dirac_conversion import group_in_to_requests, step_to_production_request
from LbAPCommon.prod_request_to_cwl import fromProductionRequestYAMLToCWL

# Parse and validate workflow YAML
rendered = render_yaml(yaml_content)
jobs_data = parse_yaml(rendered, production_name, wg)
validate_yaml(jobs_data, wg, production_name)

# Group jobs into production requests
for job_names in group_in_to_requests(jobs_data):
    request = step_to_production_request(
        production_name, jobs_data, job_names, input_spec, ap_pkg_version
    )

# Convert DIRAC output to CWL
workflow, inputs, metadata = fromProductionRequestYAMLToCWL(yaml_path)

Development

Running Tests

pixi run test              # Run all tests
pixi run test-dirac        # Run DIRAC conversion tests
pixi run test-cov          # Run with coverage report

Updating Test Fixtures

When CWL output changes, update reference fixtures:

pixi run update-fixtures   # Regenerate reference .cwl files

Code Quality

pixi run pre-commit        # Run all pre-commit hooks (linting, formatting, etc.)

Documentation

Documentation goes in the LbAPDoc project.

Related Projects

LbAPI - REST API server for Analysis Productions
LbAPLocal - Local testing tools
LbAPWeb - Web interface for Analysis Productions

License

GPL-3.0 - See COPYING for details.

Metadata

Release files for LbAPCommon 0.19.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for LbAPCommon 0.19.4
File	Size	Uploaded
lbapcommon-0.19.4.tar.gz	535.0 kB	Jul 28, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for LbAPCommon 0.19.4
File	Interpreter	ABI	Platform	Reset
lbapcommon-0.19.4-py3-none-any.whl 181.0 kB Jul 28, 2026	Python 3	none	any	Details

Total release size: 715.9 kB

Release files / lbapcommon-0.19.4.tar.gz

Download URL	lbapcommon-0.19.4.tar.gz
Size	535.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`1bac4c93a7f090bab4846ab05d89784573074dcd476e90476fa9723eb338157f`
BLAKE2b-256 checksum How to use checksums	`88a158124927fe2f7947a169a3fab8ce3fa7e0a5f9c5903860c2468a50cc4684`
Upload date	Jul 28, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.9.25`

Release files / lbapcommon-0.19.4-py3-none-any.whl

Download URL	lbapcommon-0.19.4-py3-none-any.whl
Size	181.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`a0d9a381ff76971102f8e1249fda0503d1e6d0020e0fe0685b500880bec5eaf6`
BLAKE2b-256 checksum How to use checksums	`b9bed5cea299438046f8c2ec0163653c30ff1e9d13d3af1e45897d4d38f17930`
Upload date	Jul 28, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.9.25`