LbAPCommon

Common utilities for LHCb DPA WP2 related software, including Analysis Productions workflow parsing, validation, and conversion tools.

Features

Workflow Parsing: Parse and render Analysis Productions YAML configuration files
DIRAC Conversion: Convert workflow definitions to DIRAC production requests
CWL Conversion: Convert production requests to Common Workflow Language (CWL) format
Validation & Linting: Validate workflow configurations and check for common issues
Data Models: Pydantic v2 models for workflow and production data structures

Installation

pip install LbAPCommon

For development:

# Using pixi (recommended)
pixi install
pixi run test

# Using conda/mamba
mamba env create --file environment.yaml
conda activate lbaplocal-dev
pip install -e '.[testing]'

Usage

DIRAC Production Request Generation

Convert Analysis Productions info.yaml files to DIRAC production requests. This is the primary CLI used by the Analysis Productions system:

python -m LbAPCommon <production_name> --input info.yaml --output output.json

# With specific AP package version
python -m LbAPCommon my_production --input info.yaml --output result.json --ap-pkg-version v1r0

# Process only a specific job from the workflow
python -m LbAPCommon my_production --input info.yaml --output result.json --only-include job_name

# Dump individual YAML request files for each production
python -m LbAPCommon my_production --input info.yaml --output result.json --dump-requests

# With DIRAC server credentials
python -m LbAPCommon my_production --input info.yaml --output result.json --server-credentials user password

Arguments:

production_name - Name of the production (alphanumeric + underscore, 2-200 chars)
--input - Path to the info.yaml workflow definition file
--output - Path for the output JSON file containing production requests
--ap-pkg-version - AnalysisProductions package version (default: v999999999999)
--only-include - Only process the workflow chain containing this job name
--dump-requests - Also write individual .yaml files for each production request
--server-credentials - DIRAC server credentials (username and password)

The output JSON contains:

rendered_yaml - The rendered workflow YAML after Jinja2 processing
productions - Dictionary of production requests, each containing:
- request - The DIRAC production request in LbAPI format
- input-dataset - Input dataset specification from Bookkeeping query
- dynamic_files - Auto-generated configuration files
- raw-yaml - Original YAML for the jobs in this production

CWL Conversion CLI

Convert DIRAC production request YAML files to CWL workflows:

# List productions in a file
lhcb-production-yaml-to-cwl list_productions production.yaml

# Generate CWL workflow files
lhcb-production-yaml-to-cwl generate production.yaml --output-dir ./cwl-output

# Convert a specific production
lhcb-production-yaml-to-cwl generate production.yaml --production "MyProduction"

Python API

from LbAPCommon import parse_yaml, render_yaml, validate_yaml
from LbAPCommon.dirac_conversion import group_in_to_requests, step_to_production_request
from LbAPCommon.prod_request_to_cwl import fromProductionRequestYAMLToCWL

# Parse and validate workflow YAML
rendered = render_yaml(yaml_content)
jobs_data = parse_yaml(rendered, production_name, wg)
validate_yaml(jobs_data, wg, production_name)

# Group jobs into production requests
for job_names in group_in_to_requests(jobs_data):
    request = step_to_production_request(
        production_name, jobs_data, job_names, input_spec, ap_pkg_version
    )

# Convert DIRAC output to CWL
workflow, inputs, metadata = fromProductionRequestYAMLToCWL(yaml_path)

Development

Running Tests

pixi run test              # Run all tests
pixi run test-dirac        # Run DIRAC conversion tests
pixi run test-cov          # Run with coverage report

Updating Test Fixtures

When CWL output changes, update reference fixtures:

pixi run update-fixtures   # Regenerate reference .cwl files

Code Quality

pixi run pre-commit        # Run all pre-commit hooks (linting, formatting, etc.)

Documentation

Documentation goes in the LbAPDoc project.

Related Projects

LbAPI - REST API server for Analysis Productions
LbAPLocal - Local testing tools
LbAPWeb - Web interface for Analysis Productions

License

GPL-3.0 - See COPYING for details.

Metadata

Release files for LbAPCommon 0.19.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for LbAPCommon 0.19.6
File	Size	Uploaded
lbapcommon-0.19.6.tar.gz	536.4 kB	Aug 26, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for LbAPCommon 0.19.6
File	Interpreter	ABI	Platform	Reset
lbapcommon-0.19.6-py3-none-any.whl 181.5 kB Aug 26, 2026	Python 3	none	any	Details

Total release size: 717.9 kB

Release files / lbapcommon-0.19.6.tar.gz

Download URL	lbapcommon-0.19.6.tar.gz
Size	536.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`a3c3ac4209a9713745663684654c5d3fd9a7b34b855416e0a5e705b4fd9d1f52`
BLAKE2b-256 checksum How to use checksums	`b5cf6c422f0b8b35f373e006d23a93dc951141fa0cca37717b873115b170cf99`
Upload date	Aug 26, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.9.25`

Release files / lbapcommon-0.19.6-py3-none-any.whl

Download URL	lbapcommon-0.19.6-py3-none-any.whl
Size	181.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`dbf35c2513bfa2f7b1b0d622e47dc910df63d8521ed84f70164cf1ffbe1ed4f0`
BLAKE2b-256 checksum How to use checksums	`7c0210f42218944a6720d8ac48ef01ff3f09cfe4a1a26fb549cf8e0d304b36c5`
Upload date	Aug 26, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.9.25`