LbAPCommon 0.15.11

Common utilities used by LHCb DPA WP2 related software

LbAPCommon

Common utilities for LHCb DPA WP2 related software, including Analysis Productions workflow parsing, validation, and conversion tools.

Features

• Workflow Parsing: Parse and render Analysis Productions YAML configuration files
• DIRAC Conversion: Convert workflow definitions to DIRAC production requests
• CWL Conversion: Convert production requests to Common Workflow Language (CWL) format
• Validation & Linting: Validate workflow configurations and check for common issues
• Data Models: Pydantic v2 models for workflow and production data structures

Installation

pip install LbAPCommon

For development:

# Using pixi (recommended)
pixi install
pixi run test

# Using conda/mamba
mamba env create --file environment.yaml
conda activate lbaplocal-dev
pip install -e '.[testing]'

Usage

DIRAC Production Request Generation

Convert Analysis Productions info.yaml files to DIRAC production requests. This is the primary CLI used by the Analysis Productions system:

python -m LbAPCommon <production_name> --input info.yaml --output output.json

# With specific AP package version
python -m LbAPCommon my_production --input info.yaml --output result.json --ap-pkg-version v1r0

# Process only a specific job from the workflow
python -m LbAPCommon my_production --input info.yaml --output result.json --only-include job_name

# Dump individual YAML request files for each production
python -m LbAPCommon my_production --input info.yaml --output result.json --dump-requests

# With DIRAC server credentials
python -m LbAPCommon my_production --input info.yaml --output result.json --server-credentials user password

Arguments:

• production_name - Name of the production (alphanumeric + underscore, 2-200 chars)
• --input - Path to the info.yaml workflow definition file
• --output - Path for the output JSON file containing production requests
• --ap-pkg-version - AnalysisProductions package version (default: v999999999999)
• --only-include - Only process the workflow chain containing this job name
• --dump-requests - Also write individual .yaml files for each production request
• --server-credentials - DIRAC server credentials (username and password)

The output JSON contains:

• rendered_yaml - The rendered workflow YAML after Jinja2 processing
• productions - Dictionary of production requests, each containing:
  • request - The DIRAC production request in LbAPI format
  • input-dataset - Input dataset specification from Bookkeeping query
  • dynamic_files - Auto-generated configuration files
  • raw-yaml - Original YAML for the jobs in this production

CWL Conversion CLI

Convert DIRAC production request YAML files to CWL workflows:

# List productions in a file
lhcb-production-yaml-to-cwl list_productions production.yaml

# Generate CWL workflow files
lhcb-production-yaml-to-cwl generate production.yaml --output-dir ./cwl-output

# Convert a specific production
lhcb-production-yaml-to-cwl generate production.yaml --production "MyProduction"

Python API

from LbAPCommon import parse_yaml, render_yaml, validate_yaml
from LbAPCommon.dirac_conversion import group_in_to_requests, step_to_production_request
from LbAPCommon.prod_request_to_cwl import fromProductionRequestYAMLToCWL

# Parse and validate workflow YAML
rendered = render_yaml(yaml_content)
jobs_data = parse_yaml(rendered, production_name, wg)
validate_yaml(jobs_data, wg, production_name)

# Group jobs into production requests
for job_names in group_in_to_requests(jobs_data):
    request = step_to_production_request(
        production_name, jobs_data, job_names, input_spec, ap_pkg_version
    )

# Convert DIRAC output to CWL
workflow, inputs, metadata = fromProductionRequestYAMLToCWL(yaml_path)

Development

Running Tests

pixi run test              # Run all tests
pixi run test-dirac        # Run DIRAC conversion tests
pixi run test-cov          # Run with coverage report

Updating Test Fixtures

When CWL output changes, update reference fixtures:

pixi run update-fixtures   # Regenerate reference .cwl files

Code Quality

pixi run pre-commit        # Run all pre-commit hooks (linting, formatting, etc.)

Documentation

Documentation goes in the LbAPDoc project.

Related Projects

• LbAPI - REST API server for Analysis Productions
• LbAPLocal - Local testing tools
• LbAPWeb - Web interface for Analysis Productions

License

GPL-3.0 - See COPYING for details.