LbAPCommon
Common utilities for LHCb DPA WP2 related software, including Analysis Productions workflow parsing, validation, and conversion tools.
Features
- Workflow Parsing: Parse and render Analysis Productions YAML configuration files
- DIRAC Conversion: Convert workflow definitions to DIRAC production requests
- CWL Conversion: Convert production requests to Common Workflow Language (CWL) format
- Validation & Linting: Validate workflow configurations and check for common issues
- Data Models: Pydantic v2 models for workflow and production data structures
Installation
pip install LbAPCommon
For development:
# Using pixi (recommended)
pixi install
pixi run test
# Using conda/mamba
mamba env create --file environment.yaml
conda activate lbaplocal-dev
pip install -e '.[testing]'
Usage
DIRAC Production Request Generation
Convert Analysis Productions info.yaml files to DIRAC production requests. This is the primary CLI used by the Analysis Productions system:
python -m LbAPCommon <production_name> --input info.yaml --output output.json
# With specific AP package version
python -m LbAPCommon my_production --input info.yaml --output result.json --ap-pkg-version v1r0
# Process only a specific job from the workflow
python -m LbAPCommon my_production --input info.yaml --output result.json --only-include job_name
# Dump individual YAML request files for each production
python -m LbAPCommon my_production --input info.yaml --output result.json --dump-requests
# With DIRAC server credentials
python -m LbAPCommon my_production --input info.yaml --output result.json --server-credentials user password
Arguments:
production_name- Name of the production (alphanumeric + underscore, 2-200 chars)--input- Path to theinfo.yamlworkflow definition file--output- Path for the output JSON file containing production requests--ap-pkg-version- AnalysisProductions package version (default:v999999999999)--only-include- Only process the workflow chain containing this job name--dump-requests- Also write individual.yamlfiles for each production request--server-credentials- DIRAC server credentials (username and password)
The output JSON contains:
rendered_yaml- The rendered workflow YAML after Jinja2 processingproductions- Dictionary of production requests, each containing:request- The DIRAC production request in LbAPI formatinput-dataset- Input dataset specification from Bookkeeping querydynamic_files- Auto-generated configuration filesraw-yaml- Original YAML for the jobs in this production
CWL Conversion CLI
Convert DIRAC production request YAML files to CWL workflows:
# List productions in a file
lhcb-production-yaml-to-cwl list_productions production.yaml
# Generate CWL workflow files
lhcb-production-yaml-to-cwl generate production.yaml --output-dir ./cwl-output
# Convert a specific production
lhcb-production-yaml-to-cwl generate production.yaml --production "MyProduction"
Python API
from LbAPCommon import parse_yaml, render_yaml, validate_yaml
from LbAPCommon.dirac_conversion import group_in_to_requests, step_to_production_request
from LbAPCommon.prod_request_to_cwl import fromProductionRequestYAMLToCWL
# Parse and validate workflow YAML
rendered = render_yaml(yaml_content)
jobs_data = parse_yaml(rendered, production_name, wg)
validate_yaml(jobs_data, wg, production_name)
# Group jobs into production requests
for job_names in group_in_to_requests(jobs_data):
request = step_to_production_request(
production_name, jobs_data, job_names, input_spec, ap_pkg_version
)
# Convert DIRAC output to CWL
workflow, inputs, metadata = fromProductionRequestYAMLToCWL(yaml_path)
Development
Running Tests
pixi run test # Run all tests
pixi run test-dirac # Run DIRAC conversion tests
pixi run test-cov # Run with coverage report
Updating Test Fixtures
When CWL output changes, update reference fixtures:
pixi run update-fixtures # Regenerate reference .cwl files
Code Quality
pixi run pre-commit # Run all pre-commit hooks (linting, formatting, etc.)
Documentation
Documentation goes in the LbAPDoc project.
Related Projects
- LbAPI - REST API server for Analysis Productions
- LbAPLocal - Local testing tools
- LbAPWeb - Web interface for Analysis Productions
License
GPL-3.0 - See COPYING for details.
Metadata
Release files for LbAPCommon 0.22.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| lbapcommon-0.22.0.tar.gz | 541.1 kB | Without provenance | Details for lbapcommon-0.22.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| lbapcommon-0.22.0-py3-none-any.whl | Python 3 | none | any | Details for lbapcommon-0.22.0-py3-none-any.whl |
Total release size: 724.3 kB
Release files / lbapcommon-0.22.0.tar.gz
| Download URL | lbapcommon-0.22.0.tar.gz |
|---|---|
| Size | 541.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5c58a1ec2498aca6ea8a3383e98515831fb1a2d3966a713e161d4bfd2226342d
|
|
BLAKE2b-256 checksum How to use checksums |
cb12c38276464e95fead443bb7e70e771206e19ec167f4bc0756631f23edfd16
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Release files / lbapcommon-0.22.0-py3-none-any.whl
| Download URL | lbapcommon-0.22.0-py3-none-any.whl |
|---|---|
| Size | 183.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e6a783eafdc57d2dd04de65528f3f2b9de4efe91d114d02b958a15b62a4121c6
|
|
BLAKE2b-256 checksum How to use checksums |
e3dc6f61c3c62defd3362f9427dbb70db9823cec096a0c6035b39335656063dd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Security for LbAPCommon 0.22.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.