Build large-scale task workflows using luigi, remote job submission, remote targets, and environment
Project description
Note: This project is currently under development. Version 0.1.0 will be the first, fully documented alpha release.
Use law to build complex and large-scale task workflows. It is build on top of luigi and adds abstractions for run locations, storage locations and software environments. Law strictly disentangles these building blocks and ensures they remain interchangeable and resource-opportunistic.
Key features:
CLI with auto-completion and interactive status and dependency inspection.
- Remote targets with automatic retries and local caching
WebDAV, HTTP, Dropbox, SFTP, all WLCG protocols (srm, xrootd, rfio, dcap, gsiftp, …)
- Automatic submission to batch systems from within tasks
HTCondor, LSF, gLite, ARC
- Environment sandboxing, configurable on task level
Docker, Singularity, Sub-Shells
First steps
Installation and Dependencies
Install via pip:
pip install law
This command also installs luigi and six.
Remote targets also require gfal2 and gfal2-python (optional, also via pip) to be installed.
Examples
loremipsum: The hello world of law.
workflows: Law workflows.
dropbox_targets: Working with targets that are stored on Dropbox.
wlcg_targets: Working with targets that are stored on WLCG storage elements (dCache, EOS, …). TODO.
htcondor_at_vispa: HTCondor workflows at the VISPA service.
htcondor_at_cern: HTCondor workflows at the CERN batch infrastructure.
grid_at_cern: Workflows that run jobs and store data on the WLCG.
lsf_at_cern: LSF workflows at the CERN batch infrastructure.
docker_sandboxes: Environment sandboxing using Docker. TODO.
singularity_sandboxes: Environment sandboxing using Singularity. TODO.
subshell_sandboxes: Environment sandboxing using Subshells. TODO.
parallel_optimization: Parallel optimization using scikit optimize.
CMS Single Top Analysis: Simple physics analysis using law.
Auto Completion
bash
source "$( law completion )"
zsh
autoload -Uz compinstall && compinstall
autoload -Uz bashcompinit && bashcompinit
source "$( law completion )"
Development
Source hosted at GitHub
Report issues, questions, feature requests on GitHub Issues
Contributors
Marcel R. (author)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.