Automation of resource creation to be able
Project description
tw-pywrap
tw-pywrap
is a Python wrapper for the Nextflow Tower CLI. It can be leveraged to automate the creation of all of the entities in Nextflow Tower via a simple configuration file in YAML format.
The key features are:
- Simple configuration: All of the command-line options available when using the Nextflow Tower CLI can be defined in simple YAML format.
- Infrastructure as Code: Enable users to manage and provision their infrastructure specifications.
- Automation: End-to-end creation of entities within Nextflow Tower, all the way from adding an Organization to launching pipeline(s) within that Organization.
Prerequisites
You will need to have an account on Nextflow Tower (see Plans and pricing).
1. Dependencies
tw-pywrap
requires the following dependencies:
Alternatively, you can install the dependencies via Conda by downloading and using the Conda environment file that has been supplied in this repository:
conda env create -f environment.yml
conda activate tw-pywrap
2. Installation
The scripts in this repository will eventually be packaged via pip
, but for now you can install the latest development version using the command below:
pip install git+https://github.com/seqeralabs/tw-pywrap.git@main
You can force overwrite the installation to use the latest changes with the command below:
pip install --upgrade --force-reinstall git+https://github.com/seqeralabs/tw-pywrap.git@main
3. Configuration
Create a Tower access token using the Nextflow Tower web interface via the Your Tokens page in your profile.
tw-pywrap
reads this token from the environment variable TOWER_ACCESS_TOKEN
. Please export it into your terminal as shown below:
export TOWER_ACCESS_TOKEN=<your access token>
Quick start
You must provide a YAML file that defines the options for each of the entities you would like to create in Nextflow Tower.
You will need to have an account on Nextflow Tower (see Plans and pricing). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.
Launch via YAML
-
Create a YAML file called
hello-world-config.yml
with the contents below, and customise the<YOUR_WORKSPACE>
and<YOUR_COMPUTE_ENVIRONMENT>'
entries as required:launch: - name: 'hello-world' # Workflow name workspace: '<YOUR_WORKSPACE>' # Workspace name compute-env: '<YOUR_COMPUTE_ENVIRONMENT>' # Compute environment revision: 'master' # Pipeline revision pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL
-
Launch the pipeline with
tw-pywrap
:tw-pywrap hello-world-config.yml
-
Login to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!
Launch via a Python script
You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Tower CLI by leveraging the flexibility and customisation options available in Python.
-
Download the
launch_hello_world.py
Python script and customise the<YOUR_WORKSPACE>
and<YOUR_COMPUTE_ENVIRONMENT>'
entries as required. -
Launch the pipeline with
tw-pywrap
:python launch_hello_world.py
-
Login to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!
Real world example
Please see tw-pywrap-e2e.yml
for an end-to-end example that highlights how you can use tw-pywrap
to create everything sequentially in Nextflow Tower all the way from creating a new Organization to launching a pipeline.
Templates
We have provided template YAML files for each of the entities that can be created on Tower. These can be found in the templates/
directory and should form a good starting point for you to add your own customization:
- organizations.yml
- teams.yml
- workspaces.yml
- participants.yml
- credentials.yml
- secrets.yml
- compute-envs.yml
- actions.yml
- datasets.yml
- pipelines.yml
- launch.yml
Contributions and Support
If you would like to contribute to tw-pywrap
, please see the contributing guidelines.
For further information or help, please don't hesitate to create an issue in this repository.
Credits
tw-pywrap
was written by Esha Joshi, Adam Talbot and Harshil Patel from the Scientific Development Team at Seqera Labs.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tw_pywrap-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 34864d9aa997659a43629a332469deeef086a909f339f6f1c95335259ff8f74d |
|
MD5 | 1ab3925f45a8e7a04c621fb1184c3d48 |
|
BLAKE2b-256 | 5c2118213e11122c71b8104c4a22e769faac8b49171f5ca1e6dd780b00a3ca3d |