ngs-pipeline-lib 4.3.0

Common code used by every processes in Bio Pipelines

Introduction

Provide :

• A base set of tools and classes to implement BioInfo algorithms
• Execution context, using the run script

Install

Add the CodeArtifact repository to your pyproject.toml

[[tool.poetry.source]]
name = "codeartifact"
url = "https://pdx-platform-224016688692.d.codeartifact.eu-west-1.amazonaws.com/pypi/pdx-python-libs/simple/"
secondary = true

Then authenticate your local environment to CodeArtifact

export CODEARTIFACT_AUTH_TOKEN=$(aws codeartifact get-authorization-token --domain pdx-platform --query authorizationToken --output text --profile ADX_DEV)
poetry config http-basic.codeartifact aws $CODEARTIFACT_AUTH_TOKEN

Note : The token acquired with AWS is a temporary one. Each time you want to download new packages from the CodeArtifact repository, you may have to re-do the authentication process.

Then, simply add the library to your poetry dependencies.

poetry add ngs-pipeline-lib --source codeartifact

Update

To update to a newer version of the library :

poetry update ngs-pipeline-lib

You may need to update your version constraint in the pyproject.toml file

Get started

Once the library has been installed in your project, you can implement your algorithms by extending the Algorithm class.

If you want to add specific inputs to your Algorithm, extend BaseInputs (which is a Pydantic Model) and use it as the inputs Type. In order to adds outputs, extend BaseOutputs and set the outputs_class class attribute of your algorithm as this class. If you have specific inputs or outputs classes, you should also provide them to Algorithm when subclassing it. You place them between brackets as shown below, this will help your IDE undestand what kind of object it is dealing with, thus improving the autocompletion and the tooltips.

from pydantic import Field

from ngs_pipeline_lib.base.algorithm import Algorithm
from ngs_pipeline_lib.base.inputs import BaseInputs
from ngs_pipeline_lib.base.outputs import BaseOutputs
from ngs_pipeline_lib.base.file import JsonFile

class YourInputs(BaseInputs):
  your_input: str = Field(description="Description")


class YourOutputs(BaseOutputs):
  
  def __init__(self):
        super().__init__()
        self.my_own_output = JsonFile(name="my_json_file")

class YourAlgorithm(Algorithm[YourInputs, YourOutputs]):

    outputs_class = YourOutputs

    def execute_stub(self):
        ...

    def execute_implementation(self):
        print(self.inputs.your_input)
        ...

Then, you can call your Algorithm with the following command

PROCESS_PACKAGE=your_package poetry run ngs-run --sample-id 1

If you want to only create the stub output file, add the --stub parameter.

If you're using a src package, you don't have to specify the PROCESS_PACKAGE variable. Please refer to GUIDELINES doc.

Docker build & push

This library also includes two utilitary scripts to build & push Docker image :

• ngs-build
• ngs-push

Build

This script accepts the following arguments :

Short Arg	Long Arg	Description	Mandatory ?	Default value
-e	--env-file	Path to env file to use	No	.env

This script accepts the following environment variables as parameters

ENV VAR	Description	Mandatory ?	Default value
PROCESS_NAME	Name of the process	Yes	--
IMAGE_PREFIX	Prefix used with the process name to create Docker repo name	No	ngs-pipeline-process-
TAG	Tag of the image to create	No	latest
DOCKERFILE	Relative path to Dockerfile	No	Dockerfile
PIP_REGISTRY_USERNAME	If needed, username to use for pip auth	No	--
PIP_REGISTRY_PASSWORD	If needed, password to use for pip auth	No	--

Note : the docker context used to build is .

Push

This script accepts the following arguments :

Short Arg	Long Arg	Description	Mandatory ?	Default value
-e	--env-file	Path to env file to use	No	.env

This script accepts the following environment variables as parameters

ENV VAR	Description	Mandatory ?	Defaut value
EXTERNAL_REGISTRY_URL	URL of Destination Registry	Yes	--
PROCESS_NAME	Name of the process to push	Yes	--
IMAGE_PREFIX	Prefix used in the process Docker repo name	No	ngs-pipeline-process-
TAG	Tag of the image to create	No	latest
DOCKER_USERNAME	If needed, username to use for docker auth	No	--
DOCKER_PASSWORD	If needed, password to use for docker auth	No	--

Best Practices

When implementing your process, please refer to the guidelines documentation.

License

Shield:

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

© 2023-2025 bioMérieux - all right reserved