dpypeline 0.1.0b2

Program for creating data pipelines triggered by file creation events.

dpypeline

Program for creating data pipelines triggered by file creation events.

Version

0.1.0-beta.1

Python enviroment setup

To utilise this package, it should be installed within a dedicated Conda environment. You can create this environment using the following command:

conda create --name <environment_name> python=3.10

To activate the conda environment use:

conda activate <environment_name>

Alternatively, use virtualenv to setup and activate the environment:

python -m venv <environment_name>
source <envionment_name>/bin/activate

Installation

1. Clone the repository:

git clone [e-mail]:NOC-OI/dpyepline.git

2. Navigate to the package directory:

After cloning the repository, navigate to the root directory of the package.

3. Install in editable mode:

To install dpypeline in editable mode, execute the following comman from the root directory:

pip install -e .

This command will install the library in editable mode, allowing you to make changes to the code if needed.

4. Alternative installation methods:

• Install from the GitHub repository directly:

pip install git+https://github.com/NOC-OI/dpypeline.git@main#egg=dpypeline

• Install from the PyPI repository:

pip install dpypeline

Unit tests

Run tests using pytest in the main directory:

pip install pytest
pytest

Examples

Python scripts

Examples of Python scripts explaining how to use this package can be found in the examples directory.

Command line interface (CLI)

TODO

Environment variables

There are a few environment variables that need to be set so that the application can run correctly:

• CACHE_DIR: Path to the cache directory.

Software Workflow Overview

Pipeline architectures

Thread-based pipeline

In the thread-based pipeline, Akita enqueues events into an in-memory queue. These events are subsequently consumed by ConsumerSerial, which generates jobs for sequential execution within the ThreadPipeline (an alias for BasicPipeline).

Parallel pipeline

In the parallel pipeline, Akita enqueues events into an in-memory queue. These events are then consumed by ConsumerParallel, which generates futures that are executed concurrently by multiple Dask workers.

Filesystems

Credentials to access the object store (.json file)

From inside JASMIN:

{
    "token": <Token generated using the Caringo Portal>,
    "secret": <Secret generated using the Caringo Portal>,
    "endpoint_url": "https://noc-msm-o.s3.jc.rl.ac.uk"
}

External access, from outside JASMIN:

{
    "token": <Token generated using the Caringo portal>,
    "secret": <Secret generated using the Caringo portal>,
    "endpoint_url": "https://noc-msm-o.s3-ext.jc.rl.ac.uk"
}