Skip to main content

BioMAJ process service

Project description

# About

[![PyPI version](https://badge.fury.io/py/biomaj-process.svg)](https://badge.fury.io/py/biomaj-process)

Microservice to manage the process execution of biomaj.

A protobuf interface is available in biomaj_process/message/message_pb2.py to exchange messages between BioMAJ and the download service. Messages go through RabbitMQ (to be installed).

# Protobuf

To compile protobuf, in biomaj_process/message:

protoc –python_out=. message.proto

# Development

flake8 biomaj_process

CONFIG=tests/config.yml python -m pytest -v tests/biomaj_tests.py

# Run

## Message consumer:

export BIOMAJ_CONFIG=path_to_config.yml python bin/biomaj_process_consumer.py

## Web server

If package is installed via pip, you need a file named gunicorn_conf.py containing somehwhere on local server:

def worker_exit(server, worker):

from prometheus_client import multiprocess multiprocess.mark_process_dead(worker.pid)

If you cloned the repository and installed it via python setup.py install, just refer to the gunicorn_conf.py in the cloned repository.

export BIOMAJ_CONFIG=path_to_config.yml rm -rf ..path_to/prometheus-multiproc mkdir -p ..path_to/prometheus-multiproc export prometheus_multiproc_dir=..path_to/prometheus-multiproc gunicorn -c ..path_to/gunicorn_conf.py biomaj_download.biomaj_process_web:app

Web processes should be behind a proxy/load balancer, API base url /api/process

A prometheus /metrics endpoint is exposed on web server

3.0.19

Replace nose by pytest

3.0.18:

Normalize yaml setup with other biomaj components

3.0.17:

Fix yaml load warnings

3.0.16:

In case of proxy comm failure, sleep a few seconds before retry

3.0.15:

Add proc name in logs for waiting and result

3.0.14:
Add env var BIOMAJ_HOST_DATA_DIR to use with docker process executor to specify data.dir location if different between host and container (if BIOMAJ_DATA_DIR not a subdirectory of BIOMAJ_DIR for example)

If variable is not set, container wil mount data.dir between host and container, else will mount BIOMAJ_HOST_DATA_DIR:data.dir

3.0.13:

Fix traefik prefix

3.0.12:

Update pika dependency release Add tags for traefik support

3.0.11:

Catch exception for microservices if remote server is not reachable Rename package for protobuf to avoid collisions

3.0.10:

README doc fixes Add dependency banks additional volumes for Docker

3.0.9:

Allow local_endpoint definition per service Fix volumes for docker with dependency banks

3.0.8:

Fix #1, DRMAA needs colon in front of output and error path (mandatory depending on versions)

3.0.7:

fix case if process desc or type is missing in bank properties

3.0.6:

disable web thread logging

3.0.5:

retry in case of contact failure with proxy

3.0.4:

for docker, restrict directory to bank directory and dependencies add host in prometheus stats

3.0.3:

fix prometheus + gunicorn multi process add consul supervision

3.0.2:

add logging info execute docker processes in biomaj micro service

3.0.1:

bug fixes

3.0.0:

move process management out of biomaj main package

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Built Distribution

biomaj_process-3.0.19-py2.py3-none-any.whl (26.6 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page