BioMAJ process service
Microservice to manage the process execution of biomaj.
A protobuf interface is available in biomaj_process/message/message_pb2.py to exchange messages between BioMAJ and the download service. Messages go through RabbitMQ (to be installed).
To compile protobuf, in biomaj_process/message:
protoc –python_out=. message.proto
## Message consumer: export BIOMAJ_CONFIG=path_to_config.yml python bin/biomaj_process_consumer.py
## Web server
If package is installed via pip, you need a file named gunicorn_conf.py containing somehwhere on local server:
- def worker_exit(server, worker):
- from prometheus_client import multiprocess multiprocess.mark_process_dead(worker.pid)
If you cloned the repository and installed it via python setup.py install, just refer to the gunicorn_conf.py in the cloned repository.
export BIOMAJ_CONFIG=path_to_config.yml rm -rf ..path_to/prometheus-multiproc mkdir -p ..path_to/prometheus-multiproc export prometheus_multiproc_dir=..path_to/prometheus-multiproc gunicorn -c ..path_to/gunicorn_conf.py biomaj_download.biomaj_process_web:app
Web processes should be behind a proxy/load balancer, API base url /api/process
A prometheus /metrics endpoint is exposed on web server
- Catch exception for microservices if remote server is not reachable Rename package for protobuf to avoid collisions
- README doc fixes Add dependency banks additional volumes for Docker
- Allow local_endpoint definition per service Fix volumes for docker with dependency banks
- Fix #1, DRMAA needs colon in front of output and error path (mandatory depending on versions)
- fix case if process desc or type is missing in bank properties
- disable web thread logging
- retry in case of contact failure with proxy
- for docker, restrict directory to bank directory and dependencies add host in prometheus stats
- fix prometheus + gunicorn multi process add consul supervision
- add logging info execute docker processes in biomaj micro service
- bug fixes
- move process management out of biomaj main package
Release history Release notifications
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size & hash SHA256 hash help||File type||Python version||Upload date|
|biomaj_process-3.0.11-py2.py3-none-any.whl (18.9 kB) Copy SHA256 hash SHA256||Wheel||py2.py3||Aug 22, 2018|