Data Analytics Software Framework

python module wrapper for the data analytics software framework DASF

Abstract

DASF: Messaging Python is part of the Data Analytics Software Framework (DASF, https://codebase.helmholtz.cloud/dasf), developed at the GFZ German Research Centre for Geosciences (https://www.gfz-potsdam.de). It is funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/).

DASF: Messaging Python is a RPC (remote procedure call) wrapper library for the python programming language. As part of the data analytics software framework DASF, it implements the DASF RPC messaging protocol. This message broker based RPC implementation supports the integration of algorithms and methods implemented in python in a distributed environment. It utilizes pydantic (https://pydantic-docs.helpmanual.io/) for data and model validation using python type annotations. DASF distributes messages via a central message broker. Currently we support a self-developed message broker called dasf-broker-django, as well as an ‘off-the-shelf’ solution called Apache Pulsar. (also see: Message Broker)

Documentation

see: https://dasf.readthedocs.io/en/latest/

Installation

Install this package in a dedicated python environment via

python -m venv venv
source venv/bin/activate
pip install demessaging

To use this in a development setup, clone the source code from gitlab, start the development server and make your changes::

git clone https://codebase.helmholtz.cloud/dasf/dasf-messaging-python
cd dasf-messaging-python
python -m venv venv
source venv/bin/activate
make dev-install

More detailed installation instructions my be found in the docs.

Technical note

This package has been generated from the template https://codebase.helmholtz.cloud/hcdc/software-templates/python-package-template.git.

See the template repository for instructions on how to update the skeleton for this package.

Source Code Examples

see: https://codebase.helmholtz.cloud/dasf/dasf-messaging-python/-/blob/master/ExampleMessageConsumer.py

• generate the counterpart via python ExampleMessageConsumer.py generate > ExampleMessageProducerGen.py
• call the consumer module via the generated producer, see https://codebase.helmholtz.cloud/dasf/dasf-messaging-python/-/blob/master/ExampleMessageProducer.py

Recommended Software Citation

Eggert, Daniel; Sommer, Philipp; Dransch, Doris (2021): DASF: Messaging Python: A python RPC wrapper for the data analytics software framework. GFZ Data Services. https://doi.org/10.5880/GFZ.1.4.2021.005

License information

Copyright © 2019-2024 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences

Code files in this repository are licensed under the Apache-2.0, if not stated otherwise in the file.

Documentation files in this repository are licensed under CC-BY-4.0, if not stated otherwise in the file.

Supplementary and configuration files in this repository are licensed under CC0-1.0, if not stated otherwise in the file.

Please check the header of the individual files for more detailed information.

License management

License management is handled with reuse. If you have any questions on this, please have a look into the contributing guide or contact the maintainers of dasf-messaging-python.

Contact

Philipp S. Sommer eMail: philipp.sommer@hereon.de

Helmholtz-Zentrum Hereon Max-Planck-Str. 1 21502 Geesthacht Germany