desy-mento 0.2.2

A Python (3.7+ only) package for PETRA-III beamlines at DESY for automatically triggering online data processing jobs like remote data analysis, and local visualization tasks.

Pip
Docs
CI
Tests

MENTO is a Python (3.7+) library for running remote data processing on Maxwell computers from any local beamline computer on PETRA III beamlines at DESY.

The data processing could be any task, like

• remote data analysis

• remote/local data visualization

All you need to know are:

• how to run the data processing program

• location of the input data and desired output data

MENTO contains helper functions that automate the boring bits, like

• getting access to Maxwell resources

• using time-limited credentials for remote connections

• submitting data processing jobs to the HPC resource manager (e.g. Slurm)

Here is a short YouTube video about the main uses of MENTO, from a talk given at NOBUGS 2022.

Basic usage

In its simplest form, MENTO can be used at a PETRA-III beamline to launch a data processing job on a remote HPC (Maxwell) node, without needing to know how to connect to the HPC cluster Maxwell, or how to authenticate without having a scientific account for Maxwell, or indeed how job submissions work.

A remote analysis program can be started on data acquired at the beamline, by a Python script importing a MENTO module, creating a MENTO Trigger object and calling run() on it.

Example

Here is a minimum working example:

from mento import Trigger

my_trigger = Trigger()  # default trigger will submit Slurm jobs on an HPC node

my_analysis = 'maxwell_program.exe'  # path to analysis program available on Maxwell

my_args = ['analysis', 'parameters', 'list', 'inputdatafile']  # arguments required by analysis program, including input data

my_trigger.run([my_analysis, my_args])  # connect to a remote machine with appropriate credentials and start the analysis job there

Note that paths to input data files and output directories can be specified as you see them at the beamline, and they will be automagically found in the remote core filesystem on Maxwell.

Paths to the analysis program on Maxwell, will of course need to refer to the ‘remote’ paths as seen from a Maxwell node, since MENTO cannot know about them from the beamline.

A dummy example of running MENTO to do live data analysis during data acquisition is shown here.

More detailed usage instructions are on the MENTO usage pages.

Installation

Pre-built MENTO Python wheels are available on PyPI.

The easiest way to install MENTO would be to use pip. running the following command from your shell/command prompt:

python3 -m pip install desy-mento

Alternatively, the package can be downloaded as an artifact from the Gitlab repository.

Citing MENTO

If you used MENTO and found it relevant to the work you are publishing, please consider citing the MENTO introduction article that appears in the SRI 2021 proceedings.

You can use the BibTeX citation given below:

@article{Vijay_Kartik_2022,
doi = {10.1088/1742-6596/2380/1/012104},
url = {https://dx.doi.org/10.1088/1742-6596/2380/1/012104},
year = {2022},
month = {dec},
publisher = {IOP Publishing},
volume = {2380},
number = {1},
pages = {012104},
author = {S Vijay Kartik and Michael Sprung and Fabian Westermeier and Anton Barty},
title = {MENTO: Automated near real-time data analysis at PETRA III},
journal = {Journal of Physics: Conference Series},
}

Alternatively, you can use this CITATION.cff file to export the citation to your favourite format (BibTeX/APA/RIS/EndNote/CodeMeta).

Licences

• All code is licensed under GPL-3.0-or-later

• All documentation is licensed under CC-BY-4.0

• All other files are licensed under CC0-1.0

Full texts of the licences can be found in LICENSES.

This project aims to be REUSE compliant.

Contributing

Contributions in all forms are welcome - be it issues, bug reports, or code!

The MENTO project is on DESY Gitlab.

If you have a Helmholtz AAI account, you can log in directly and open issues. Alternatively, you can send an email to [e-mail] with bug reports and comments, or even patches.

API documentation

API documentation can be found on the MENTO API pages.

The documentation is auto-generated, so it should be as up-to-date as the comments in the source code. ;)

Contact

For questions and critiques, please contact S. Vijay Kartik <[e-mail]>.