Skip to main content

MENTO - Run remote data processing from the beamline

Pip

https://img.shields.io/pypi/v/desy-mento

Docs

https://img.shields.io/badge/MENTO_webpage-blue

CI

https://gitlab.desy.de/fs-sc/mento/badges/master/pipeline.svg

Tests

https://gitlab.desy.de/fs-sc/mento/badges/master/coverage.svg

MENTO is a Python (3.7+) library for running remote data processing on Maxwell computers from any local beamline computer on PETRA III beamlines at DESY.

The data processing could be any task, like

  • remote data analysis

  • remote/local data visualization

All you need to know are:

  • how to run the data processing program

  • location of the input data and desired output data

MENTO contains helper functions that automate the boring bits, like

  • getting access to Maxwell resources

  • using time-limited credentials for remote connections

  • submitting data processing jobs to the HPC resource manager (e.g. Slurm)

Here is a short YouTube video about the main uses of MENTO, from a talk given at NOBUGS 2022.

Basic usage

In its simplest form, MENTO can be used at a PETRA-III beamline to launch a data processing job on a remote HPC (Maxwell) node, without needing to know how to connect to the HPC cluster Maxwell, or how to authenticate without having a scientific account for Maxwell, or indeed how job submissions work.

A remote analysis program can be started on data acquired at the beamline, by a Python script importing a MENTO module, creating a MENTO Trigger object and calling run() on it.

Example

Here is a minimum working example:

from mento import Trigger

my_trigger = Trigger()  # default trigger will submit Slurm jobs on an HPC node

my_analysis = 'maxwell_program.exe'  # path to analysis program available on Maxwell

my_args = ['analysis', 'parameters', 'list', 'inputdatafile']  # arguments required by analysis program, including input data

my_trigger.run([my_analysis, my_args])  # connect to a remote machine with appropriate credentials and start the analysis job there

Note that paths to input data files and output directories can be specified as you see them at the beamline, and they will be automagically found in the remote core filesystem on Maxwell.

Paths to the analysis program on Maxwell, will of course need to refer to the ‘remote’ paths as seen from a Maxwell node, since MENTO cannot know about them from the beamline.

A dummy example of running MENTO to do live data analysis during data acquisition is shown here.

More detailed usage instructions are on the MENTO usage pages.

Installation

Pre-built MENTO Python wheels are available on PyPI.

The easiest way to install MENTO would be to use pip. running the following command from your shell/command prompt:

python3 -m pip install desy-mento

Alternatively, the package can be downloaded as an artifact from the Gitlab repository.

Citing MENTO

If you used MENTO and found it relevant to the work you are publishing, please consider citing the MENTO introduction article that appears in the SRI 2021 proceedings.

You can use the BibTeX citation given below:

@article{Vijay_Kartik_2022,
doi = {10.1088/1742-6596/2380/1/012104},
url = {https://dx.doi.org/10.1088/1742-6596/2380/1/012104},
year = {2022},
month = {dec},
publisher = {IOP Publishing},
volume = {2380},
number = {1},
pages = {012104},
author = {S Vijay Kartik and Michael Sprung and Fabian Westermeier and Anton Barty},
title = {MENTO: Automated near real-time data analysis at PETRA III},
journal = {Journal of Physics: Conference Series},
}

Alternatively, you can use this CITATION.cff file to export the citation to your favourite format (BibTeX/APA/RIS/EndNote/CodeMeta).

Licences

  • All code is licensed under GPL-3.0-or-later

  • All documentation is licensed under CC-BY-4.0

  • All other files are licensed under CC0-1.0

Full texts of the licences can be found in LICENSES.

This project aims to be REUSE compliant.

Contributing

Contributions in all forms are welcome - be it issues, bug reports, or code!

The MENTO project is on DESY Gitlab.

If you have a Helmholtz AAI account, you can log in directly and open issues. Alternatively, you can send an email to vijay.kartik@desy.de with bug reports and comments, or even patches.

API documentation

API documentation can be found on the MENTO API pages.

The documentation is auto-generated, so it should be as up-to-date as the comments in the source code. ;)

Contact

For questions and critiques, please contact S. Vijay Kartik <vijay.kartik@desy.de>.

Metadata

Release files for desy-mento 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for desy-mento 0.2.2
File Interpreter ABI Platform Actions
desy_mento-0.2.2-py3-none-any.whl Python 3 none any Details for desy_mento-0.2.2-py3-none-any.whl

Release files / desy_mento-0.2.2-py3-none-any.whl

Download URL desy_mento-0.2.2-py3-none-any.whl
Size 26.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c3847bb7aac9915a3aa5ac3c0d7b53a4a601d2ebc75eef4ab6b36b10a6197683
BLAKE2b-256 checksum
How to use checksums
d6a94a60511dbd3bbb1025150151f470aa42fa0d29f3f1b62248ce998571696b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.7.17

Security for desy-mento 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page