nomad-north-emtools

NOMAD NORTH plugin for analysis of electron microscopy data

This nomad plugin was generated with Cookiecutter along with @nomad's cookiecutter-nomad-plugin template.

Development

If you want to develop locally this plugin, clone the project and in the plugin folder, create a virtual environment (you can use Python 3.10, 3.11 or 3.12):

git clone https://github.com/FAIRmat-NFDI/nomad-north-emtools.git
cd nomad-north-emtools
python3.11 -m venv .pyenv
. .pyenv/bin/activate

Make sure to have pip upgraded:

pip install --upgrade pip

We recommend installing uv for fast pip installation of the packages:

pip install uv

Install the nomad-lab package:

uv pip install -e '.[dev]'

Run the tests

You can run locally the tests:

python -m pytest -sv tests

where the -s and -v options toggle the output verbosity.

Our CI/CD pipeline produces a more comprehensive test report using the pytest-cov package. You can generate a local coverage report:

uv pip install pytest-cov
python -m pytest --cov=src tests

Run linting and auto-formatting

We use Ruff for linting and formatting the code. Ruff auto-formatting is also a part of the GitHub workflow actions. You can run locally:

ruff check .
ruff format . --check

Debugging

For interactive debugging of the tests, use pytest with the --pdb flag. We recommend using an IDE for debugging, e.g., VSCode. If that is the case, add the following snippet to your .vscode/launch.json:

{
  "configurations": [
      {
        "name": "<descriptive tag>",
        "type": "debugpy",
        "request": "launch",
        "cwd": "${workspaceFolder}",
        "program": "${workspaceFolder}/.pyenv/bin/pytest",
        "justMyCode": true,
        "env": {
            "_PYTEST_RAISE": "1"
        },
        "args": [
            "-sv",
            "--pdb",
            "<path-to-plugin-tests>",
        ]
    }
  ]
}

where <path-to-plugin-tests> must be changed to the local path to the test module to be debugged.

The settings configuration file .vscode/settings.json automatically applies the linting and formatting upon saving the modified file.

Documentation on Github pages

To view the documentation locally, install the related packages using:

uv pip install -r .[docs]

Run the documentation server:

mkdocs serve

Adding this plugin to NOMAD

Currently, NOMAD has two distinct flavors that are relevant depending on your role as an user:

1. A NOMAD Oasis: any user with a NOMAD Oasis instance.
2. Local NOMAD installation and the source code of NOMAD: internal developers.

Adding this plugin in your NOMAD Oasis

Read the NOMAD plugin documentation for all details on how to deploy the plugin on your NOMAD instance.

Adding this plugin in your local NOMAD installation and the source code of NOMAD

We now recommend using the dedicated nomad-distro-dev repository to simplify the process. Please refer to that repository for detailed instructions.

Publish note

In the GitHub actions workflow for publishing the nomad-north-emtools plugin to PyPI, we commented out the deploy job . If you want to publish the plugin to PyPI, you need to set up your project in PyPI. There are several online tutorials on publishing a Python package to PyPI, e.g., How to Publish a Python Package to PyPI. After that, you can uncomment the deploy job in the workflow file and push the changes to GitHub. The workflow will be triggered and the package will be published to PyPI when you create a new release on GitHub.

In our Python package publishing workflow, before building the package, we update the image tag in the NORTHTool entry point to the latest release version of the image (e.g., v0.1.5), and then publish the package to PyPI.

However, the updated image tag in NORTHTool is not pushed back to the GitHub repository. Therefore, the image tag in the GitHub repository always remains set to main, even when you check out a specific release tag. For this reason, we recommend installing the plugin from PyPI, where the entry point always contains the correct image tag corresponding to the release.

If you download a ZIP file of a specific release from GitHub, the image tag in the entry point will still be set to main, which is not correct. In that case, you can either manually update the image tag in the entry point to the correct release version (e.g., v0.1.5), or install the plugin directly from PyPI.

Template update

We use cruft to update the project based on template changes. To run the check for updates locally, run cruft update in the root of the project. More details see the instructions on cruft website.

Main contributors