nomad-ml-workflows

A NOMAD plugin for supporting machine-learning workflows. It currently provides the Export Entries action, which turns a permission-scoped NOMAD search into a reusable JSON, Parquet, or CSV dataset.

📦 Installation

Install the package with pip:

pip install nomad-ml-workflows

To add the plugin to your NOMAD deployment, see Adding this plugin to NOMAD.

✨ Key functionality

The Export Entries action lets users:

• select entries with a NOMAD search query and ownership scope;
• export complete archives or selected archive paths, optionally resolving references;
• generate JSON, Parquet, or CSV datasets; and
• save the generated dataset to a staging project/upload as a ZIP archive or directory.

JSON preserves the selected archive structure. Parquet provides typed tabular data and preserves nested quantity values, while CSV represents nested values as JSON text.

Action input

The action form groups its fields into Search options and Export options. Its underlying input shape is:

{
  "user_id": "<injected by NOMAD>",
  "upload_id": "<destination-project-id>",
  "search_settings": {
    "owner": "visible",
    "max_entries": 1000,
    "query": "{\n  \"entry_type\": \"ELNSample\"\n}",
    "required": [
      {
        "type": "include",
        "path": "results.method",
        "resolve_references": false
      }
    ]
  },
  "export_settings": {
    "file_format": "parquet",
    "create_zip_archive": true
  }
}

user_id is required by the workflow but is not shown in the action form. The other fields have these semantics:

The current public model supports include directives only. Exclusion is not yet available through the action form.

Generated output

The generated ZIP archive or directory contains:

The metadata records the original input, search timing, export-limit state, and the num_entries_available, num_entries_selected, and num_entries_exported counts. It also records nomad_deployment_api_host, nomad_version, and nomad_ml_workflows_version. A zero-match export still contains metadata.json and an empty selected_entries.json.

⚙️ Configuration

Configure the action entry point in the nomad.yaml of the NOMAD Oasis:

plugins:
  entry_points:
    options:
      nomad_ml_workflows.actions:export_entries:
        max_entries_export_limit: 100000
        # Deployment cap for one Export Entries action.

        read_archives_timeout: 7200
        # Start-to-close timeout, in seconds, for reading archives and
        # writing the selected output format.

        max_write_buffer_size_bytes: 4194304
        # Target maximum uncompressed Arrow bytes accumulated before a
        # Parquet or CSV flush. One oversized row may exceed this target.

🚀 Adding this plugin to NOMAD

NOMAD Oasis

Follow the NOMAD plugin installation documentation to add and enable the plugin in an Oasis.

Local NOMAD development installation

Use the dedicated nomad-distro-dev repository for an integrated local NOMAD development environment.

🛠️ Development

Clone the repository and create a virtual environment with Python 3.10, 3.11, or 3.12:

git clone https://github.com/FAIRmat-NFDI/nomad-ml-workflows.git
cd nomad-ml-workflows
python3.12 -m venv .pyenv
. .pyenv/bin/activate
python -m pip install --upgrade pip
python -m pip install uv
uv pip install -e '.[dev]'

Run the focused Export Entries tests:

pytest tests/actions/export_entries/test_models.py \
  tests/actions/export_entries/test_utils.py

Run linting and formatting checks with Ruff:

ruff check .
ruff format . --check

For interactive test debugging, pass --pdb to pytest. To serve the documentation locally, use the development dependencies and run:

mkdocs serve

👥 Main contributors

📄 License

This project is licensed under the MIT License. See LICENSE.