nomad-ml-workflows

A NOMAD plugin for supporting machine-learning workflows. It currently provides the Export Entries action, which turns a permission-scoped NOMAD search into a reusable JSON, Parquet, or CSV dataset.

📦 Installation

Install the package with pip:

pip install nomad-ml-workflows

To add the plugin to your NOMAD deployment, see Adding this plugin to NOMAD.

✨ Key functionality

The Export Entries action lets users:

• select entries with a NOMAD search query and ownership scope;
• export complete archives or selected archive paths, optionally resolving references;
• generate JSON, Parquet, or CSV datasets; and
• save the generated dataset to a staging project/upload as a ZIP archive or directory.

JSON preserves the selected archive structure. Parquet provides typed tabular data and preserves nested quantity values as a sharded dataset, while CSV represents nested values as JSON text.

Action input

The action form groups its fields into Search options and Export options. Its underlying input shape is:

{
  "user_id": "<injected by NOMAD>",
  "upload_id": "<destination-project-id>",
  "search_settings": {
    "owner": "visible",
    "max_entries": 1000,
    "query": "{\n  \"entry_type\": \"ELNSample\"\n}",
    "required": [
      {
        "type": "include",
        "path": "results.method",
        "resolve_references": false
      }
    ]
  },
  "export_settings": {
    "file_format": "parquet",
    "create_zip_archive": true
  }
}

user_id is required by the workflow but is not shown in the action form. The other fields have these semantics:

The current public model supports include directives only. Exclusion is not yet available through the action form.

Generated output

The generated ZIP archive or directory contains:

The metadata records the original input, search timing, export-limit state, and the num_entries_available, num_entries_selected, and num_entries_exported counts. It also records nomad_deployment_api_host, nomad_version, and nomad_ml_workflows_version. A zero-match export still contains metadata.json and an empty selected_entries.json.

PyArrow reads the complete Parquet dataset directly from its directory:

import pyarrow.parquet as pq

table = pq.read_table('data.parquet')

⚙️ Configuration

Configure the action entry point in the nomad.yaml of the NOMAD Oasis:

plugins:
  entry_points:
    options:
      nomad_ml_workflows.actions:export_entries:
        max_entries_export_limit: 100000
        # Deployment cap for one Export Entries action.

        read_archives_timeout: 7200
        # Start-to-close timeout, in seconds, for reading archives and
        # writing the selected output format.

        write_tabular_timeout: 7200
        # Start-to-close timeout, in seconds, for writing the output 
        # tabular artifact.

        max_write_buffer_size_bytes: 67108864  # 64 MB
        # Maximum number of encoded NDJSON input bytes represented by parsed
        # rows buffered before writing to the output tabular file. Protects
        # against rows with large encoded representations. One oversized row
        # may exceed this target.

        max_write_buffer_size_rows: 1024
        # Maximum number of parsed rows buffered before writing to the output
        # tabular file. Protects against many small or sparse rows whose Python
        # and Arrow representations are much larger than their NDJSON bytes.

For deployments where PyArrow uses the mimalloc memory-pool backend, the CPU action worker can be started with immediate memory purging enabled:

MIMALLOC_PURGE_DELAY=0

The tabular exporter requests that unused Arrow memory be released after every batch. Setting the environment variable MIMALLOC_PURGE_DELAY=0 asks mimalloc to return unused pages to the operating system immediately. In local measurements, this produced slightly lower post-export RSS.

This setting is optional and must be present in the CPU worker environment before the worker starts. Immediate purging may trade some allocation performance for lower retained RSS.

🚀 Adding this plugin to NOMAD

NOMAD Oasis

Follow the NOMAD Oasis configuration documentation to add and enable the plugin in an Oasis.

Local NOMAD development installation

Use the dedicated nomad-distro-dev repository for an integrated local NOMAD development environment.

🛠️ Development

Clone the repository and create a virtual environment with Python 3.10, 3.11, or 3.12:

git clone https://github.com/FAIRmat-NFDI/nomad-ml-workflows.git
cd nomad-ml-workflows
python3.12 -m venv .pyenv
. .pyenv/bin/activate
python -m pip install --upgrade pip
python -m pip install uv
uv pip install -e '.[dev]'

Run the focused Export Entries tests:

pytest tests/actions/export_entries/test_models.py \
  tests/actions/export_entries/test_utils.py

Run linting and formatting checks with Ruff:

ruff check .
ruff format . --check

For interactive test debugging, pass --pdb to pytest. To serve the documentation locally, use the development dependencies and run:

mkdocs serve

👥 Main contributors

📄 License

This project is licensed under the MIT License. See LICENSE.