icat_indexer

Synchronises ICAT dataset metadata into Elasticsearch.

Two complementary sync mechanisms run in parallel:

• Nightly scheduler: indexes all investigations whose release date falls on today
• ActiveMQ listener: reacts in real time when a release date changes or a dataset DOI is minted

When an investigation goes back under embargo (release date moved to the future), its datasets are automatically removed from the index.

The ES index mapping is generated automatically from icat_esrf_definitions models, so it stays in sync with the metadata schema without manual maintenance.

Installation

pip install icat_indexer

Or in development mode:

pip install -e ".[dev]"

Configuration

Copy .env.example to .env and fill in your credentials:

cp .env.example .env

Usage

Run the nightly scheduler:

icat-indexer-scheduler

Run the ActiveMQ listener:

icat-indexer-listener

Both can run simultaneously as separate processes or services.

ActiveMQ message format

Investigation release date changed (index or delete all datasets):

{"type": "investigation_release_changed", "investigationId": "12345"}

Dataset DOI minted (always index, never delete):

{"type": "dataset_doi_minted", "datasetIds": ["67890", "75315"]}

Development

pip install -e ".[dev]"
pytest
ruff check . --fix  
ruff format  

License

MIT - see LICENSE.md.

Documentation

https://icat_indexer.readthedocs.io/