MEx artificial

Create artificial data for the MEx project.

Project

The Metadata Exchange (MEx) project is committed to improve the retrieval of RKI research data and projects. How? By focusing on metadata: instead of providing the actual research data directly, the MEx metadata catalog captures descriptive information about research data and activities. On this basis, we want to make the data FAIR¹ so that it can be shared with others.

Via MEx, metadata will be made findable, accessible and shareable, as well as available for further research. The goal is to get an overview of what research data is available, understand its context, and know what needs to be considered for subsequent use.

RKI cooperated with D4L data4life gGmbH for a pilot phase where the vision of a FAIR metadata catalog was explored and concepts and prototypes were developed. The partnership has ended with the successful conclusion of the pilot phase.

After an internal launch, the metadata will also be made publicly available and thus be available to external researchers as well as the interested (professional) public to find research data from the RKI.

For further details, please consult our project page.

Contact
For more information, please feel free to email us at mex@rki.de.

Publisher

Robert Koch-Institut
Nordufer 20
13353 Berlin
Germany

Package

Create artificial extracted items, transform them into merged items and write the results into a configured sink.

License

This package is licensed under the MIT license. All other software components of the MEx project are open-sourced under the same license as well.

Development

Installation

• install python on your system
• on unix, run make install
• on windows, run .\mex.bat install

Linting and testing

• run all linters with make lint or .\mex.bat lint
• run unit and integration tests with make test or .\mex.bat test
• run just the unit tests with make unit or .\mex.bat unit

Updating dependencies

• update boilerplate files with cruft update
• update global requirements in requirements.txt manually
• update git hooks with pre-commit autoupdate
• update package dependencies using uv sync --upgrade
• update github actions in .github/workflows/*.yml manually

Creating release

• run mex release RULE to release a new version where RULE determines which part of the version to update and is one of major, minor, patch.

Container workflow

• build image with make image
• run directly using docker make run
• start with docker compose make start

Container verification

Images released to GHCR are signed using cosign.

To verify an image manually: cosign verify --certificate-identity-regexp "https://github.com/robert-koch-institut/mex-artificial/.github/workflows/release.yml@refs/heads/main" --certificate-oidc-issuer "https://token.actions.githubusercontent.com" ghcr.io/robert-koch-institut/mex-artificial:<tag>

Python release verification

Python release artifacts (source distributions and wheels) published to GitHub Releases are signed keyless using sigstore.

To verify a release artifact manually, download the artifact (e.g. mex_artificial-<tag>-py3-none-any.whl) and its Sigstore bundle (mex_artificial-<tag>-py3-none-any.whl.sigstore.json), then run either:

Using sigstore:

sigstore verify identity \
  --bundle <path-to-bundle> \
  --cert-identity "https://github.com/robert-koch-institut/mex-artificial/.github/workflows/release.yml@refs/heads/main" \
  --cert-oidc-issuer "https://token.actions.githubusercontent.com" \
  <path-to-artifact>

Using cosign:

cosign verify-blob \
  --bundle <path-to-bundle> \
  --certificate-identity "https://github.com/robert-koch-institut/mex-artificial/.github/workflows/release.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  <path-to-artifact>

Pre-built workflow

• you can run the latest artificial data generator without building it locally
• just pull it from the container registry and configure using cli arguments
• docker run -it -u $(id -u):$(id -g) -v $(pwd):/out ghcr.io/robert-koch-institut/mex-artificial:latest --count=1000 --chattiness=10
• using -u $(id -u):$(id -g) to run the process using your local user
• using -v $(pwd):/out to specify an output directory for the resulting ndjson
• --count controls the number of items to generate
• --chattiness controls the number of words in textual fields

Commands

• run uv run artificial --help to print instructions