DataLad extension for semantic metadata handling

Overview

This software is a DataLad extension that equips DataLad with an alternative command suite for metadata handling (extraction, aggregation, filtering, and reporting).

Command(s) currently provided by this extension

meta-extract -- run an extractor on a file or dataset and emit the resulting metadata (stdout).
meta-filter -- run an filter over existing metadata and return the resulting metadata (stdout).
meta-add -- add a metadata record or a list of metadata records (possibly received on stdin) to a metadata store, usually to the git-repo of the dataset.
meta-aggregate -- aggregate metadata from multiple local or remote metadata-stores into a local metadata store.
meta-dump -- reporting metadata from local or remote metadata stores. Allows to select metadata by file- or dataset-path matching patterns including dataset versions and dataset IDs.
meta conduct -- execute processing pipelines that consist of a provider which emits objects that should be processed, e.g. files or metadata, and a pipeline of processors, that perform operations on the provided objects, such as metadata-extraction and metadata-adding.Processors are usually executed in parallel. A few pipeline definitions are provided with the release.

Commands currently under development:

meta-export -- write a flat representation of metadata to a file-system. For now you can export your metadata to a JSON-lines file named metadata-dump.jsonl:
```
 datalad meta-dump -d <dataset-path> -r >metadata-dump.jsonl
```
meta-import -- import a flat representation of metadata from a file-system. For now you can import metadata from a JSON-lines file, e.g. metadata-dump.jsonl like this:
```
 datalad meta-add -d <dataset-path> --json-lines -i metadata-dump.jsonl
```
meta-ingest-previous -- ingest metadata from metalad<=0.2.1.

Additional metadata extractor implementations

Compatible with the previous families of extractors provided by datalad and by metalad, i.e. metalad_core, metalad_annex, metalad_custom, metalad_runprov
New metadata extractor paradigm that distinguishes between file- and dataset-level extractors. Included are two example extractors, metalad_example_dataset, and metalad_example_file
metalad_external_dataset and metalad_external_file, a dataset- and a file-extractors that execute external processes to generate metadata allow processing of the externally created metadata in datalad.
metalad_studyminimeta -- a dataset-level extractor that reads studyminimeta yaml files and produces metadata that contains a JSON-LD compatible description of the data in the input file

Indexers

Provides indexers for the new datalad indexer-plugin interface. These indexers convert metadata in proprietary formats into a set of key-value pairs that can be used by datalad search to search for content.
indexer_studyminimeta -- converts studyminimeta JSON-LD description into key-value pairs for datalad search.
indexer_jsonld -- a generic JSON-LD indexer that aims at converting any JSON-LD descriptions into a set of key-value pairs that reflect the content of the JSON-LD description.

Installation

Before you install this package, please make sure that you install a recent version of git-annex. Afterwards, install the latest version of datalad-metalad from PyPi. It is recommended to use a dedicated virtualenv:

# create and enter a new virtual environment (strongly recommended)
virtualenv --system-site-packages --python=python3 ~/env/datalad
. ~/env/datalad/bin/activate

# install from PyPi
pip install datalad-metalad

Support

For general information on how to use or contribute to DataLad (and this extension), please see the DataLad website or the main GitHub project page. The documentation is found here: http://docs.datalad.org/projects/metalad

All bugs, concerns and enhancement requests for this software can be submitted here: https://github.com/datalad/datalad-metalad/issues

If you have a problem or would like to ask a question about how to use DataLad, please submit a question to NeuroStars.org with a datalad tag. NeuroStars.org is a platform similar to StackOverflow but dedicated to neuroinformatics.

All previous DataLad questions are available here: http://neurostars.org/tags/datalad/

Acknowledgements

This DataLad extension was developed with support from the German Federal Ministry of Education and Research (BMBF 01GQ1905), and the US National Science Foundation (NSF 1912266).

Metadata

Release files for datalad-metalad 0.4.22

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datalad-metalad 0.4.22
File	Size	Uploaded	Provenance
datalad_metalad-0.4.22.tar.gz	198.2 kB	Jan 23, 2024	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for datalad-metalad 0.4.22
File	Interpreter	ABI	Platform	Reset
datalad_metalad-0.4.22-py3-none-any.whl 228.3 kB Jan 23, 2024 Without provenance	Python 3	none	any	Details

Total release size: 426.5 kB

Release files / datalad_metalad-0.4.22.tar.gz

Download URL	datalad_metalad-0.4.22.tar.gz
Size	198.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`14c48598de4fd23298ac0b326f8d9d1b215fef756d67dd4d173108cedbad1756`
BLAKE2b-256 checksum How to use checksums	`f94fd0f0ce074bb8024bc47d02564bd75742dec91ad08220e7cac28c0b7a81e1`
Upload date	Jan 23, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.9.18`

Release files / datalad_metalad-0.4.22-py3-none-any.whl

Download URL	datalad_metalad-0.4.22-py3-none-any.whl
Size	228.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`915200eb7e483d9d45dad07557bf59e6d6e81c5506e21726eda0e5fb6f1383f8`
BLAKE2b-256 checksum How to use checksums	`cfd18dac93b6cab935e222861fe27e1c0235204dcaf072e7d1c88da156037dbf`
Upload date	Jan 23, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.9.18`

Security for datalad-metalad 0.4.22

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.