SciDatS

SciDatS is a python package for storing and retrieving scientific data stored as parquet files with JSON-LD metadata (semantically annotated JSON - Linked Data).

This Scientific Data Standard is designed as a data exchange format to enable exchange/synchronisation of Scientific Data, maintaining all metadata between different laboratories.

Features

• efficient storage and retrieval of scientific data and metadata in a single file (parquet with JSON-LD metadata)
• convenient functions for retrieving data and metadata
• improved tooling based on pydantic and rdflib
• reading and writing for SciDatS files
• coupling to the LabDataReader framework - for transforming proppriatory lab data into a semantically annotated SciDatSa format.
• recommended metadata formats should be DCAT-application profiles, e.g. DCAT-AP-PLUS or Chem-DCAT-AP

Design criteria

Here are some of the criteria the data / metadata standard has to fulfil (and in brackets the selected technology) :

• data and metadata storage for scientific / machine learning needs (semantic annotation, based on ontologies, derivatives of owlready2)

  • proper nullable data / missing data handling (pyarrow / parquet)

  • data modalities, like range / limits, type / continuous / categorial / variable treatment in case of range violation (parquet metadata)

  • cardinality (parquet metadata)

• efficient storage (parquet)

• metadata and data stored at one place (parquet)

• metadata conservation when saving / loading / processing (parquet -> arrow)

• fast data exchange (arrow flight, MinIO active replication)

• fast loading (fastparquet, pyarrow)

• fast data processing without in-memory re-writing after loading ( pandas with pyarrow backend, arrow flight, polars)

• "modalities" for the machine learning models

• semantic annotations / metadata in RDF compliant format - for creating instances of ontology classes and SPARQL reasoning (JSON-LD, rdflib, owlready2)

• fast data processing (direct loading into pyarrow driven dataframe )

• programming language agnostic / independent (parquet)

• easy to use (SciDatS / labDataReader framework, currently in implementation by me)

• commonly used in ETL pipelines (Apache Spark, prefect, ... )

• suitable for S3 file storage systems (MinIO)

Installation

You can install SciDatS via pip (or your favourite package manager):

# using pip
pip install scidats
# using uv
uv add scidats
uv sync --group dev --group test

Tutorials

A tutorial on how to use SciDatS can be found in the scidat_demo_tutorial.ipynb Jupyter notebook in the jupyter folder.

Documentation

The Documentation can be found here: https://opensourcelab/scientificdata.gitlab.io/scidats

ReadTheDocs: https://scidats.readthedocs.io

Source Code: https://gitlab.com/opensourcelab/scientificdata/scidats

SciDatS is a python package for storing and retrieving scientific data stored in JSON-LD (semantically annotated JSON - Linked Data).

Contributors ✨

Thanks goes to these wonderful people (emoji key):

This project follows the all-contributors specification. Contributions of any kind welcome!

Credits

This package was created with Copier and the pypackage-template project template.