RE-MAT Data

This package provides a simple interface to download and upload datasets used in the ReMat project.

Installation

This is a python package and can be installed using pip:

pip install remat-data-downloader

Authentication

You need to obtain a Clowder API key to use this package. You can obtain this key by logging into the RE-MAT Clowder instance and going to your user settings page (grey silhouette in the upper right corner of the page). Click on the API Keys tab and create a new key. Save this key in a file nameed clowder_key.txt in the directory where you will be running the remat-download-data command.

Usage

The remat-data command can be run from your command line. It has commands to interact with Clowder spaces and with datasets.

List Spaces

remat-data spaces list

                      Clowder Spaces
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Name              ┃ ID                       ┃ datasets ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Front velocities  │ 6674972be4b0a2d1b9ba0228 │ 303      │
│ DSC Post Cures    │ 6669d4d0e4b0a2d1b9b9a797 │ 228      │
│ DSC Cure Kinetics │ 64343b6be4b01a23c58bad90 │ 307      │
└───────────────────┴──────────────────────────┴──────────┘

Download Datasets

Now that you know the unique ID for the space you want to download, you can download the datasets using the download command. This command will create subdirectories for each dataset and download the metadata as a json file and the DSC Curve as a csv file.

You can re-run this command, and it will skip downloads of datasets that already exist in the directory.

remat-data spaces download 6669d4d0e4b0a2d1b9b9a797

List Collections

Collections are another way to organize datasets in Clowder. You can list all available collections using the following command:

remat-data collections list

                      Clowder Collections
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Name              ┃ ID                       ┃ datasets ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Example Collection│ 507f1f77bcf86cd799439011 │ 150      │
└───────────────────┴──────────────────────────┴──────────┘

Download from Collections

Similar to downloading from spaces, you can download all datasets from a specific collection using its unique ID. This command will download metadata, DSC_Curve.csv files, and any MP4 video files for each dataset in the collection.

remat-data collections download 507f1f77bcf86cd799439011

Like the space download command, this will skip datasets that have already been downloaded to avoid redundant transfers.

Upload Files to Space

Using the upload command, you can upload files to a space. The command takes the space name as a flag, optional --name flag for the dataset name and a list of file name to upload. If dataset name is not mentioned, A default dataset is created and the files are uploaded under the newly created dataset

Valid values of spaces are: --Cure, --PostCure, --FrontVelocity │

For help run

remat-data spaces upload --help

Command to upload:

 remat-data spaces upload --Cure --name TEST-dataset-2 test3.csv DSC_Curve.csv

Configuration

Application configuration and space name to UUID mapping is stored in config.py Add new spaces to this config.py file