Dataset Manager

Library to pre-process CSV files from Research Portal into usable datasets.

Installation

pip install minder.dataset-manager

Example

import logging
import asyncio
import sys
from typing import Optional
from minder.dataset_manager._utils import Dataset
from minder.dataset_manager.datasets import LabelledUtiDataset
from minder.research_portal_client import Configuration, JobManager


logging.basicConfig(level=logging.INFO)

Configuration.set_default(
    Configuration(
        access_token="---REDACTED---",
    )
)


async def example1():
    job_ids = ["c25249e0-82ff-43d1-9676-f3cead0228b9"]
    async with JobManager() as job_manager:
        files = Dataset.download(job_ids, job_manager)
        dataset = LabelledUtiDataset.create(job_ids, files)
        dataset.save("./my-dataset.npz")


async def example2():
    job_ids = ["c25249e0-82ff-43d1-9676-f3cead0228b9"]
    existing_dataset = "./my-dataset.npz"
    async with JobManager() as job_manager:
        download_task = Dataset.download(job_ids, job_manager)
        try:
            previous_dataset: Optional[Dataset] = None
            if existing_dataset.exists():
                previous_dataset = LabelledUtiDataset.load(existing_dataset)
        finally:
            files = await download_task

        new_dataset = LabelledUtiDataset.create(job_ids, files)

        dataset = (
            await previous_dataset.update(new_dataset, job_manager=job_manager)
            if previous_dataset is not None
            else new_dataset
        )
        dataset.save("./my-dataset.npz")


async def main():
    await example1()
    await example2()


if sys.platform == "win32"::
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

asyncio.run(main())

Development

Useful commands

Setup

poetry install

Run tests

poetry run pytest

Code Coverage

This command consists of 2 parts:

• running tests with coverage collection
• formatting the report: report (text to stdout), xml (GitLab compatible: cobertura), html (visual)

poetry run coverage run -m pytest && poetry run coverage report -m

Linting

poetry run flake8

Formatting

poetry run black .

Type Checking

poetry run mypy .