grand-challenge-forge 0.5.0

Generation of challenge packs

🛠️ grand-challenge-forge

A utility that generates distributable items that help challenge organizers set up their challenge more easily on Grand-Challenge.org.

---

Install

Install via PyPi:

pip install grand-challenge-forge
grand-challenge-forge --help

📦 Challenge packs

A challenge pack consists of challenge-tailored examples for the following:

• A script to automate uploading data to an archive
• A submission algorithm that can be submitted to a challenge phase
• An evaluation method that evaluates algorithm submissions and generates performance metrics for ranking

Usage

grand-challenge-forge pack-context.json

Will use the context found in pack-context.json and generate a pack at the current working directory in a directory dist/ (default).

Example of the content of pack-context.json

{
    "challenge": {
        "slug": "challenge-slug",
        "phases": [
            {
                "slug": "phase-slug",
                "archive": {
                    "url": "https://grand-challenge.org/archives/archive-slug/"
                },
                "algorithm_inputs": [
                    {
                        "slug": "input-ci-slug",
                        "kind": "Segmentation",
                        "super_kind": "Image",
                        "relative_path": "images/input-value"
                    },
                    {
                        "slug": "another-input-ci-slug",
                        "kind": "Anything",
                        "super_kind": "File",
                        "relative_path": "another-input-value.json"
                    }
                ],
                "algorithm_outputs": [
                    {
                        "slug": "output-ci-slug",
                        "kind": "Image",
                        "super_kind": "Image",
                        "relative_path": "images/output-value"
                    },
                    {
                        "slug": "another-output-ci-slug",
                        "kind": "Anything",
                        "super_kind": "File",
                        "relative_path": "output-value.json"
                    }
                ]
            },
            {
                "slug": "another-phase-slug",
                "archive": {
                    "url": "https://grand-challenge.org/archives/another-archive-slug/"
                },
                "algorithm_inputs": [
                    {
                        "slug": "input-ci-slug",
                        "kind": "Image",
                        "super_kind": "Image",
                        "relative_path": "images/input-value"
                    }
                ],
                "algorithm_outputs": [
                    {
                        "slug": "another-output-ci-slug",
                        "kind": "Anything",
                        "super_kind": "File",
                        "relative_path": "output-value.json"
                    }
                ]
            }
        ]
    }
}

Alternatively, you generate a pack by providing a JSON string directly:

grand-challenge-forge --output-dir /tmp '{ "challenge": { "slug": "a-slug"...'

This will output a pack directory in the /tmp directory.

[!NOTE] By default, the forge does quality checks on the pack that may require docker. You can disable these via -n

🏗️ Development

Install locally

Install grand-challenge-forge locally (requires poetry):

git clone https://github.com/DIAGNijmegen/rse-grand-challenge-forge.git
cd rse-grand-challenge-forge
poetry install
poetry run grand-challenge-forge --help

Pre-commit hooks

Several linters and stylers run to check the formatting during continuous integration. Ensure they are run before committing by installing pre-commit.

Running Tests

use tox to run all tests across all supported python versions:

pip install tox
tox

Dependencies

Under the hood grand-challenge-forge uses:

• Click
  • a composable command line interface toolkit
• Cookiecutter
  • a utility that creates projects from project templates