ska-sdc2-scoring-utils 1.0.0

Utility scripts for interacting with SKA SDC2 scoring service.

Science Data Challenge 2 scoring service CLI

This package contains two simple command line interface (CLI) applications for interaction with the SDC2 online scoring service:

• sdc2-score : A tool for upload and check the status of submissions.
• sdc2-score-admin: A tool managing SDC2 users and groups.

Installation

This package can be installed with:

pip install ska-sdc2-scoring-utils

Using the sdc2-score command

Create a submission and generate its SUBMISSION_ID using

sdc2-score create CATALOGUE_FILE

Retrieve information about the submission using

sdc2-score info SUBMISSION_ID

All submissions belonging to a team can be printed using

sdc2-score ls

By default, all commands are run against the 'full' dataset. The -c flag can be used to run commands against the 'dev' dataset.

Credentials:

Credentials can either be specified using the command line options as follows:

sdc2-score create -u USER -p PASSWORD CATALOGUE_FILE

or are loaded from the following environment variables:

• SDC2_SCORER_USER
• SDC2_SCORER_PASSWORD

If both are supplied, the command line option flags are used.

Summary usage:

Usage:
    sdc2-score leaderboard [--sub-info] [--score-info] [--raw] [-c VERSION]
    sdc2-score create CATALOGUE_FILE [-u USERNAME] [-p PASSWORD] [-c VERSION] [--hidden]
    sdc2-score info SUBMISSION_ID [-c VERSION] [--raw]
    sdc2-score ls [-u USERNAME] [-p PASSWORD] [-n LIMIT] [--score-info] [--raw] [-c VERSION]
    sdc2-score -h | --help | --version

Options:
    --sub-info           Display additional submission information.
    --score-info         Display additional score information.
    --raw                Display the raw response from the SDC scoring service.
    --hidden             If passed, submission will be hidden from leaderboards
    -c VERSION           Set the catalogue version (1.dev, 1.ldev, 1.full) [default: 1.full].
    -n LIMIT             Maximum number of submissions to list [default: 20].
    -u USER              Scoring service username.
    -p PASSWORD          Scoring service user password.
    -h, --help           Show this screen.
    --version            Show the version.

Arguments
    CATALOGUE_FILE      Path to catalogue file to submit.
    SUBMISSION_ID       Submission ID, minimally matched.

Using the sdc2-score-admin command

The admin client is intended for SDC2 challenge administrators and requires that the following environment variables are set:

• SDC2_SCORER_ADMIN_USER
• SDC2_SCORER_ADMIN_PASSWORD

Once these are set, the basic usage settings are as follows:

Usage:
    sdc2-score-admin group ls [-u] [--raw]
    sdc2-score-admin group add GROUP_NAME...
    sdc2-score-admin group rm GROUP_ID...
    sdc2-score-admin user ls [--raw]
    sdc2-score-admin user add FIRST_NAME LAST_NAME USERNAME EMAIL GROUP [PASSWORD]
    sdc2-score-admin user rm USER_ID...
    sdc2-score-admin submission ls [-n LIMIT] [-c VERSION] [--raw]
    sdc2-score-admin submission rm SUBMISSION_ID [-c VERSION]
    sdc2-score-admin (-h | --help | --version)

Options:
    -u                   Show users in the group.
    --raw                Display unformatted API response JSON object.
    -n LIMIT             Maximum number of submissions to show [default: 20].
    -c VERSION           Catalogue version (1.full, 1.dev) [default: 1.full].
    -h, --help           Show this screen.
    --version            Show the version.

Arguments:
    EMAIL                Email address of the user.
    FIRST_NAME           User's first name.
    GROUP_ID             Group name or ID, minimally matched.
    GROUP_NAME           Name of the group.
    LAST_NAME            User's last name.
    PASSWORD             Password for the user. Optional, autogenerated if not set.
    SUBMISSION_ID        UUID for a submission, minimally matched
    USER_ID              User name or ID, minimally matched.
    USERNAME             User's user name

Developer notes

For local development, commands can also be run and tested using:

python -m ska_sdc2_scoring_utils.sdc2_score
python -m ska_sdc2_scoring_utils.sdc2_score_admin