The command line and Python client for GPAS.
Project description
GPAS Client
The command line interface for the GPAS platform.
The client enables privacy-preserving sequence data submission and retrieval of analytical output files. Prior to upload, sample identifiers are anonymised and human host sequences are removed. A computer with Linux or MacOS is required to use the client. When running human read removal prior to upload a computer with a modern multi-core processor and at least 16GB of RAM is recommended.
Install
There are two recommended methods for installing the GPAS Client, either by using the popular package and environment manager Conda or by using our publicly available Docker container which we build at release time.
Installing Miniconda
If a Conda package manager is already installed, skip to Installing the client, otherwise the following instructions have been taken from the Miniconda install process documentation
Installing Miniconda on Linux
In a terminal console, install Miniconda with the following instructions and accepting default options:
```bash
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
```
Installing Miniconda on MacOS
The client requires the Conda platform to be using x86_64 when creating the environment.
-
If your Mac has an Apple processor, using Terminal, firstly run:
mkdir -p ~/miniconda3 curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh -o ~/miniconda3/miniconda.sh bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 rm -rf ~/miniconda3/miniconda.sh
-
Initialise Miniconda using either of the following commands depending on your Shell (Bash|ZSH)
~/miniconda3/bin/conda init bash ~/miniconda3/bin/conda init zsh
Installing or updating the client with Miniconda
The client has at least one dependency that requires bioconda, which itself
depends on conda-forge. Note that for the conda create step (see below), installation can be very slow,
so please leave it running. For more verbose output, you can add the -v or -vv flags, though
it is not recommended to show the full debug output with -vvv as this has been seen to lead to OOM errors.
Linux
conda create -y -n gpas -c conda-forge -c bioconda hostile==1.1.0
conda activate gpas
pip install --upgrade gpas
MacOS
Please note the additional argument --platform osx-64 in this command, compared to the above.
conda create --platform osx-64 -y -n gpas -c conda-forge -c bioconda hostile==1.1.0
conda activate gpas
pip install --upgrade gpas
A simple test to verify installation would be to run a version check:
gpas --version
gpas auth
GPAS client version: 3.0.1
Usage: gpas auth [OPTIONS]
Authenticate with GPAS.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --host TEXT API hostname (for development) │
│ --check-expiry Check for a current token and print the expiry if exists │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Most actions with the GPAS CLI require that the user have first authenticated with the GPAS server with their login credentials. Upon successfully authentication, a bearer token is stored in the user's home directory and will be used on subsequent CLI usage.
The token is valid for 7 days and a new token can be retrieved at anytime.
Usage
Running gpas auth will ask for your username and password for GPAS, your password will not be shown
in the terminal session.
$ gpas auth
14:04:31 INFO: GPAS client version 2.0.0rc1
14:04:31 INFO: Authenticating with portal.gpas.global
Enter your username: gpas-user@eit.org
Enter your password:
14:04:50 INFO: Authenticated (/Users/<user>/.config/gpas/tokens/portal.gpas.global.json)
Troubleshooting Authentication
How do I get an account for GPAS?
Creating a Personal Account:
Navigate to GPAS and click on “Sign Up”. Follow the instructions to create a user account.
Shortly after filling out the form you'll receive a verification email. Click the link in the email to verify your account and email address. If you don’t receive the email, please contact gpas.support@eit.org.
You are now ready to start using GPAS.
What happens when my token expires?
If you haven't already retrieved a token, you will receive the following error message.
$ gpas upload tests/data/illumina-2.csv
12:46:42 INFO: GPAS client version 2.0.0rc1
12:46:43 INFO: Getting credit balance for portal.gpas.global
12:46:43 ERROR: FileNotFoundError: Token not found at /Users/<user>/.config/gpas/tokens/portal.gpas.global.json, have you authenticated?
If your token is invalid or expired, you will receive the following message
14:03:26 INFO: GPAS client version 2.0.0rc1
14:03:26 ERROR: AuthorizationError: Authorization checks failed! Please re-authenticate with `gpas auth` and
try again.
How can I check my token expiry before long running processes?
You can check the expiry of your token with the following command:
$ gpas auth --check-expiry
14:05:52 INFO: GPAS client version 2.0.0rc1
14:05:52 INFO: Current token for portal.gpas.global expires at 2024-08-13 14:04:50.672085
gpas balance
GPAS client version: 3.0.1
Usage: gpas balance [OPTIONS]
Check your GPAS account balance.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --host TEXT API hostname (for development) │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Credits are required to upload samples and initiate the analysis process. Users can check their credit balance in the
header of the GPAS Portal or by using the gpas balance command when logged in.
Usage
gpas balance
15:56:56 INFO: GPAS client version 2.0.0
15:56:56 INFO: Getting credit balance for portal.gpas.global
15:56:57 INFO: Your remaining account balance is 1000 credits
gpas upload
GPAS client version: 3.0.1
Usage: gpas upload [OPTIONS] UPLOAD_CSV
Validate, decontaminate and upload reads to GPAS.
Creates a mapping CSV file which can be used to download output files with original sample names.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --threads INTEGER Number of alignment threads used during decontamination │
│ --save Retain decontaminated reads after upload completion │
│ --host TEXT API hostname (for development) │
│ --skip-fastq-check Skip checking FASTQ files for validity │
│ --skip-decontamination Skips decontamination step prior to upload │
│ --output-dir DIRECTORY Output directory for the cleaned FastQ files, defaults to clean-files/<timestamp>/. │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Where samples may contain human reads we strongly recommend using the provided decontamination functionality. This is best practice to minimise the risk of personally identifiable information being uploaded to the cloud.
The upload command performs metadata validation and client-side removal of human reads for each of your samples, before uploading sequences to GPAS for analysis.
To generate a CSV file to use with this command see the build-csv documentation.
Credits
Credits are required to upload samples and initiate the analysis process. Users can check their credit balance in the
header of the GPAS Portal or by using the gpas balance command. More information can be found in the
gpas balance section.
Each sample for Mycobacterium genomic sequencing will require 10 credits whereas SARS-CoV-2 sample sequencing will require 1 credits. During the upload command process, a balance check is performed to ensure the user has enough credits for the number of samples in the batch. Credits are then deducted when sample files are successfully uploaded and ready for processing.
Human Read Removal
A 4GB human genome index is downloaded the first time you run gpas upload. If for any reason this is interrupted,
run the upload command again. Upload will not proceed until the index has been downloaded and passed an integrity
check. You may optionally download the index ahead of time using the command gpas download-index.
By default, the upload command will first run gpas decontaminate to attempt to remove human reads prior to
uploading the input samples to GPAS, this option can be overridden but only do so if you're aware of the risks
stated above.
To retain the decontaminated FASTQ files uploaded to GPAS, include the optional --save flag. To perform
decontamination without uploading anything, use the gpas decontaminate command.
During upload, a mapping CSV is created (e.g. a5w2e8.mapping.csv) linking your local sample names with their randomly
generated remote names. Keep this file safe, as it is useful for downloading and relinking results later, it cannot be
recreated after this step without re-uploading the same samples again.
Usage
gpas upload my-first-batch.csv
15:41:57 INFO: GPAS client version 2.0.0
15:41:57 INFO: Getting credit balance for portal.gpas.global
15:41:59 INFO: Your remaining account balance is 1000 credits
15:41:59 INFO: Performing FastQ checks and gathering total reads
15:41:59 INFO: Calculating read count in: /Users/<user>/samples/ERR4809187_1.fastq.gz
15:42:00 INFO: Calculating read count in: /Users/<user>/samples/ERR4809187_2.fastq.gz
15:42:02 INFO: 3958206.0 reads in FASTQ file
15:42:02 INFO: Removing human reads from ILLUMINA FastQ files and storing in /Users/<user>/code/gpas/client
15:42:02 INFO: Hostile version 1.1.0. Mode: paired short read (Bowtie2)
15:42:02 INFO: Found cached standard index human-t2t-hla-argos985-mycob140
15:42:02 INFO: Cleaning...
15:43:39 INFO: Cleaning complete
15:43:39 INFO: The mapping file gx5y5p.mapping.csv has been created.
15:43:39 INFO: You can monitor the progress of your batch in GPAS here: "..."
15:43:39 INFO: Uploading my-first-sample
15:45:27 INFO: Uploaded 66433ffc-3c10-4576-8502-56b4805c7ecc_1.fastq.gz
15:45:27 INFO: Uploading my-first-sample
15:49:20 INFO: Uploaded 66433ffc-3c10-4576-8502-56b4805c7ecc_2.fastq.gz
15:49:21 INFO: Upload complete. Created gx5y5p.mapping.csv (keep this safe)
15:49:21 INFO: Getting credit balance for portal.gpas.global
15:49:23 INFO: Your remaining account balance is 990 credits
gpas upload --skip-decontamination my-first-batch.csv
15:41:57 INFO: GPAS client version 2.0.0
15:41:57 INFO: Getting credit balance for portal.gpas.global
15:41:59 INFO: Your remaining account balance is 1000 credits
15:41:59 INFO: Performing FastQ checks and gathering total reads
15:41:59 INFO: Calculating read count in: /Users/<user>/samples/ERR4809187_1.fastq.gz
15:42:00 INFO: Calculating read count in: /Users/<user>/samples/ERR4809187_2.fastq.gz
15:42:02 INFO: 3958206.0 reads in FASTQ file
15:42:02 INFO: Removing human reads from ILLUMINA FastQ files and storing in /Users/<user>/code/gpas/client
15:43:39 INFO: The mapping file gx5y5p.mapping.csv has been created.
15:43:39 INFO: You can monitor the progress of your batch in GPAS here: "..."
15:43:39 INFO: Uploading my-first-sample
15:45:27 INFO: Uploaded 66433ffc-3c10-4576-8502-56b4805c7ecc_1.fastq.gz
15:45:27 INFO: Uploading my-first-sample
15:49:20 INFO: Uploaded 66433ffc-3c10-4576-8502-56b4805c7ecc_2.fastq.gz
15:49:21 INFO: Upload complete. Created gx5y5p.mapping.csv (keep this safe)
15:49:21 INFO: Getting credit balance for portal.gpas.global
15:49:23 INFO: Your remaining account balance is 990 credits
gpas build-csv
GPAS client version: 3.0.1
Usage: gpas build-csv [OPTIONS] SAMPLES_FOLDER
Command to create upload csv from SAMPLES_FOLDER containing sample fastqs.
Use max_batch_size to split into multiple separate upload csvs.
Adjust the read_suffix parameters to match the file endings for your read files.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * --output-csv FILE Path to output CSV file [required] │
│ * --batch-name TEXT Batch name [required] │
│ * --collection-date [%Y-%m-%d] Collection date (YYYY-MM-DD) [default: 2025-11-12] │
│ [required] │
│ * --country TEXT 3-letter Country Code [required] │
│ --instrument-platform [illumina|ont] Sequencing technology [default: illumina] │
│ --subdivision TEXT Subdivision in ISO 3166-2 format, see │
│ https://www.iso.org/standard/72483.html [default: ""] │
│ --district TEXT District [default: ""] │
│ --specimen-organism [mycobacteria|sars-cov-2] Specimen organism [default: mycobacteria] │
│ --amplicon-scheme [|Automatic Amplicon scheme, use only when SARS-CoV-2 is the │
│ Detection|COVID-AMPLISEQ-V1|COVID-ARTIC-V3|COVID-ARTIC specimen organism │
│ -V4.1|COVID-ARTIC-V5.0-5.2.0_1200|COVID-ARTIC-V5.0-5.3 │
│ .2_400|COVID-MIDNIGHT-1200|COVID-VARSKIP-V1a-2b] │
│ --ont_read_suffix TEXT Read file ending for ONT fastq files [default: │
│ .fastq.gz] │
│ --illumina_read1_suffix TEXT Read file ending for Illumina read 1 files [default: │
│ _1.fastq.gz] │
│ --illumina_read2_suffix TEXT Read file ending for Illumina read 2 files [default: │
│ _2.fastq.gz] │
│ --max-batch-size INTEGER [default: 50] │
│ --host TEXT API hostname (for development) │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
This command generates a CSV from a given directory of fastq sample files. An example of such a CSV file is given in the assets directory. A CSV file in this format is required to run the gpas upload command.
Note: the CSV file must be located in the same directory as the sample.fastq files to be used with the upload command.
Usage
gpas build-csv ~/Downloads/samples --batch-name <batch-name> --country <three-letter-country-code>
for ex:
gpas build-csv ~/Downloads/samples --batch-name mybatch123 --country GBR
This will generate a CSV file in the samples folder named upload.csv, prompting users to manually fill in optional fields later (like instrument-platform, amplicon-scheme, etc.). Alternatively, these optional parameters can be passed directly via the CLI rather than filling them in manually later; the example below shows how to include some of these, but for the full list of available options, refer to gpas build-csv --help.
for ex:
gpas build-csv ~/Downloads/samples \
--batch-name mybatch123 \
--country GBR \
--instrument-platform illumina \
--specimen-organism sars-cov-2 \
--amplicon-scheme "Automatic Detection"
gpas decontaminate
GPAS client version: 3.0.1
Usage: gpas decontaminate [OPTIONS] INPUT_CSV
Decontaminate reads from provided csv samples.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --output-dir DIRECTORY Output directory for the cleaned FastQ files, defaults to clean-files/<timestamp>/. │
│ --threads INTEGER Number of alignment threads used during decontamination │
│ --skip-fastq-check Skip checking FASTQ files for validity │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
This command will attempt to remove human reads from a given input CSV file, in the same structure as the input CSV that would be used for uploading to GPAS, an example can be found here.
By default, the processed files will be output in the same directory that the command is run in, but you can choose a
different directory with the --output-dir argument.
Usage
$ gpas decontaminate tests/data/illumina.csv
15:24:39 INFO: GPAS client version 2.0.0rc1
15:24:39 INFO: Performing FastQ checks and gathering total reads
15:24:39 INFO: Calculating read count in: /Users/<user>/code/gpas/client/tests/data/reads/tuberculosis_1_1.fastq
15:24:39 INFO: Calculating read count in: /Users/<user>/code/gpas/client/tests/data/reads/tuberculosis_1_2.fastq
15:24:39 INFO: 2.0 reads in FASTQ file
15:24:39 INFO: Removing human reads from ILLUMINA FastQ files and storing in /Users/<user>/code/gpas/client
15:24:39 INFO: Hostile version 1.1.0. Mode: paired short read (Bowtie2)
15:24:39 INFO: Found cached standard index human-t2t-hla-argos985-mycob140
15:24:39 INFO: Cleaning...
15:24:39 INFO: Cleaning complete
15:24:39 INFO: Human reads removed from input samples and can be found here: /Users/<user>/code/gpas/client
gpas download
GPAS client version: 3.0.1
Usage: gpas download [OPTIONS] SAMPLES
Download input and output files associated with sample IDs or a mapping CSV file.
That are created during upload.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --filenames TEXT Comma-separated list of output filenames to download │
│ --inputs Also download decontaminated input FASTQ file(s) │
│ --output-dir DIRECTORY Output directory for the downloaded files. │
│ --rename/--no-rename Rename downloaded files using sample names when given a mapping CSV │
│ --host TEXT API hostname (for development) │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The download command retrieves the output (and/or input) files associated with a batch of samples given a mapping CSV generated during upload, or one or more sample GUIDs. When a mapping CSV is used, by default downloaded file names are prefixed with the sample names provided at upload. Otherwise, downloaded files are prefixed with the sample GUID.
Usage
# Download the main reports for all samples in a5w2e8.mapping.csv
gpas download a5w2e8.mapping.csv
# Download the main and speciation reports for all samples in a5w2e8.mapping.csv
gpas download a5w2e8.mapping.csv --filenames main_report.json,speciation_report.json
# Download the main report for one sample
gpas download 3bf7d6f9-c883-4273-adc0-93bb96a499f6
# Download the final assembly for one M. tuberculosis sample
gpas download 3bf7d6f9-c883-4273-adc0-93bb96a499f6 --filenames final.fasta
# Download the main report for two samples
gpas download 3bf7d6f9-c883-4273-adc0-93bb96a499f6,6f004868-096b-4587-9d50-b13e09d01882
# Save downloaded files to a specific directory
gpas download a5w2e8.mapping.csv --output-dir results
# Download only input fastqs
gpas download a5w2e8.mapping.csv --inputs --filenames ""
The complete list of --filenames available for download varies by sample, and can be found in the Downloads section of
sample view pages in GPAS.
gpas validate
GPAS client version: 3.0.1
Usage: gpas validate [OPTIONS] UPLOAD_CSV
Validate a given upload CSV.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --host TEXT API hostname (for development) │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The validate command will check that a Batch can be created from a given CSV and if your user account has permission
to upload the samples, the individual FastQ files are then checked for validity. These checks are already performed
by default with the upload command but using this can ensure validity without committing to the subsequent upload
if you're looking to check a CSV during writing it.
gpas query-raw
GPAS client version: 3.0.1
Usage: gpas query-raw [OPTIONS] SAMPLES
Fetch metadata for one or more SAMPLES in JSON format.
SAMPLES should be command separated list of GUIDs or path to mapping CSV.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --host TEXT API hostname (for development) │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The query-raw command fetches either the raw metadata of one more samples given a mapping CSV
generated during upload, or one or more sample GUIDs.
Usage
# Query all available metadata in JSON format
gpas query-raw a5w2e8.mapping.csv
gpas query-status
GPAS client version: 3.0.1
Usage: gpas query-status [OPTIONS] SAMPLES
Fetch processing status for one or more SAMPLES.
SAMPLES should be command separated list of GUIDs or path to mapping CSV.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --output_json Output status in JSON format │
│ --host TEXT API hostname (for development) │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The query-status command fetches the current processing status of one or more samples in a mapping CSV
generated during upload, or one or more sample GUIDs.
Usage
# Query the processing status of all samples in a5w2e8.mapping.csv
gpas query-status a5w2e8.mapping.csv
# Query the processing status of a single sample
gpas query-status 3bf7d6f9-c883-4273-adc0-93bb96a499f6
gpas get-pipelines
GPAS client version: 3.0.1
Usage: gpas get-pipelines [OPTIONS]
Get valid pipelines from the server.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The get-pipelines command fetches the valid pipelines and their versions from the server.
Usage
gpas get-pipelines
gpas get-amplicon-schemes
GPAS client version: 3.0.1
Usage: gpas get-amplicon-schemes [OPTIONS]
Get valid amplicon schemes from the server.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --host TEXT API hostname (for development) │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The get-amplicon-schemes command fetches the valid amplicon schemes from the server.
Usage
gpas get-amplicon-schemes
gpas autocomplete
GPAS client version: 3.0.1
Usage: gpas autocomplete [OPTIONS]
Enable shell autocompletion.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
This command will output the steps required to enable auto-completion in either a Bash or ZSH shell, follow the output to enable autocompletion, this will need to be executed on every new shell session, instructions are provided on how to make this permanent depending on your environment. More information and instructions for other shells can be found in the Click documentation.
Usage
$ gpas autocomplete
Run this command to enable autocompletion:
eval "$(_GPAS_COMPLETE=bash_source gpas)"
Add this to your ~/.bashrc file to enable this permanently:
command -v gpas > /dev/null 2>&1 && eval "$(_GPAS_COMPLETE=bash_source gpas)"
Tab completion can optionally be enabled by adding the lines output by the command to your shell source files.
This will enable the ability to press tab after writing gpas to list possible sub-commands. It can also be used
for sub-command options, if -- is entered prior to pressing tab.
Support
For technical support, please open an issue or contact gpas.support@eit.org
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file gpas-3.0.1.tar.gz.
File metadata
- Download URL: gpas-3.0.1.tar.gz
- Upload date:
- Size: 93.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: python-requests/2.32.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
245beb41360be1e16e71f832d129ccafc3e637ca0f33f838860b326380037f3f
|
|
| MD5 |
35a35c7c231eebe47b9c3d95b8f180b0
|
|
| BLAKE2b-256 |
e5daace5876119c7ea35976d00aa6a1ab5e18c0966a0281b27b54fdf01ff81ea
|
File details
Details for the file gpas-3.0.1-py3-none-any.whl.
File metadata
- Download URL: gpas-3.0.1-py3-none-any.whl
- Upload date:
- Size: 52.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: python-requests/2.32.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f0d2885b9777ea023898e87ffd5781896c2d4dd85b598257c279518b68f2d622
|
|
| MD5 |
1a59e88ddc3391b64e58291535cdc444
|
|
| BLAKE2b-256 |
70e4d89d533a324c3de779b95b5d418f6d538ace0cc533dea1498a28e5c72bf9
|
