HiFi Solves Human WGS data uploader

As part of the HiFi Solves Consortium, organizations will run their sequencing data through PacBio's Human Whole Genome Sequencing (WGS) pipeline.

This package handles uploading all required raw data to the organization's cloud and staging the sample metadata that describes it. Output files are automatically ingested into Publisher and made available on hifisolves.org.

Submitting workflow runs is not part of this package. Once your data is uploaded, submit runs from Instruments. The workflow itself, along with its version, defaults and transformations, is registered in your namespace by DNAstack rather than by this tool, so there is nothing to configure here.

Requirements

• python3.11+
• Version 3.0.0 or newer of this package
• Credentials for the relevant backend (supported backends: AWS, Azure, GCP)

Installation

python3 -m pip install hifi-solves-run-humanwgs

Minimum version

Do not use any release below 3.0.0. Those versions register the workflow and submit runs themselves. Both are now managed by DNAstack in your namespace, so an older release will register competing workflow versions and submit runs it should not.

Check what you have with hifisolves-upload --version, and upgrade with python3 -m pip install --upgrade hifi-solves-run-humanwgs. Releases below 3.0.0 are yanked from PyPI, so a plain install or upgrade will not select one, but an exact pin in an existing script or lockfile still can.

Command name

The command is hifisolves-upload.

Two earlier names, hifisolves-ingest and run-humanwgs, still work and do exactly the same thing, so existing scripts do not need changing. Both predate the move of workflow submission to Instruments, and run-humanwgs in particular now names something this tool no longer does. Invoking either prints a notice pointing at hifisolves-upload; prefer the new name in anything written from here on.

Script usage

Arguments

usage: hifisolves-upload [-h] [-v] -s SAMPLE_INFO -b {AWS,GCP,AZURE}
                         -o ORGANIZATION [--verbose]

Upload genomics data and sample metadata in preparation for running PacBio's
official Human WGS pipeline. Workflow runs are submitted from Instruments, not
from this tool.

options:
  -h, --help            show this help message and exit
  -v, --version         Program version
  -s, --sample-info SAMPLE_INFO
                        Path to sample info CSV or TSV. This file should have
                        columns [sample_id, hifi_reads]. Any other columns are
                        ignored. See documentation for more information on the
                        format of this file.
  -b, --backend {AWS,GCP,AZURE}
                        Backend where infrastructure is set up
  -o, --organization ORGANIZATION
                        Organization identifier; used to infer bucket names
  --verbose             Use verbose logging

Sample info file

The sample info file defines the set of samples that will be uploaded.

This information is organized into a CSV or TSV file with the following columns:

† There can be more than one BAM for a sample. If so, add a row per BAM, repeating the sample_id. movie_bams is still accepted as a column name in place of hifi_reads, but not both in the same file.

Example sample info files

Example - single sample

Here we have a single sample, HG005, with two associated BAMs found at the local paths bams/HG005_1.hifi_reads.bam and bams/HG005_2.hifi_reads.bam.

sample_id,hifi_reads
HG005,bams/HG005_1.hifi_reads.bam
HG005,bams/HG005_2.hifi_reads.bam

Example - several samples

Here we have three samples. HG006 and HG007 each have several BAMs, so there are multiple rows for each.

sample_id,hifi_reads
HG005,bams/HG005_1.hifi_reads.bam
HG006,bams/HG006_1.hifi_reads.bam
HG006,bams/HG006_2.hifi_reads.bam
HG007,bams/HG007_1.hifi_reads.bam
HG007,bams/HG007_2.hifi_reads.bam
HG007,bams/HG007_3.hifi_reads.bam

Running the script

The script uploads input files and stages sample metadata. It does not submit workflow runs; submit those from Instruments once the upload completes and your samples have been indexed.

The script is idempotent, so it is safe to re-run. Any BAM already present at the destination is skipped rather than uploaded again, so a re-run after an interruption transfers only what is still missing.

--upload-only and --region are still accepted but ignored, so existing invocations do not break. They are hidden from --help and should be dropped from new callers: uploading is now the only mode, and the region was only ever needed to build the workflow inputs that this tool no longer generates. Uploading itself does not use a region.

Environment Variables

WORKBENCH_URL

By default, the script will use workbench.omics.ai as the Workbench URL. You can override this by setting the WORKBENCH_URL environment variable:

export WORKBENCH_URL="custom-workbench.example.com"

Example upload command - AWS

# AWS credentials
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN=""

# Used for naming upload and output buckets
ORGANIZATION=""

hifisolves-upload \
    --sample-info sample_info.csv \
    --backend aws \
    --organization "${ORGANIZATION}"

Example upload command - Azure

# Azure credentials; needs Read, Add, Write, Create, Delete, List
export AZURE_STORAGE_SAS_TOKEN=""

# Used for naming upload and output buckets; this is going to be == the storage account name
ORGANIZATION=""

hifisolves-upload \
    --sample-info sample_info.csv \
    --backend Azure \
    --organization "${ORGANIZATION}"

If you have files already uploaded in the target storage account, their paths may be referenced in the format /<storage_account>/rawdata/path/to/file.

Copying Azure <> Azure

If source files are currently in cloud storage, they can be copied into the target storage account rather than copying from local -> cloud.

BAM URLs in the sample_info CSV file should be in the format /<src_storage_account>/<src_storage_container>/path/to/movie.bam.

An additional env variable, SOURCE_CONTAINER_SAS_TOKEN, should be defined. This SAS token should have Read and List permissions on the source container.

# SAS token for the source bucket (R/L)
export SOURCE_CONTAINER_SAS_TOKEN=""

# SAS token for the destination bucket (R/A/W/C/D/L)
export AZURE_STORAGE_SAS_TOKEN=""

ORGANIZATION=""

hifisolves-upload \
    --sample-info sample_info.csv \
    --backend Azure \
    --organization "${ORGANIZATION}"

Example upload command - GCP

# GCP credentials - GOOGLE_APPLICATION_CREDENTIALS should point towards a JSON file containing service account information
export GOOGLE_APPLICATION_CREDENTIALS=""

# Used for naming upload and output buckets
ORGANIZATION=""

hifisolves-upload \
	--sample-info sample_info.csv \
	--backend gcp \
	--organization "${ORGANIZATION}"

Development

Tests

Note that you will need access to have the active cloud-specific credentials below set for the various cloud backends to run the tests.

See this secret for the values you'll need to set here.

# Required AWS credentials
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN=""
# alternatively - just AWS_PROFILE
# export AWS_PROFILE=""

# Required Azure credentials
_Note that these credentials will eventually expire_
# R/A/W/C/D/L on destination container
export AZURE_STORAGE_SAS_TOKEN=""
# R/L on src container
export SOURCE_CONTAINER_SAS_TOKEN=""

# Path to service account JSON
export GOOGLE_APPLICATION_CREDENTIALS=""

python3 -m unittest discover -b -s tests

Building Packages

A Makefile resides in the root directory of the package. Rules have been created to clean, build, release and push a package release. The following sections provide details on building and pushing the code to a repository

Building the package

make build

This creates a Python package to be uploaded to the Python Package Index (PyPI: pypi.org).

Building and pushing the Docker image

make docker-build

This rule builds a Docker image and pushes it to a container registry.