HiFi Solves Human WGS data uploader

As part of the HiFi Solves Consortium, organizations will run their sequencing data through PacBio's Human Whole Genome Sequencing (WGS) pipeline.

This package handles uploading all required raw data to the organization's cloud and staging the sample metadata that describes it. Output files are automatically ingested into Publisher and made available on hifisolves.org.

Submitting workflow runs is not part of this package. Once your data is uploaded, submit runs from Instruments. The workflow itself, along with its version, defaults and transformations, is registered in your namespace by DNAstack rather than by this tool, so there is nothing to configure here.

Requirements

• python3.13+
• Credentials for the relevant backend (supported backends: AWS, Azure, GCP)

Installation

python3 -m pip install hifi-solves-run-humanwgs

Command name

The command is hifisolves-upload.

Two earlier names, hifisolves-ingest and run-humanwgs, still work and do exactly the same thing, so existing scripts do not need changing. Both predate the move of workflow submission to Instruments, and run-humanwgs in particular now names something this tool no longer does. Invoking either prints a notice pointing at hifisolves-upload; prefer the new name in anything written from here on.

Script usage

Arguments

usage: hifisolves-upload [-h] [-v] [-s SAMPLE_INFO] [-m MOVIE_BAMS] [-c FAM_INFO] -b {AWS,GCP,AZURE} -o ORGANIZATION [--verbose]
                         [--families FAMILIES]

Upload genomics data and sample metadata in preparation for running PacBio's official Human WGS pipeline. Workflow runs are submitted from
Instruments, not from this tool.

options:
  -h, --help            show this help message and exit
  -v, --version         Program version
  -b, --backend {AWS,GCP,AZURE}
                        Backend where infrastructure is set up
  -o, --organization ORGANIZATION
                        Organization identifier; used to infer bucket names
  --verbose             Use verbose logging
  --families FAMILIES   Comma-separated list of family IDs to process. If not specified, all families in the sample info file will be processed.

Sample information:
  Provide either --sample-info, OR both --movie-bams and --fam-info

  -s, --sample-info SAMPLE_INFO
                        Path to sample info CSV or TSV. This file should have columns [family_id, sample_id, movie_bams, father_id, mother_id, sex].
                        See documentation for more information on the format of this file.
  -m, --movie-bams MOVIE_BAMS
                        Path to movie bams CSV or TSV. This file should have columns [sample_id, movie_bams]. Repeated rows for each sample can be
                        added if the sample has more than one associated bam.
  -c, --fam-info FAM_INFO
                        Path to family information. This file should have columns [family_id, sample_id, father_id, mother_id, sex]. It can
                        optionally have additional phenotype columns (columns 6-end), but this information will not be used.

Sample info file

The sample info file defines the set of samples that will be run through the workflow. The workflow can either be run on individual samples or on families (typically trios, where sequencing data exists for the mother, father, and child). Samples that share a family ID belong to one workflow run; that grouping is what you will select when submitting from Instruments.

This information is organized into a CSV file with the following columns:

† There can be more than one BAM for a sample. If this is the case, a new row should be generated for each additional movie_bam; family_id and sample_id must be set for these fields, but information from other fields need not be repeated.

Example sample info files

All samples for all runs (singleton and family-based runs) may be included in a single sample info file; every unique family ID in the sample_info CSV corresponds to a separate workflow run.

Singleton

Here we have a single sample, HG005, with two associated movie bams found at the local paths bams/HG005_1.hifi_reads.bam and bams/HG005_2.hifi_reads.bam. The sample is being run alone so father_id and mother_id are left blank. Sex information only needs to be included once and can be omitted for further rows associated with the same sample_id.

family_id,sample_id,movie_bams,father_id,mother_id,sex
HG005,HG005,bams/HG005_1.hifi_reads.bam,,,MALE
HG005,HG005,bams/HG005_2.hifi_reads.bam,,,

Trio

Here we have a trio of samples: a child (HG005), father (HG006), and mother (HG007). The mother and father samples have several associated movie_bams, so there are multiple rows for each.

family_id,sample_id,movie_bams,father_id,mother_id,sex
hg005-trio,HG005,bams/HG005_1.hifi_reads.bam,HG006,HG007,MALE
hg005-trio,HG006,bams/HG006_1.hifi_reads.bam,,,MALE
hg005-trio,HG006,bams/HG006_2.hifi_reads.bam,,,
hg005-trio,HG007,bams/HG007_1.hifi_reads.bam,,,FEMALE
hg005-trio,HG007,bams/HG007_2.hifi_reads.bam,,,
hg005-trio,HG007,bams/HG007_3.hifi_reads.bam,,,

Alternative to the sample info file - --movie-bams and --fam-info

Instead of providing a --sample-info file, you may choose to organize your information into two separate files: --movie-bams, and --fam-info.

Movie bams

Provided using the --movie-bams argument.

† There can be more than one movie bam for a sample. If this is the case, a new row should be generated for each additional movie_bam.

Example movie bam file

sample_id,movie_bams
HG005,bams/HG005_1.hifi_reads.bam
HG006,bams/HG006_1.hifi_reads.bam
HG006,bams/HG006_2.hifi_reads.bam
HG007,bams/HG007_1.hifi_reads.bam
HG007,bams/HG007_2.hifi_reads.bam
HG007,bams/HG007_3.hifi_reads.bam

Family information

Provided using the --fam-info argument. This file is related to PLINK's fam info format, with some modifications (namely, a header is required, and there can be multiple (or zero) phenotypes columns; note that phenotype information is discarded here).

Example fam info file

family_id,sample_id,father_id,mother_id,sex,HP:0001250,HP:0001263
hg005-trio,HG005,HG006,HG007,1,2,2
hg005-trio,HG006,,,1,1,1
hg005-trio,HG007,,,2,1,1

Running the script

The script uploads input files and stages sample metadata. It does not submit workflow runs; submit those from Instruments once the upload completes and your samples have been indexed.

--upload-only and --region are still accepted but ignored, so existing invocations do not break. They are hidden from --help and should be dropped from new callers: uploading is now the only mode, and the region was only ever needed to build the workflow inputs that this tool no longer generates. Uploading itself does not use a region.

Environment Variables

WORKBENCH_URL

By default, the script will use workbench.omics.ai as the Workbench URL. You can override this by setting the WORKBENCH_URL environment variable:

export WORKBENCH_URL="custom-workbench.example.com"

Filtering to specific families

Use the --families option to process only a subset of families from your sample info file. This is useful for:

• Uploading data for specific families only
• Staging a subset of families first to check the upload before committing all data

# Process a single family
hifisolves-upload \
    --sample-info sample_info.csv \
    --families "hg005-trio" \
    --backend aws \
    --organization "${ORGANIZATION}"

# Process multiple families
hifisolves-upload \
    --sample-info sample_info.csv \
    --families "hg005-trio,hg006-trio,singleton-1" \
    --backend aws \
    --organization "${ORGANIZATION}"

If any specified family is not found in the sample info file, the script will exit with an error listing the missing families and available families.

Example upload command - AWS

# AWS credentials
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN=""

# Used for naming upload and output buckets
ORGANIZATION=""

hifisolves-upload \
    --sample-info sample_info.csv \
    --backend aws \
    --organization "${ORGANIZATION}"

Example upload command - Azure

# Azure credentials; needs Read, Add, Write, Create, Delete, List
export AZURE_STORAGE_SAS_TOKEN=""

# Used for naming upload and output buckets; this is going to be == the storage account name
ORGANIZATION=""

hifisolves-upload \
    --sample-info sample_info.csv \
    --backend Azure \
    --organization "${ORGANIZATION}"

If you have files already uploaded in the target storage account, their paths may be referenced in the format /<storage_account>/rawdata/path/to/file.

Copying Azure <> Azure

If source files are currently in cloud storage, they can be copied into the target storage account rather than copying from local -> cloud.

BAM URLs in the sample_info CSV file should be in the format /<src_storage_account>/<src_storage_container>/path/to/movie.bam.

An additional env variable, SOURCE_CONTAINER_SAS_TOKEN, should be defined. This SAS token should have Read and List permissions on the source container.

# SAS token for the source bucket (R/L)
export SOURCE_CONTAINER_SAS_TOKEN=""

# SAS token for the destination bucket (R/A/W/C/D/L)
export AZURE_STORAGE_SAS_TOKEN=""

ORGANIZATION=""

hifisolves-upload \
    --sample-info sample_info.csv \
    --backend Azure \
    --organization "${ORGANIZATION}"

Example upload command - GCP

# GCP credentials - GOOGLE_APPLICATION_CREDENTIALS should point towards a JSON file containing service account information
export GOOGLE_APPLICATION_CREDENTIALS=""

# Used for naming upload and output buckets
ORGANIZATION=""

hifisolves-upload \
	--sample-info sample_info.csv \
	--backend gcp \
	--organization "${ORGANIZATION}"

Development

Tests

Note that you will need access to have the active cloud-specific credentials below set for the various cloud backends to run the tests.

See this secret for the values you'll need to set here.

# Required AWS credentials
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN=""
# alternatively - just AWS_PROFILE
# export AWS_PROFILE=""

# Required Azure credentials
_Note that these credentials will eventually expire_
# R/A/W/C/D/L on destination container
export AZURE_STORAGE_SAS_TOKEN=""
# R/L on src container
export SOURCE_CONTAINER_SAS_TOKEN=""

# Path to service account JSON
export GOOGLE_APPLICATION_CREDENTIALS=""

python3 -m unittest discover -b -s tests

Setting the workflow version

The workflow version is comprised of two parts:

• WORKFLOW_VERSION: This is the version of the HumanWGS workflow in use; it should refer to a specific tagged version of this workflow
• WORKFLOW_SUB_VERSION: This is the revision of the HumanWGS workflow, and is used when we need to make changes to the workflow that are not present in PacBio's official workflow

Changing any part of either of these versions will force a new version of the workflow to be created in the user's namespace. Changing the WORKFLOW_VERSION or the major version of the WORKFLOW_SUB_VERSION will also result in the script resetting the run status for all samples back to unprocessed; changing just the minor or patch version of WORKFLOW_SUB_VERSION will allow any run with the same major WORKFLOW_SUB_VERSION to be picked up & used to determine run status. This allows small bugfixes to be made to the workflow in order to enable running samples without the need to fully reprocess all samples when a new version of the workflow is registered.

Building Packages

A Makefile resides in the root directory of the package. Rules have been created to clean, build, release and push a package release. The following sections provide details on building and pushing the code to a repository

Building the package

make build

This creates a Python package to be uploaded to the Python Package Index (PyPI: pypi.org).

Building and pushing the Docker image

make docker-build

This rule builds a Docker image and pushes it to a container registry.