variantgrid_api

Python API client for VariantGrid Open source Variant database and analysis platform

See changelog

Install

python3 -m pip install variantgrid_api

Example

from variantgrid_api.api_client import VariantGridAPI
from variantgrid_api.data_models import EnrichmentKit

api = VariantGridAPI(server="https://variantgrid.com", api_token="YOUR_API_TOKEN")
enrichment_kit = EnrichmentKit(name="idt_haem", version=1)
result = api.create_enrichment_kit(enrichment_kit)

Annotate a VCF and download it back

Upload a VCF, have VariantGrid import + annotate any novel variants, then download the cohort-level annotated export (all samples, single-sample VCFs included). Annotation can take a while, so the quickest way in is the vg_api command line tool: the first call uploads, and running the same command again downloads the result once it's ready.

$ export VARIANTGRID_API_TOKEN=YOUR_API_TOKEN
$ vg_api annotate_vcf input.vcf.gz -o results/
Uploaded input.vcf.gz (id=13256).
Annotating input.vcf.gz - run the same command again later to download.

$ vg_api annotate_vcf input.vcf.gz -o results/        # still annotating
input.vcf.gz: not ready yet - pipeline Success, import 100.0%, 2 annotation runs remaining. Run the same command again later.

$ vg_api annotate_vcf input.vcf.gz -o results/        # once it's done
Annotated vcf written to results/input.vcf_annotated_v254_GRCh38.vcf.gz

Add --verbose to log each request, response and the full upload status to stderr.

From Python it's upload_file() / poll_upload_status() / download_annotated(), or the blocking annotate_vcf() one-liner. See Annotate a VCF on the wiki for batches, the submit-now/download-later pattern, and all the options.

Patients, specimens and extractions

VariantGrid can record which patient, specimen and extraction a lab's sequencing came from. This needs a server at or after SACGF/variantgrid#1716.

from variantgrid_api.data_models import Patient, Specimen, Extraction, ExternalReference, NucleicAcid

api.create_patient(Patient(patient_code="C0000001"))
api.create_specimen(Specimen(patient="C0000001", reference_id="2600000001"))
api.create_extraction(Extraction(specimen="2600000001", reference_id="2600000001C",
                                 nucleic_acid_source=NucleicAcid.DNA))
api.upload_file("sample.vcf.gz", path=None,
                metadata={"extraction": "2600000001C", "genome_build": "GRCh37"})

A bare string names a record by its local reference. Use ExternalReference(code=..., external_type=...) to name it by a LIMS identifier instead. See examples/example_tso500.py for a full run.

Once the sample sheet is registered, link its sequencing samples to their extractions in one call, keyed on sample name (what SequencingSampleLookup.sample_name holds):

api.link_sequencing_sample_extractions(sample_sheet_lookup, {"sample_1": "2600000001C",
                                                             "sample_2": "2600000002C"})

It returns a list with one {"sequencing_sample", "match_status", "match_error", "extraction"} per sample. An extraction the server doesn't have yet is Pending for that sample only (HTTP 202) and attaches itself once the extraction is created; a sample name that isn't on the sheet is a 400 for the whole batch. On a server from before SACGF/variantgrid#2176 (no link_extractions_bulk feature) the client makes one link_sequencing_sample_extraction call per sample instead, so pipelines can use the bulk call unconditionally. link_sequencing_sample_extraction is still there for a single late link. See examples/example_haem_20_999.py.

BAMs and CRAMs

A sequencing sample can have several alignment files, eg a BAM and its recalibrated BAM, or a BAM plus a CRAM. List them in SequencingFile.alignment_files (and QC.alignment_files), the file the variant caller ran on first:

from variantgrid_api.data_models import AlignmentFile, SequencingFile, SingleSampleVCF

SequencingFile(sample_name="sample_1",
               alignment_files=[AlignmentFile(path="/data/sample_1.bam", aligner=aligner),
                                AlignmentFile(path="/data/sample_1.cram", aligner=aligner)],
               vcf_files=[SingleSampleVCF(path="/data/sample_1.vcf.gz", variant_caller=variant_caller)])

AlignmentFile.file_type (AlignmentFileType.BAM / CRAM) is optional and inferred from the path. A server without the alignment_files feature gets the old wire format instead: one sequencing file record per alignment file as bam_file, and a QC's first alignment file as its bam_file. A CRAM there also needs the cram_alignment_files feature. BamFile and the bam_file fields still work but are deprecated.

Talking to more than one VariantGrid version

Servers of different ages accept different calls. The client asks the server which features it has (GET api/v1/capabilities, fetched once on first use), and each call that needs a newer server checks that list first: the patient / specimen / extraction calls, specimen measures, link_sequencing_sample_extraction and link_sequencing_sample_extractions, the annotate flow (poll_upload_status, wait_for_annotation, download_annotated, annotate_vcf), and upload_file with metadata or file_type. A server without the capabilities endpoint counts as legacy and reports no features.

By default an unsupported call raises UnsupportedFeatureError. To run the same code against old and new servers, skip those calls instead: they log a warning and return None. Two calls downgrade rather than skip when the server has the older form of the feature: upload_file on a server without upload metadata drops the metadata and still uploads the file, and link_sequencing_sample_extractions on a server with only link_extraction makes one call per sample.

from variantgrid_api.api_client import VariantGridAPI, UnsupportedFeaturePolicy

api = VariantGridAPI(server, api_token, unsupported_feature_policy=UnsupportedFeaturePolicy.SKIP)
api.create_patient(patient)        # skipped on a server without patients

# Uploads only when the server has an importer for this file type
api.upload_file("sample_CombinedVariantOutput.tsv", path=None,
                file_type="dragen_tso500_combined_variant_output")

Where the fallback isn't simply "do nothing", branch on api.supports("feature") or api.accepts_upload("file_type"). api.capabilities.version is useful for logging which server you reached. For tests, MockVariantGridAPI(capabilities=ServerCapabilities.LEGACY) behaves like an old server.

DRAGEN TSO 500 run-level files

A TSO 500 run's CombinedVariantOutput.tsv (the pair's TMB / MSI / GIS) and MetricsOutput.tsv (library QC) are uploaded with the run's name as sequencing_run metadata, the only metadata they take:

from variantgrid_api.data_models import UploadFileType

api.upload_file("ExampleSample_CombinedVariantOutput.tsv", path=None,
                metadata={"sequencing_run": sequencing_run.name},
                file_type=UploadFileType.DRAGEN_TSO500_COMBINED_VARIANT_OUTPUT)
api.upload_file("MetricsOutput.tsv", path=None, metadata={"sequencing_run": sequencing_run.name},
                file_type=UploadFileType.DRAGEN_TSO500_METRICS_OUTPUT)

• MetricsOutput requires sequencing_run - the file never names its run.
• CombinedVariantOutput names its run 'NA', so the server takes the run from sequencing_run. Without it, the server uses the registered run whose current sample sheet names the pair's sample IDs, and if no registered run names the pair the import fails. Register the run and sample sheet first, and send sequencing_run anyway.

The server reads TMB / MSI / GIS from the CombinedVariantOutput (SACGF/variantgrid#1904), which replaced specimen measures. See examples/example_tso500.py for a full run.

Permission to write sequencing data

Posts to the server's seqauto/api/ endpoints - the sequencing create_* calls from create_experiment through create_sequencing_data and the QC calls, and the link_sequencing_sample_extraction(s) calls - need a superuser's API token, or one belonging to a member of the server's SeqAuto write group (seqauto_api_write unless the server's SEQAUTO_API_WRITE_GROUP setting says otherwise). Anyone else gets a 403 (requests.HTTPError). Reads, such as sequencing_run_has_vcf(), need only a valid token.

Testing

# Install required testing packages
python3 -m pip install -e ".[test]"
python3 -m pytest --cov=variantgrid_api