PG-SUI

Population Genomic Supervised and Unsupervised Imputation.

About PG-SUI

PG-SUI is a Python 3 API that uses machine learning to impute missing values from population genomic SNP data. There are several supervised and unsupervised machine learning algorithms available to impute missing data, as well as some non-machine learning imputers that are useful.

Below is some general information and a basic tutorial. For more detailed information, see our API Documentation.

How PG-SUI works

PG-SUI fills missing SNP genotypes using unsupervised neural networks, supervised machine-learning models, or deterministic methods. Its neural models learn to reconstruct genotype calls using observed values as training targets. To evaluate an imputer, PG-SUI temporarily masks some observed genotypes, predicts them, and compares the predictions with their known values. After fitting, it fills the genotypes that were missing in the original input. Those calls have no known truth, so they are excluded from accuracy scores.

The following schematic animation illustrates the variational autoencoder (VAE) genotype imputation workflow. Its predictions are illustrative, not measured results.

Unsupervised Imputation Methods

Unsupervised imputers include four neural network models:

• Variational Autoencoder (VAE) 1
  • Encodes genotypes as a latent distribution, then decodes them to reconstruct observed calls. Training excludes originally missing calls from the reconstruction loss; the trained model predicts those calls.
• Autoencoder 2
  • Learns a lower-dimensional representation of the genotypes and reconstructs observed calls. Its predictions fill originally missing positions.
• Nonlinear PCA (NLPCA)
  • Learns per-sample latent representations and a decoder network to reconstruct genotypes.
• Unsupervised backpropagation (UBP)
  • Initializes sample representations with PCA, then refines the decoder and representations in stages.

The animations below show the four neural imputation workflows.

Supervised Imputation Methods

Supervised methods use scikit-learn's IterativeImputer, which is based on multivariate imputation by chained equations (MICE) 3. It predicts missing genotypes at each SNP site using other sites as features. Users can configure how many other SNP features are considered for each prediction. PG-SUI supports these classifiers:

• ImputeRandomForest
• ImputeHistGradientBoosting

See the scikit-learn documentation for more information on IterativeImputer and each of the classifiers.

The animations below illustrate the supervised imputation models.

Non-Machine Learning (Deterministic) Methods

We also include several deterministic options for imputing missing data, including:

• Per-population mode per SNP site
• Overall mode per SNP site
• Reference allele (REF genotype)

The animations below illustrate each deterministic model.

Installing PG-SUI

PG-SUI supports both pip and conda distributions. Both are kept current with up-to-date releases.

Installation with Pip

To install PG-SUI with pip, do the following. It is strongly recommended to install pg-sui in a virtual environment.

python3 -m venv .pgsui-venv
source .pgsui-venv/bin/activate
pip install pg-sui

Installation with Anaconda

To install PG-SUI with Anaconda, do the following:

conda create -n pgsui-env --strict-channel-priority --override-channels \
    -c btmartin721 -c conda-forge -c bioconda \
    python=3.12 pg-sui
conda activate pgsui-env

Docker Container

We also maintain a Docker image that comes with PG-SUI preinstalled. This can be useful for automated workflows such as Nextflow or Snakemake.

docker pull btmartin721/pg-sui:latest
docker run -it --rm btmartin721/pg-sui:latest pg-sui --help

Optional MacOS GUI

PG-SUI ships an optional Electron GUI (Graphical User Interface) wrapper around the Python CLI. Currently for the GUI, only MacOS is supported.

1. Install the Python-side extras (FastAPI/ uvicorn helper) if you want to serve from Python: pip install pg-sui[gui]
2. Install Node.js and fetch the app dependencies: pgsui-gui-setup
3. Launch the graphical interface: pgsui-gui

The GUI shells out to the same CLI underneath, so presets, overrides, and YAML configs behave identically.

Input Data

You can read your input files as a GenotypeData object from the SNPio package. SNPio supports the VCF, PHYLIP, STRUCTURE, and GENEPOP input file formats.

# Import snpio. Automatically installed with pg-sui.
from snpio import VCFReader

# Read in VCF alignment.
# SNPio also supports PHYLIP, STRUCTURE, and GENEPOP input file formats.
data = VCFReader(
    filename="pgsui/example_data/phylogen_subset14K.vcf.gz",
    popmapfile="pgsui/example_data/popmaps/phylogen_nomx.popmap", # optional
    force_popmap=True, # optional
)

Supported Imputation Methods

There are several supported algorithms PG-SUI uses to impute missing data. Each one can be run by calling the corresponding class. You must provide a GenotypeData instance as the first positional argument.

You can import all the supported methods with the following:

from pgsui import ImputeVAE, ImputeAutoencoder, ImputeRefAllele, ImputeMostFrequent, ImputeRandomForest, ImputeHistGradientBoosting

Unsupervised Imputers

The four unsupervised imputers can be run by initializing them with the SNPio GenotypeData object and then calling fit() and transform().

# Initialize the models, then fit and impute
vae = ImputeVAE(data) # Variational autoencoder
vae.fit()
vae_imputed = vae.transform()

ae = ImputeAutoencoder(data) # standard autoencoder
ae.fit()
ae_imputed = ae.transform()

The *_imputed objects are NumPy arrays of IUPAC single-character codes that are compatible with SNPio's GenotypeData objects.

Supervised Imputers

Various supervised imputation options are supported, and these use the same API design.

# Supervised IterativeImputer classifiers

# Random Forest
rf = ImputeRandomForest(data)
rf.fit()
imputed_rf = rf.transform()

# HistGradientBoosting
hgb = ImputeHistGradientBoosting(data)
hgb.fit()
imputed_hgb = hgb.transform()

Non-machine learning methods

The following deterministic methods are supported. ImputeMostFrequent supports the mode-per-population or overall (global) mode options to inform imputation.

# Per-population, per-locus mode
pop_mode = ImputeMostFrequent(data, by_populations=True)
pop_mode.fit()
imputed_pop_mode = pop_mode.transform()

# Per-locus mode
mode = ImputeMostFrequent(data, by_populations=False)
mode.fit()
imputed_mode = mode.transform()

Or, always replace missing values with the reference allele.

ref = ImputeRefAllele(data)
ref.fit()
imputed_ref = ref.transform()

Command-Line Interface

Run the PG-SUI CLI with pg-sui (installed alongside the library). The CLI follows the same precedence model as the Python API:

code defaults < preset (--preset) < YAML (--config) < explicit CLI flags < --set key=value.

Recent releases add explicit switches for the simulated-missingness workflow shared by the neural and supervised models:

• --sim-strategy selects one of random, random_weighted, random_weighted_inv, nonrandom, nonrandom_weighted.
• --sim-prop sets the proportion of observed calls to temporarily mask when building the evaluation set.

Example:

pg-sui \
  --input data.vcf.gz \
  --popmap pops.popmap \
  --models ImputeVAE ImputeAutoencoder \
  --preset balanced \
  --sim-strategy random_weighted_inv \
  --sim-prop 0.3 \
  --prefix ae_and_vae \
  --n-jobs 4 \
  --tune-n-trials 100 \
  --set tune.enabled=True

CLI overrides cascade into every selected model, so a single invocation can evaluate multiple imputers with a consistent simulation strategy and output prefix.

STRUCTURE inputs accept a few extra flags for parsing metadata:

pg-sui \
  --input data.str \
  --format structure \
  --structure-has-popids \
  --structure-allele-start-col 2 \
  --structure-allele-encoding '{"1":"A","2":"C","3":"G","4":"T","-9":"N"}'

References

1. Kingma, D.P. & Welling, M. (2013). Auto-encoding variational bayes. In: Proceedings of the International Conference on Learning Representations (ICLR). arXiv:1312.6114 [stat.ML].

2. Hinton, G.E., & Salakhutdinov, R.R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504-507.

3. Stef van Buuren, Karin Groothuis-Oudshoorn (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software 45: 1-67.