G2PInsight

What it is: G2PInsight is a command-line pipeline for genotype-to-phenotype (G2P) analysis. You provide genotype data (VCF or PLINK) and a phenotype file; the tool prepares a training matrix, fits machine-learning models, explains SNP importance with SHAP, and can score new samples — no coding required.

What it is for: breeding and genetics studies where you want to predict traits, compare ML methods, find important genomic regions, and generate publication-ready figures.

How to use it: run four stages in order (predict is optional):

preprocess  →  train or train-all  →  visualize  →  predict (optional)

Supports classification and regression, optional GWAS/LD SNP filtering in preprocess, and bundled PLINK / GEMMA binaries.

Table of Contents

• What you get
• Quick Start
• Pipeline
• Installation
• Commands
  • preprocess
  • train
  • train-all
  • predict
  • visualize
• Output layout
• FAQ
• Developer guide
• License

What you get

Models: LightGBM, RandomForest, XGBoost, SVM, CatBoost, Logistic (classification only). train-all skips Logistic for regression.

Important conventions

• train / train-all need preprocess *_metadata.json (-j) — not a raw matrix path alone.
• GWAS/LD runs in preprocess only; training does not re-filter SNPs.
• Wait for preprocess to finish (Metadata file generated successfully) before training. In batch jobs, use set -e.
• Large SNP counts: use preprocess -f 2–4; for train-all, prefer --parallel_models 1 unless job memory is very large (each parallel worker copies the full matrix).

Quick Start

# 1. Preprocess
G2PInsight preprocess --vcf data/genotype.vcf -p data/phenotype.txt -o results/

# 2. Train (check metadata exists first)
test -f results/preprocess/results_metadata.json && echo "OK"
G2PInsight train -j results/preprocess/results_metadata.json -m LightGBM -o results/

# 3. Visualize
G2PInsight visualize \
  -i results/train/LightGBM/LightGBM_shap_values.txt \
  -I results/train/LightGBM/LightGBM_plotting_data.json \
  -o results/plot/lightgbm

Whole-genome or millions of SNPs — preprocess with filtering, then train conservatively:

G2PInsight preprocess --bfile genotype -p pheno.txt -o results/ -f 4
G2PInsight train-all -j results/preprocess/results_metadata.json -o results/ \
  --parallel_models 1 --threads 4

Pipeline

  genotype + phenotype
         │
         ▼
   preprocess  ──►  *_train_data.txt + *_metadata.json
         │
         ▼
   train / train-all  ──►  model.pkl, metrics, SHAP, plotting JSON
         │
         ├──► visualize  ──►  PNG / HTML
         └──► predict    ──►  predictions.tsv

Use -o my_project/ as the project folder. Preprocess writes my_project/preprocess/my_project_metadata.json — that is the path for train -j.

Installation

Requirements: Python 3.8–3.12, Linux (recommended) or macOS. PLINK and GEMMA are bundled.

pip install G2PInsight
pip install "G2PInsight[bed]"   # strongly recommended for large preprocess jobs
# pip install "G2PInsight[all]"  # bed-reader + psutil

From source:

git clone https://github.com/chenrf0407/G2P_tool.git
cd G2P_tool && pip install -e ".[bed]"

G2PInsight --version
G2PInsight -h

Without bed-reader, preprocess falls back to slower PLINK --recodeA. Install with pip install "G2PInsight[bed]" (or pip install bed-reader) for the default fast .bed path.

Commands

1. preprocess

What it is

Turns VCF / PLINK genotypes plus a two-column phenotype file into a sample × SNP training matrix and a metadata JSON that downstream stages require.

What you get

SNP columns are named chr_position (e.g. 1_123456).

How to run

G2PInsight preprocess (--bfile|--file|--vcf) <genotype> -p <phenotype> -o <output> [options]

Typical run (GWAS + LD feature selection):

G2PInsight preprocess --vcf data.vcf -p pheno.txt -o results/ -f 4

Large genome (prefer bed-reader + SNP filtering):

G2PInsight preprocess --bfile genotype -p pheno.txt -o results/ -f 4

Key options

Sample split: After MAF/geno QC, preprocess always fixes train/test (custom IDs or sorted-ID 80/20). Feature selection uses train only; test is held out for later evaluation. IDs are written to *_train_ids.txt / *_test_ids.txt and sample_split in metadata.

Matrix backend (fixed): auto — use bed-reader when available, else PLINK --recodeA; in-process cache on; chromosome recode serial.

Inputs: exactly one of --bfile, --file, --vcf. Phenotype: two columns, no header (sample ID + value).

2. train

What it is

Trains one ML model from *_metadata.json: load matrix → use preprocess sample_split → optional hyperparameter search → K-fold CV on training split → final fit → test metrics → SHAP.

What you get

How to run

G2PInsight train -j <metadata.json> -m <model> -o <output_dir> [options]

# Default (no hyperparameter search)
G2PInsight train -j results/preprocess/results_metadata.json -m LightGBM -o results/

# Large SNP set
G2PInsight train -j results/preprocess/results_metadata.json -m LightGBM -o results/ \
  --threads 4

# Enable hyperparameter search
G2PInsight train -j results/preprocess/results_metadata.json -m LightGBM -o results/ \
  --hyperparameter_search

Evaluation: Uses the train/test split fixed in preprocess (default 80/20 or custom ID files). Fixed 5-fold CV and optional tuning use the training split only; {Model}_metrics.json reports the held-out test set.

Flow (default): 5-fold CV on train split (model defaults) → final fit → SHAP. With --hyperparameter_search: hold-out tuning on the train split → CV → final fit → SHAP.

Key options

Before loading data, train checks metadata and logs memory guidance.

3. train-all

What it is

Trains every model for your task on the same data split. The parent reads the matrix once; each worker process runs the full pipeline for one model. After all finish, the best model (by held-out test performance) is kept; other model folders are removed.

What you get

Best model rule: classification → AUC (else accuracy); regression → Pearson r on test set.

How to run

G2PInsight train-all -j <metadata.json> -o <output_dir> [options]

# Recommended default
G2PInsight train-all -j results/preprocess/results_metadata.json -o results/ \
  --parallel_models 1 --threads 4

# Millions of SNPs
G2PInsight train-all -j results/preprocess/results_metadata.json -o results/ \
  --parallel_models 1 --threads 4

# Opt-in hyperparameter search
G2PInsight train-all -j results/preprocess/results_metadata.json -o results/ \
  --hyperparameter_search --parallel_models 1 --threads 4

Shares train options except -m. Extra: --parallel_models (default 1). On very wide matrices, high --parallel_models needs substantial RAM (each worker copies the full matrix).

4. predict

What it is

Applies a trained .pkl model to new samples. Aligns features to training SNPs (missing → zero-filled).

What you get

<output_dir>/predict/{Model}_predictions.tsv — columns sample, prediction (plus prob_class_* for classification).

How to run

G2PInsight predict -i <input> -m <model.pkl> -o <output_dir> [--task_type <type>]

G2PInsight predict -i new_genotype -m results/train/LightGBM/LightGBM_model.pkl -o results/ --task_type regression

Use the same genome build and SNP naming as training (*_training_features.json).

5. visualize

What it is

Renders PNG/HTML from training artifacts. Training writes data; visualize only plots.

Inputs

At least one of -i / -I is required. -o PREFIX is required. Outputs go under <parent_of_PREFIX>/visualize/.

Output files

-i

-I

How to run

G2PInsight visualize [-i <shap_values.txt>] [-I <plotting_data.json>] -o <prefix> [--top_snps N]

G2PInsight visualize -I results/train/LightGBM/LightGBM_plotting_data.json -o results/plot/perf
G2PInsight visualize -i results/train/LightGBM/LightGBM_shap_values.txt -o results/plot/shap --top_snps 20

Training defaults to --shap_dependence_top 50; keep visualize --top_snps ≤ that cap (or raise the train cap / use 0 for all).

Code map: G2PInsight/bin/visualization.py — (1) shared style helpers, (2) EnhancedGenomeVisualizer for -i, (3) SHAP dependence/summary, (4) performance/CV for -I. Entry: main.run_unified_visualization.

Output layout

results/
├── preprocess/
│   ├── results_train_data.txt[.gz]
│   ├── results_metadata.json       ← train -j
│   └── phenotype_distribution_*.png
├── train/
│   ├── LightGBM/                   ← or only BestModel/ after train-all
│   │   ├── LightGBM_model.pkl
│   │   ├── LightGBM_metrics.json
│   │   ├── LightGBM_shap_values.txt
│   │   ├── LightGBM_plotting_data.json
│   │   └── ...
│   ├── best_model_info.json        ← train-all
│   ├── model_comparison_report.json
│   └── all_models_cv_results.json
├── predict/
│   └── LightGBM_predictions.tsv
└── plot/visualize/
    └── *.png / *.html

Temp files under {output}/tmp/ are removed after successful runs.

FAQ

train requires metadata — why?

train and train-all need *_metadata.json from preprocess. It records the matrix path, samples, SNP count, and task type.

Where is the metadata file?

<output>/preprocess/{prefix}_metadata.json. Example: -o results/ → results/preprocess/results_metadata.json.

train-all keeps only one model folder

By design: best model by test set metrics. See best_model_info.json and all_models_cv_results.json for the full comparison.

train-all fails or workers crash

Symptom: Worker process crashed, all models failed, or OOM under high --parallel_models.

What to do:

1. ≥ 500k SNPs: prefer --parallel_models 1 unless memory is abundant.
2. Re-run preprocess with -f 2 or -f 4 if too many SNPs were kept.
3. Use --parallel_models 1 --threads 4 (omit --hyperparameter_search).
4. Increase job memory; or test one model: G2PInsight train -m LightGBM ....

Training or load is very slow

1. Fewer SNPs in preprocess (-f 2–4).
2. Uncompressed *_train_data.txt loads faster than .gz.
3. Keep default (no --hyperparameter_search).

SHAP dependence file is huge

Default is --shap_dependence_top 50. Use 0 only if you need all SNPs (can be very large). Genome-wide *_shap_values.txt is unchanged.

Preprocess slow or killed

1. Do not use --no-filter-snps on whole-genome data; use -f 2–4.
2. Install with pip install "G2PInsight[bed]" (or pip install bed-reader) for the default fast .bed backend.

predict: constant values or no feature match

1. Same genome build and chr_pos naming as training.
2. Check sample IDs match.
3. For VCF/PLINK, training SNPs are extracted automatically — ensure those SNPs exist in the new data.

visualize: missing effect column

Use training-generated *_shap_values.txt with feature, importance_abs, effect.

Which preprocess backend was used?

Log lines: Genotype conversion backend: bed (fast) or raw (PLINK --recodeA).

Developer guide

assocG2P/G2PInsight/
├── main.py
└── bin/
    ├── preprocess.py
    ├── modeltraining.py
    ├── visualization.py
    ├── gemma_gwas.py
    └── plink_ld.py

pip install -e ".[bed]"
G2PInsight preprocess -h

License

MIT License