MiGenPro - Microbial Genome Processing Toolkit

MiGenPro: A flexible linked data framework for phenotype-genotype prediction of microbial traits using machine learning.

Functionalities

• Genome annotation of genome accessions (downloaded automatically) or your own FASTA files.
• Data formatting and cleaning for microbial genome datasets.
• Conversion of raw query data into structured feature and phenotype matrices.
• Advanced filtering options to remove low-frequency features or phenotypes.
• Feature selection via variance thresholding and mutual information ranking.
• Automatic resampling with SMOTE (numerical), SMOTEN (categorical), or SMOTENC (mixed).
• Parallel processing support for efficient handling of large datasets.
• Easy training and prediction with machine learning models on microbial characteristics.

Quickstart

Prerequisites

• Docker is required for the genome annotation step (--annotation). Ensure the Docker daemon is running and your user has permission:

  sudo usermod -aG docker $USER
  # Log out and back in for group changes to take effect
  

• Java (JRE 11+) is required for SAPP-based SPARQL queries. The SAPP jar (v2.2.0) is included in the repository at binaries/SAPP-2.0.jar.
• Git LFS is required to check out the large files in this repository: the BacDive phenotype graph (data/bacdive.hdt.gz), the SAPP jar, the manuscript matrices and the manuscript models. Install it (e.g. conda install -c conda-forge git-lfs or your package manager) and run git lfs install once before cloning. If you cloned without it, run git lfs pull inside the repository afterwards. Without Git LFS these files are small text pointers and the commands below fail.

Installation

MiGenPro targets Python >= 3.10. Install from source using Poetry (recommended) or pip:

git lfs install
git clone https://gitlab.com/wurssb/migenpro.git
cd migenpro
conda create -n migenpro python=">=3.10,<3.15"
conda activate migenpro
# Using Poetry (recommended)
poetry install
# Or using pip
pip install -r installation/requirements.txt
pip install -e .

Full demo (end-to-end)

This command queries phenotypes, downloads and annotates 20 genomes, queries the annotations, then trains and predicts:

migenpro --df --gq --ml --annotation \
  --sapp_jar ./binaries/SAPP-2.0.jar \
  --phenotype_query_file sparql_phenotype:demo_gram.sparql \
  --phenotype_hdt_file ./data/bacdive.hdt.gz \
  --genome_query_file sparql_genome:DomainCopyNumber.sparql \
  --abs_frequency 1 \
  --threads 20 \
  --sampling_type auto --train --predict --output ./demo_output

Note: The --annotation step downloads genomes from ENA/NCBI and runs the full CWL annotation workflow (Bakta, InterProScan, etc.) in Docker containers. With 20 demo genomes this typically takes 30-60 minutes depending on hardware and network.

ML only (skip annotation, provide your own matrices)

If you already have feature and phenotype matrices, you can skip the annotation and genome querying steps:

migenpro --ml \
  --feature_matrix ./output/feature_matrix.tsv \
  --phenotype_matrix ./output/phenotype_matrix.tsv \
  --sampling_type auto --train --predict --output ./output/

Sampling strategies

Feature selection

Features with zero variance (constant across all samples) are always removed before training. Additional filtering options:

When --top_percentile is used, variance filtering runs first to strip stale features, then mutual information is computed on the reduced set for speed.

The --param_grids flag can be used to optimise the parameters of the machine learning models. An example json file is available at: tests/resources/param_grid.json

The individual steps:

1. Querying phenotype graphs
2. Annotating genomes
3. Querying the annotated genomes
4. Training machine learning models
5. Predicting phenotypes with existing models
6. Feature importance analysis
7. Summarising the results

1. Querying phenotype graphs

migenpro --df \
    --phenotype_query_file sparql_phenotype:demo_gram.sparql \
    --phenotype_hdt_file ./data/bacdive.hdt.gz \
    --abs_frequency 1 \
    --sapp_jar binaries/SAPP-2.0.jar \
    --output ./output/

2. Annotating genomes

Genome annotation is done by default using the workflow: https://workflowhub.eu/workflows/1170/ this can be changed using the --cwl_file flag with a workflow of your choice granted that it takes a fasta file as input.

Note on --threads: genomes are annotated one after another (sequentially). --threads controls the number of threads each genome's workflow may use (Bakta, InterProScan, ...), not the number of genomes annotated in parallel. Annotating many genomes therefore takes a multiple of the single-genome time; to speed things up you can start several migenpro --annotation runs in parallel (finished genomes are skipped on restart).

By default the genomes listed in the first column of the phenotype matrix from step 1 are downloaded from ENA and annotated. Genome accessions are recognised with or without a version suffix (GCA_001038485 or GCA_001038485.1). For more than ~100 genomes, pass the NCBI datasets CLI with --dataset_bin to download from NCBI instead.

migenpro --annotation \
    --phenotype_matrix ./output/phenotype_matrix.tsv \
    --threads 8 \
    --output ./output/

Annotations are written to <output>/genomes/ (or <genome_dir>/genomes/ with --genome_dir); genomes that already have an annotation (*.SAPP.hdt.gz) are skipped, so an interrupted run can simply be restarted. Add --yaml_only to only write the workflow input files without running the workflow.

Annotating your own genomes (FASTA)

To annotate your own assemblies instead of downloading genomes, pass them with --fasta. It accepts:

• a single FASTA file (.fasta, .fna or .fa, optionally gzipped, e.g. isolate_01.fasta.gz),
• a directory: every FASTA file directly inside it is annotated,
• a text file with one FASTA path or URL per line.

migenpro --annotation \
    --fasta ./my_genomes/ \
    --threads 8 \
    --output ./output/

migenpro --gq \
    --genome_query_file sparql_genome:DomainCopyNumber.sparql \
    --output ./output/

Genome identifiers. Each genome is named after its file without the FASTA and gzip extensions: my_genomes/isolate_01.fasta.gz becomes isolate_01. If the file name contains an assembly accession (e.g. GCA_000005845.2_ASM584v2_genomic.fna.gz), that accession is used instead. This identifier is the row name in feature_matrix.tsv, so use the same identifiers in the first column of your phenotype matrix:

# genome	phenotype
isolate_01	positive
isolate_02	negative

Feature and phenotype matrices are matched on the first 13 characters of the identifier (the length of an accession such as GCA_000005845), so keep your file names unique within their first 13 characters (isolate_01, isolate_02, ... is fine; my_sequencing_run_A and my_sequencing_run_B are not).

You can then train on your genomes (step 4) or predict with an existing model (step 5).

Predicting phenotypes for your own genome(s)

The repository ships pre-trained models (trained on the manuscript data) under migenpro/data/manuscript_models/ for the phenotypes gram, motility, oxygen, temperature and spore. To predict these phenotypes for one of your own genomes:

1. Annotate the genome and build its feature matrix (steps 2 and 3), either from an accession or your own FASTA:

   migenpro --annotation --gq \
       --fasta ./my_genome.fasta.gz \
       --genome_query_file sparql_genome:DomainCopyNumber.sparql \
       --sapp_jar binaries/SAPP-2.0.jar \
       --output ./my_prediction/
   

2. Write a phenotype matrix that lists your genome with a placeholder label (labels are ignored in prediction-only mode):

   # genome	phenotype
   my_genome	query
   

3. Combine your genome's feature matrix with the feature matrix the model was trained on (the domain columns must be aligned; columns missing from your genome's matrix are treated as zero), then load the model and predict:

   migenpro --ml \
       --load_model migenpro/data/manuscript_models/migenpro_xgb_models/motility_XGBClassifier.pkl \
       --predict \
       --feature_matrix ./combined_feature_matrix.tsv \
       --phenotype_matrix ./my_phenotype_matrix.tsv \
       --output ./my_prediction/predictions/
   

The prediction and its per-class confidence are in ./my_prediction/predictions/protein_domains/XGBClassifier/XGBClassifier-predictions.tsv.

3. Querying the annotated genomes

Runs a SPARQL query on every annotation (*.hdt.gz) under <output>/genomes/ (or under --genome_dir) and combines the results into <output>/feature_matrix.tsv.

migenpro --gq \
    --genome_query_file sparql_genome:DomainCopyNumber.sparql \
    --sapp_jar binaries/SAPP-2.0.jar \
    --output ./output/

4. Training machine learning models

We will now use the default parameters for training the models. If you wish to optimise the parameters you can do this using the --param_grids flag.

migenpro --ml \
      --feature_matrix ./output/feature_matrix.tsv \
      --phenotype_matrix ./output/phenotype_matrix.tsv \
      --output ./output/

With feature selection and resampling:

migenpro --ml \
      --feature_matrix ./output/feature_matrix.tsv \
      --phenotype_matrix ./output/phenotype_matrix.tsv \
      --min_variance 0.01 \
      --top_percentile 20 \
      --sampling_type auto \
      --output ./output/

5. Predicting phenotypes with existing models

Trained models are saved as pickle files under <output>/protein_domains/<Classifier>/<Classifier>_protein_domains.pkl. To predict with such a model instead of training a new one, pass it with --load_model together with --predict. The feature matrix must contain the same protein domain features the model was trained on (see steps 2 and 3).

Prediction-only mode. When --load_model and --predict are given without --train, MiGenPro skips the train/test split and predicts on every genome listed in the phenotype matrix that is also present in the feature matrix. The phenotype labels are not used, so a single row with a placeholder label is enough — this is the way to predict phenotypes for new genomes with pre-trained models. Results are written to <output>/protein_domains/<Classifier>/<Classifier>-predictions.tsv:

migenpro --ml \
      --load_model ./output/protein_domains/RandomForestClassifier/RandomForestClassifier_protein_domains.pkl \
      --predict \
      --feature_matrix ./output/feature_matrix.tsv \
      --phenotype_matrix ./output/phenotype_matrix.tsv \
      --output ./output/predictions/

With --train (or without --load_model), the previous behaviour is kept: predictions are made on the train/test split of the intersected genomes and scored against the observed phenotypes (<Classifier>-{train,test}.tsv).

6. Feature importance analysis

migenpro --fi \
        --models path/to/models \
        --feature_matrix path/to/features.tsv \
        --phenotype_matrix path/to/phenotype.tsv \
        --output ./output/

7. Summarising the results

Wait for the script to finish and retrieve the results of your prediction from the output directory. There the predictions are given in the following format:

migenpro --summarise \
        --output ./output/

Contributing

Pull requests are welcome. Please run the test suite before submitting:

python -m pytest tests/              # fast tests
python -m pytest --run-main tests/   # also runs the resource-intensive integration tests

Clone the repository

git clone https://gitlab.com/wurssb/migenpro.git
cd migenpro

Installing the development dependencies

conda create -n migenpro python=">=3.10,<3.15"
conda activate migenpro
poetry install

Recreating the results from the study

The input matrices are included in this repository under migenpro/data/manuscript_matrices/, and the complete study output is archived on Zenodo. For step-by-step instructions to reproduce every phenotype and experiment track, see REPRODUCING_RESULTS.md.

Citation

If you use MiGenPro in your research, please cite the MiGenPro publication. The study output is archived on Zenodo.

License

MiGenPro is released under the MIT License. Data files under data/, output/, and tests/resources/genomes are released under CC0 1.0 Universal.

Maintainers

Jasper J. Koehorst (@jjkoehorst) and Mike Loomans (@MikeLoomans1999)