MiGenPro - Microbial Genome Processing Toolkit

MiGenPro: A flexible linked data framework for phenotype-genotype prediction of microbial traits using machine learning.

Functionalities

• Genome annotation based on taxonomy identifiers.
• Data formatting and cleaning for microbial genome datasets.
• Conversion of raw query data into structured feature and phenotype matrices.
• Advanced filtering options to remove low-frequency features or phenotypes.
• Parallel processing support for efficient handling of large datasets.
• Easy training and prediction with machine learning models on microbial characteristics.

Quickstart

Installation with pip in a special conda environment

conda create -n migenpro -c bioconda ;   
conda activate migenpro ;
pip install migenpro

Run the workflow from phenotype graph to phenotype prediction using the following command:

 migenpro --df --gq --ml --annotation  --train --predict  \
          --sapp_jar ./binaries/SAPP-2.0.jar  \
          --phenotype_query_file sparql_phenotype:demo_gram.sparql   \
          --phenotype_hdt_file ./data/bacdive.hdt.gz   \
          --genome_query_file sparql_genome:DomainCopyNumber.sparql   \
          --abs_frequency 1   --threads 20   \
          --sampling_type SMOTEN --output ./demo_output  \
          --genome_dir ./demo_output/genomes

The --param_grids flag can be used to optimise the parameters of the machine learning models. An example json file is available at: tests/resources/param_grid.json

The individual steps:

1. Querying phenotype graphs
2. Annotating genomes
3. Querying the annotated genomes
4. Training machine learning models
5. Predicting phenotypes with existing models
6. Feature importance analysis
7. Summarising the results

1. Querying phenotype graphs

migenpro --df \
    --phenotype_query_file sparql_phenotype:demo_gram.sparql \
    --phenotype_hdt_file ./data/bacdive.hdt.gz \
    --abs_frequency 1 \
    --sapp_jar binaries/SAPP-2.0.jar \
    --output ./demo_output/

2. Annotating genomes

Genome annotation is done by default using the workflow: https://workflowhub.eu/workflows/1170/ this can be changed using the --cwl_file flag with a workflow of your choice granted that it takes a fasta file as input. You can speed up this process with the --threads flag.

migenpro --annotation \
    --genome_query_file sparql_genome:DomainCopyNumber.sparql \
    --sapp_jar ./binaries/SAPP-2.0.jar \
    --phenotype_matrix ./demo_output/phenotype_matrix.tsv \
    --output ./demo_output/ \
    --genome_dir ./demo_output/genomes

3. Querying the annotated genomes

migenpro --gq \
    --genome_query_file sparql_genome:DomainCopyNumber.sparql \
    --sapp_jar ./binaries/SAPP-2.0.jar \
    --output ./demo_output/ \
    --genome_dir ./demo_output/genomes

4. Training machine learning models

We will now use the default parameters for training the models. If you wish to optimise the parameters you can do this using the --param_grids flag. Here we set an absolute frequency of 1 as a per phenotype minimum frequency for the demo.

migenpro --ml --train --predict \
      --feature_matrix ./demo_output/feature_matrix.tsv \
      --phenotype_matrix ./demo_output/phenotype_matrix.tsv \
      --output ./demo_output/ \
      --abs_frequency 1 \
      --species_frequency 1

5. Predicting phenotypes with existing models

You can do this through the docker container or from the source code.

1. You will need to obtain a protein domain matrix of the desired genomes you can do this using the java code.
2. For ease of use we will use the python scripts that were made with the following command. The default output directory is "output/mloutput" if desired you can change this using the --output [output_directory_location]

migenpro --ml --predict \
      --feature_matrix ./demo_output/feature_matrix.tsv \
      --phenotype_matrix ./demo_output/phenotype_matrix.tsv \
      --output ./demo_output/ \
      --abs_frequency 1 \
      --model ./demo_output/protein_domains/DecisionTreeClassifier/DecisionTreeClassifier_protein_domains.pkl

6. Feature importance analysis

migenpro --fi \
        --models  ./demo_output/protein_domains/ \
        --feature_matrix ./demo_output/feature_matrix.tsv \
        --phenotype_matrix ./demo_output/phenotype_matrix.tsv \
        --output ./demo_output/

7. Summarising the results

Wait for the script to finish and retrieve the results of your prediction from the output directory. There the predictions are given in the following format:

migenpro --summarise \
        --output ./demo_output/

Contributing

Pull the git repo:

git pull git@gitlab.com:pig-paradigm/migenpro.git
cd migenpro

Installing the needed dependencies.

A pip requirements.txt file is located in the installation directory which you can install using the following command.

conda create -n migenpro python=3.12.5 --file installation/requirements.txt

Recreating the results from the study

The files needed to recreate our results are located on https://zenodo.org/records/16995284. Apply the steps from this tutorial namely the /data_visualisation/construct_all_graphs_from_summaries.ipynb to recreate the graphs.

Maintainers

Jasper J. Koehorst (@jjkoehorst) and Mike Loomans (@MikeLoomans1999)