Microbiome LDA Topic Modeling Workflow

Overview

This workflow implements topic modeling for microbiome data analysis. Two LDA implementations (gensim/MALLET) and NMF are available.

For LDA in environmental microbiome: Huo, P., Vargas Ribera, P., Rediers, H. et al. Latent Dirichlet Allocation reveals tomato root-associated bacterial interactions responding to hairy root disease. Environmental Microbiome 20, 161 (2025). (https://doi.org/10.1186/s40793-025-00822-2)

For LDA in pig enterotyping: Comer L, Huo P, Colleluori C, et al. From forest to farm: the impact of a broad spectrum of lifestyles on the porcine gut microbiota[J]. Current Research in Microbial Sciences, 2026: 100576. (https://doi.org/10.1016/j.crmicr.2026.100576)

For NMF: Enterosignatures define common bacterial guilds in the human gut microbiome, Frioux, Clémence et al., Cell Host & Microbe, Volume 31, Issue 7, 1111 - 1125.e6 (https://doi.org/10.1016/j.chom.2023.05.024)

Workflow Steps

The workflow consists of four steps. We are using an IBD data set to demonstrate.

The demo data (notebooks/data/) is the gut microbiome 16S dataset from the Inflammatory Bowel Disease Multi-omics Database (IBDMDB / HMP2), obtained via the Bioconductor package HMP2Data and filtered exactly as in the MIDASim paper (samples with library size ≥ 3000 and taxa present in ≥ 2 samples, yielding 146 samples × 614 taxa).

He, M., Zhao, N. & Satten, G.A. MIDASim: a fast and simple simulator for realistic microbiome data. Microbiome 12, 135 (2024). (https://doi.org/10.1186/s40168-024-01822-z)

All workflow functions are imported from the top-level package:

from lda4microbiome import preprocess, training, k_evaluation, viz

Step 1: Data Transforming

• Input ASV abundance table and taxonomy file paths
• ASV abundance table requires a CSV file where the index contains samples and columns contain features (ASV IDs), with values as raw counts instead of relative abundance values
• Taxonomy table requires a CSV file where the index contains ASV IDs and columns include taxonomy levels (Domain, Kingdom, Phylum, Class, Order, Family, Genus, and Species)

output_path = str(REPO_ROOT / "notebooks" / "demo_output")
prep = preprocess(
    base_directory=output_path,
    asv_df=asv,
    taxa_df=taxa,
    taxa_level="ASV",
)

Step 2: Model Building

Select a range of models you want to evaluate.

models = training(
    output_path,
    MC_range=range(2,11),
    implementation="lda_gensim",  # 'lda_mallet', 'nmf' are available
)

Step 3: Model Evaluation

We provide two evaluations. One is basic metrics: perplexity and coherence score for LDA; squared R error and coherence score for NMF. The other is our designed StripeSankey, which directly illustrates how samples change across models.

k_evaluation(output_path, plot="metrics")

k_evaluation(output_path, plot="stripesankey", min_flow_samples=5,
                            metadata_df=metadata,
                            hover_field=["diagnosis", "biopsy_location", "is_inflamed"],
                            high_threshold=0.67)

k_evaluation(output_path, plot="meta_sankey",
             metadata_df=metadata, hover_field=["diagnosis", "biopsy_location"],
             high_threshold=0.67, k_list=[6,7,8,9,10])

plot="stripesankey" — cluster-lineage view showing how samples move between MCs across K:

plot="meta_sankey" — the same flows encoded by a metadata field (here diagnosis):

Step 4: Single Result Visualization

Once k is decided, we provide publication-ready visualizations. For the sample-MC relationship: annotated heatmap and stacked bar chart. For the MC-Feature relationship: heatmap.

k_selected = 4
v = viz(
    base_directory=output_path,
    k_value=k_selected,
    metadata_df=metadata,
    onestop=True
)

v.heatmap_sample(
    include=["diagnosis", "biopsy_location", "is_inflamed"],
    sort="diagnosis",
    save_plot=True
)

v.bar_sample(groupby="diagnosis")

v.heatmap_mc(
    feature_level="Genus",
    use_top_tokens=True,
    top_n=10,
    colorscale="Blues",
)

Sample–MC relationship. heatmap_sample — sample-MC probabilities with metadata annotation strips:

bar_sample — stacked MC composition grouped by a metadata column:

MC–Feature relationship. heatmap_mc — top features per MC:

Installation

Requires Python ≥ 3.9. You can install this package from PyPI:

pip install lda4microbiome

Acknowledgements

The MALLET LDA implementation is run through little-mallet-wrapper, a Python wrapper around MALLET (MAchine Learning for LanguagE Toolkit) by Andrew McCallum.

License

This project is licensed under the MIT License — see the LICENSE file for details.