moducomp 0.7.45

moducomp: metabolic module completeness and complementarity for microbiomes.

moducomp: metabolic module completeness of genomes and metabolic complementarity in microbiomes

moducomp is a bioinformatics pipeline designed to identify and analyze metabolic module completeness and complementarity in microbial communities. It processes genomic data (protein sequences in FAA format) to map KEGG Orthology (KO) terms to KEGG modules, calculates module completeness for individual genomes and combinations of genomes within a microbiome, specifically reports potential complementary relationships where a metabolic module is 100% complete in a combination of N genomes but not in any individual member or smaller subset.

Features

• Annotation of protein sequences using eggNOG-mapper to obtain KO terms.
• Mapping of KOs to KEGG metabolic modules based on kegg-pathways-completeness-tool to obtain metabolic module completeness.
• Parallel processing support for faster KPCT (KEGG Pathways Completeness Tool) analysis with automatic chunking and checkpointing.
• Reporting of module completeness for individual genomes.
• Calculation of module completeness for N-member genome combinations.
• Generation of complementarity reports highlighting modules completed through genome partnerships.
• Tracks and reports the actual proteins that are responsible for the completion of the module in the combination of N genomes.
• Automatic resource monitoring with timestamped logs tracking CPU usage, memory consumption, and runtime for reproducibility.
• Consistent logging to stdout/stderr with a per-command resource summary emitted at the end of each run.
• Built-in validation (moducomp validate) for scientific consistency checks across annotations, KO matrices, KPCT outputs, and complementarity reports.

Installation (Recommended)

Make sure you have Pixi installed:

curl -fsSL https://pixi.sh/install.sh | sh

Install moducomp with Pixi:

pixi global install \
  -c conda-forge \
  -c bioconda \
  -c https://repo.prefix.dev/astrogenomics \
  moducomp

Setup data (required)

moducomp needs the eggNOG-mapper database to run. Use moducomp setup to download the data using ModuComp's built-in downloader (with resume/retry) and record the location for future runs. For upstream details, see the eggNOG-mapper setup guide: eggNOG-mapper database setup.

moducomp setup

By default, the data are stored in ${XDG_DATA_HOME:-~/.local/share}/moducomp/eggnog, and moducomp will auto-detect that location on future runs without needing EGGNOG_DATA_DIR.

To use a custom location (or point to an existing download):

moducomp setup --eggnog-data-dir /path/to/eggnog-data

Add --force to re-download the data if the directory already exists. Use --prompt if you want to confirm each download interactively.

Quick test

Small test data sets ship with moducomp. After installation you can confirm the pipeline by running:

moducomp test --ncpus 16 --calculate-complementarity 2

The test command runs in low-memory mode by default. If you have plenty of RAM and want full-memory mode, add --fullmem (or --full-mem).

Developer install (Pixi)

If you want to download the code and develop locally:

git clone https://github.com/NeLLi-team/moducomp.git
cd moducomp
pixi install

Quick check

moducomp --help

If you are running from the repository with Pixi:

pixi run python -m moducomp --help

You should see the command line help without errors.

Usage

moducomp provides two main commands: pipeline and analyze-ko-matrix. You can run these commands using Pixi tasks defined in pyproject.toml or directly within the Pixi environment.

Pipeline overview

The diagram below shows the main stages executed by ModuComp.

graph TD
    A([Start run]) --> B[Initialize logging and resource monitoring]
    B --> C{Input type}
    C -->|pipeline| D[Validate genome directory]
    C -->|analyze-ko-matrix| H[Load existing KO matrix]
    D --> E[Prepare genomes: adapt headers or copy to tmp]
    E --> F[Merge genomes into single FAA]
    F --> G[Run eggNOG-mapper if needed]
    G --> H[Create KO matrix kos_matrix.csv]
    H --> I[Convert KO matrix to KPCT input]
    I --> J[Run KPCT with parallel fallback]
    J --> K[Create module completeness matrix]
    K --> L{Complementarity requested?}
    L -->|Yes| M[Generate complementarity reports]
    L -->|No| N[Skip]
    M --> O[Write outputs and logs]
    N --> O
    O --> P[Optional cleanup of tmp]
    P --> Q([Pipeline complete])

CLI options and defaults

This section lists all CLI options implemented today, along with their default values.

pipeline command (positional args: genomedir, savedir)

Option	Default	Description
--ncpus, -n	16	Number of CPU cores to use for eggNOG-mapper and KPCT.
--calculate-complementarity, -c	0	Complementarity size to compute (0 disables).
--adapt-headers/--no-adapt-headers	false	Adapt FASTA headers to `genome
--del-tmp/--keep-tmp	true	Delete temporary files after completion.
--lowmem/--fullmem (--low-mem/--full-mem)	lowmem	Run eggNOG-mapper without --dbmem to reduce RAM.
--verbose/--quiet	false	Enable verbose progress output.
--validate/--no-validate	validate	Run post-run validation checks.
--validate-report/--no-validate-report	validate-report	Write validation_report.json in the output directory.
--validate-strict/--validate-lenient	lenient	Treat validation warnings as failures when strict.
--log-level, -l	INFO	Logging level: DEBUG, INFO, WARNING, ERROR.
--eggnog-data-dir	EGGNOG_DATA_DIR	Path to eggNOG-mapper data (sets EGGNOG_DATA_DIR).

test command (bundled test genomes)

Option	Default	Description
--output-dir, -o	output_test_moducomp_<DATETIME>	Output directory for test run.
--ncpus, -n	2	CPU cores for the test run.
--calculate-complementarity, -c	2	Complementarity size to compute (0 disables).
--adapt-headers/--no-adapt-headers	false	Adapt FASTA headers before the test.
--del-tmp/--keep-tmp	true	Delete temporary files after the test completes.
--lowmem/--fullmem (--low-mem/--full-mem)	lowmem	Low-memory mode is the default for tests.
--verbose/--quiet	verbose	Verbose output is the default for tests.
--validate/--no-validate	validate	Run post-run validation checks.
--validate-report/--no-validate-report	validate-report	Write validation_report.json in the output directory.
--validate-strict/--validate-lenient	lenient	Treat validation warnings as failures when strict.
--log-level, -l	INFO	Logging level: DEBUG, INFO, WARNING, ERROR.
--eggnog-data-dir	EGGNOG_DATA_DIR	Path to eggNOG-mapper data (sets EGGNOG_DATA_DIR).

analyze-ko-matrix command (positional args: kos_matrix, savedir)

Option	Default	Description
--calculate-complementarity, -c	0	Complementarity size to compute (0 disables).
--kpct-outprefix	output_give_completeness	Prefix for KPCT output files.
--del-tmp/--keep-tmp	true	Delete temporary files after completion.
--ncpus, -n	16	CPU cores for KPCT parallel processing.
--verbose/--quiet	false	Enable verbose progress output.
--validate/--no-validate	validate	Run post-run validation checks.
--validate-report/--no-validate-report	validate-report	Write validation_report.json in the output directory.
--validate-strict/--validate-lenient	lenient	Treat validation warnings as failures when strict.
--log-level, -l	INFO	Logging level: DEBUG, INFO, WARNING, ERROR.

validate command (positional args: savedir)

Option	Default	Description
--mode	auto	Validation mode: auto, pipeline, or ko-matrix.
--calculate-complementarity, -c	auto-detect	Expected complementarity size (0 disables).
--kpct-outprefix	output_give_completeness	KPCT output prefix used during analysis.
--strict/--lenient	lenient	Treat warnings as failures when strict.
--report	none	Write JSON validation report to this path.
--verbose/--quiet	false	Enable verbose progress output.
--log-level, -l	INFO	Logging level: DEBUG, INFO, WARNING, ERROR.

setup command

Option	Default	Description
--eggnog-data-dir	${XDG_DATA_HOME:-~/.local/share}/moducomp/eggnog	Destination for eggNOG-mapper data (sets EGGNOG_DATA_DIR).
--yes/--prompt, -y	yes	Automatically accept core database downloads (use --prompt to confirm each).
--force, -f	false	Re-download data even if files already exist.
--log-level, -l	INFO	Logging level: DEBUG, INFO, WARNING, ERROR.
--verbose/--quiet	verbose	Stream downloader output to the console.

Performance and parallel processing

moducomp includes parallel processing capabilities for the KPCT (KEGG Pathways Completeness Tool) analysis, which can significantly improve performance for large datasets:

• Automatic chunking: Input files are automatically split into chunks for parallel processing
• Checkpointing: Resume capability if processing is interrupted - already processed chunks are automatically detected and skipped
• Fallback mechanism: If parallel processing fails, the system automatically falls back to sequential processing
• Configurable CPU usage: Use the --ncpus parameter to control how many CPU cores to use

CPU Configuration:

• The --ncpus parameter controls the number of CPU cores used for both eggNOG-mapper annotation and KPCT analysis
• For KPCT parallel processing, the system creates the same number of chunks as CPU cores specified
• Example: --ncpus 8 will use 8 cores and create 8 chunks for optimal parallel processing

Validation (QC)

Use the built-in validator to check scientific consistency across outputs after a run. The validator compares:

• KO sets and counts between eggNOG-mapper annotations and kos_matrix.csv
• KO sets between kos_matrix.csv and ko_file_for_kpct.txt
• KPCT contigs vs pathways outputs
• Module completeness ranges and combination naming
• Complementarity reports versus module completeness values
• Protein provenance fields (pipeline mode) or placeholders (KO-matrix mode)

Example:

# Validation runs by default after pipeline/analyze/test.
# Use --no-validate to disable or --no-validate-report to skip JSON output.
# When validation reports errors (or warnings in strict mode), the command exits non-zero.

# Validate a pipeline run and write a JSON report
moducomp validate /path/to/output --mode pipeline --report /path/to/output/validation_report.json

# Validate KO-matrix mode outputs (non-default KPCT prefix)
moducomp validate /path/to/output --mode ko-matrix --kpct-outprefix my_prefix

# Treat warnings as failures
moducomp validate /path/to/output --strict

⚠️ Important note 1

Prepare FAA files: Ensure FAA headers are in the form >genomeName|proteinId, or use the --adapt-headers option to format your headers into >fileName_prefix|protein_id_counter.

⚠️ Important note 2

moducomp is specifically designed for large scale analysis of microbiomes with hundreds of members, and works on Linux systems with at least 64GB of RAM. For robustness, low-memory mode is now the default for pipeline and test. If you have ample RAM and want full-memory mode, add --fullmem (--full-mem).

Notes on bundled test data

You can override the bundled data location with MODUCOMP_DATA_DIR. When working from source, the bundled test genomes live at moducomp/data/test_genomes.

Use moducomp setup to download and register the eggNOG data directory. The location is stored in ${XDG_CONFIG_HOME:-~/.config}/moducomp/config.json so future runs can find the data automatically.

From source (Pixi):

pixi run python -m moducomp setup --eggnog-data-dir /path/to/eggnog-data

Running with your samples

If you are running from the repository with Pixi, replace moducomp below with pixi run python -m moducomp.

pipeline command

Use the pipeline command to process a directory of genome FAA files from scratch.

moducomp pipeline \
    /path/to/your/faa_files \
    /path/to/your/output_directory \
    --ncpus <number_of_cpus_to_use> \
    --calculate-complementarity <N>  # 0 to disable, 2 for 2-member, 3 for 3-member complementarity.
    # Optional flags:
    # --fullmem                  # Optional: Use full-mem if you have ample RAM (default is low-mem)
    # --adapt-headers             # If your FASTA headers need modification
    # --del-tmp/--keep-tmp        # Delete or keep temporary files
    # --eggnog-data-dir /path     # If EGGNOG_DATA_DIR is not set
    # --verbose                   # Enable verbose output with detailed progress information

analyze-ko-matrix command

Use the analyze-ko-matrix command if you already have a KO matrix file (CSV format, where rows are genomes/combinations, columns are KOs, and values are KO counts).

The KO matrix file should have a taxon_oid column for genome identifiers, and subsequent columns for each KO (e.g., K00001, K00002) with integer counts.

moducomp analyze-ko-matrix \
    /path/to/your/kos_matrix.csv \
    /path/to/your/output_directory \
    --ncpus <number_of_cpus_to_use> \
    --calculate-complementarity <N>  # 0 to disable, 2 for 2-member, 3 for 3-member complementarity.

    # Optional flags:
    # --keep-tmp                  # Keep temporary files
    # --verbose                   # Enable verbose output with detailed progress information

Parallel processing features

moducomp includes advanced parallel processing capabilities for improved performance:

KPCT parallel processing

When using the --ncpus parameter with a value greater than 1, moducomp automatically enables parallel processing for the KPCT (KEGG Pathways Completeness Tool) analysis:

• Automatic chunking: Input files are split into ncpus chunks for optimal load balancing
• Concurrent processing: Multiple chunks are processed simultaneously using multiprocessing.ProcessPoolExecutor
• Resume capability: If processing is interrupted, completed chunks are automatically detected and skipped on restart
• Automatic fallback: If parallel processing fails, the system seamlessly falls back to sequential processing

Performance tips

• CPU cores: Start with --ncpus 8 for moderate datasets, increase to --ncpus 16 or higher for large datasets
• Memory considerations: Each parallel worker requires memory; reduce --ncpus if you encounter memory issues
• Large datasets: For datasets with hundreds of genomes, parallel processing can reduce KPCT analysis time by 50-80%

Example with parallel processing

# For large datasets with sufficient resources
moducomp pipeline ./large_genome_collection ./output_large --ncpus 32 --calculate-complementarity 3

# For moderate datasets with verbose output
moducomp analyze-ko-matrix ./ko_matrix.csv ./output_moderate --ncpus 16 --calculate-complementarity 2 --verbose

# For systems with limited memory (default behavior)
moducomp pipeline ./genomes ./output_lowmem --ncpus 8 --calculate-complementarity 2

# For systems with ample RAM
moducomp pipeline ./genomes ./output_fullmem --ncpus 8 --fullmem --calculate-complementarity 2

Expected outputs

The sections below describe the expected output files, naming conventions, and the column-level meaning of each file. These details are the same for moducomp pipeline and moducomp test (pipeline mode), and the subset noted for moducomp analyze-ko-matrix (KO-matrix mode).

Naming conventions

Genome identifiers are stored as taxon_oid. In pipeline mode, ModuComp expects protein headers in the format genome_id|protein_id. If you set --adapt-headers, ModuComp rewrites headers to >genomeName|protein_N, where genomeName is the FAA filename stem. Combination identifiers use __ (double underscore), for example GenomeA__GenomeB, and n_members in module_completeness.tsv records the size of each combination.

Pipeline mode outputs (moducomp pipeline, moducomp test)

• emapper_out.emapper.annotations: Full eggNOG-mapper annotations. The #query column must match genome_id|protein_id. KEGG_ko entries are prefixed ko:KXXXXX and are converted to KXXXXX for downstream matrices.
• kos_matrix.csv: Genome × KO count matrix. Columns: taxon_oid followed by KO IDs (e.g., K00001). Values are integer protein counts per KO.
• ko_file_for_kpct.txt: KPCT input file. Each line starts with taxon_oid followed by the set of KO IDs present in that genome or combination. If --calculate-complementarity is N>=2, combinations up to N are included as GenomeA__GenomeB.
• output_give_completeness_contigs.with_weights.tsv: KPCT module results per genome/combination. Columns: contig (genome/combination ID), module_accession, completeness (0–100), pathway_name, pathway_class, matching_ko (KO weights), missing_ko.
• output_give_completeness_pathways.with_weights.tsv: Same rows and order as the contigs file, but without the contig column. This is provided for compatibility with legacy tools; prefer the contigs file when you need genome-level provenance.
• module_completeness.tsv: Pivoted module completeness matrix. Columns: n_members, taxon_oid, followed by KEGG module IDs (M00001, …). Values are numeric percentages in the range 0–100.
• module_completeness_complementarity_Nmember.tsv: Complementarity report for N-member combinations (only when --calculate-complementarity N is set). Columns: taxon_oid_1..N, completeness_taxon_oid_1..N, module_id, module_name, pathway_class, matching_ko, proteins_taxon_oid_1..N. Protein fields list contributing proteins per KO (from eggNOG-mapper) as {'KXXXXX': 'genome|protein'}.
• logs/moducomp.log: Detailed run log with structured progress messages and per-command resource summaries.
• logs/resource_usage_YYYYMMDD_HHMMSS.log: Resource monitoring log capturing wall time, CPU time, CPU utilization, peak RAM, and exit code for each monitored command.
• tmp/ (only if --keep-tmp): Intermediate files such as merged_genomes.faa, emapper_output/, and KPCT chunk outputs.
• validation_report.json (default when validation is enabled): JSON report produced by the validator.

KO-matrix mode outputs (moducomp analyze-ko-matrix)

• kos_matrix.csv: A copy of the input KO matrix (same format as above).
• ko_file_for_kpct.txt: KPCT input generated from the KO matrix. If --calculate-complementarity is set, combination lines are added using GenomeA__GenomeB identifiers.
• output_give_completeness_contigs.with_weights.tsv: KPCT module results per genome/combination (same format as pipeline mode).
• output_give_completeness_pathways.with_weights.tsv: Same rows as the contigs file, without the contig column.
• module_completeness.tsv: Module completeness matrix (same format as pipeline mode).
• module_completeness_complementarity_Nmember.tsv: Complementarity report. Protein contribution columns are filled with No protein data available for <genome> because no eggNOG-mapper annotations are available in KO-matrix mode.
• logs/moducomp.log and logs/resource_usage_YYYYMMDD_HHMMSS.log: Standard run logs and resource summaries.
• validation_report.json (default when validation is enabled): JSON report produced by the validator.

Citation

Villada, JC. & Schulz, F. (2025). Assessment of metabolic module completeness of genomes and metabolic complementarity in microbiomes with moducomp . moducomp (v0.5.1) Zenodo. https://doi.org/10.5281/zenodo.16116092

Tool Citations

If you use ModuComp in a study, please also cite the wrapped tools and data sources:

• eggNOG-mapper: Cantalapiedra CP et al. (2021). “eggNOG-mapper v2: Functional Annotation, Orthology Assignments, and Domain Prediction at the Metagenomic Scale.” Molecular Biology and Evolution. DOI: 10.1093/molbev/msab293. GitHub: https://github.com/eggnogdb/eggnog-mapper
• KEGG Pathways Completeness Tool (KPCT): GitHub: https://github.com/EBI-Metagenomics/kegg-pathways-completeness-tool. KEGG source: Kanehisa M, Goto S. (2000). “KEGG: Kyoto Encyclopedia of Genes and Genomes.” Nucleic Acids Research. DOI: 10.1093/nar/28.1.27