Multiprocessing for Pathway-Tools
Pathway-tools multiprocessing script
mpwt is a python package for running Pathway-Tools on multiple genomes using multiprocessing.
There is no guarantee that this script will work, it is a Work In Progress in early state.
Table of contents
You must have an environment where Pathway-Tools is installed. Pathway-Tools can be obtained here. For some versions you need to have Blast installed on you system, for further informations look at this page.
If your OS doesn’t support Pathway-Tools, you can use a docker. If it’s your case, look at Pathway-Tools Multiprocessing Docker. It is a dockerfile that will create container with Pathway-Tools, its dependancies and this package. You just need to give a Pathway-Tools installer as input.
pip install mpwt
The script takes a folder containing sub-folders as input. Each sub-folder contains a genbank file. Genbank files must have the same name as the folder in which they are located and also finished with a .gbk.
Folder_input ├── species_1 │ └── species_1.gbk ├── species_2 │ └── species_2.gbk ├── species_3 │ └── species_3.gbk ..
Pathway-Tools will run on each genbank file. It will create an output folder inside the folder containing all the result files from the PathoLogic inference for each species. You can also choose another output folder.
Genbank file example:
LOCUS scaffold1 XXXXXX bp DNA linear INV DD-MMM-YYYY DEFINITION My species genbank. ACCESSION scaffold1 VERSION scaffold1 KEYWORDS Key words. SOURCE Source ORGANISM Species name Taxonomy; Of; My; Species; With; The; Genus. FEATURES Location/Qualifiers source 1..XXXXXX /scaffold="scaffold1" /db_xref="taxon:taxonid" gene START..STOP /locus_tag="gene1" mRNA START..STOP /locus_tag="gene1" CDS START..STOP /locus_tag="gene1" /db_xref="InterPro:IPRXXXXXX" /EC_number="X.X.X.X" /translation="AMINOAACIDSSEQUENCE"
Three input files are created by mpwt. Informations are extracted from the genbank file. myDBName corresponds to the name of the folder and the genbank file. taxonid corresponds to the taxonid in the db_xref of the source feature in the genbank. species_name is extracted from the genbank file.
organism-params.dat: ID myDBName STORAGE FILE NCBI-TAXON-ID taxonid NAME species_name genetic-elements.dats: NAME ANNOT-FILE gbk_pathname // script.lisp: (in-package :ecocyc) (select-organism :org-id 'myDBName) (create-flat-files-for-current-kb)
mpwt is usable as a command line.
mpwt -f path/to/folder/input [-o path/to/folder/output] [-d] [-v]
Optional argument are identified by .
-f –folder input folder as described in Input data.
-o –output output folder containing PGDB data or dat files (see -d arguments).
-d –dat will create dat files and only move them inside the output folder.
-c –cpu the number of cpu used for the multiprocessing.
-r –reduce delete files in ptools-local to reduce size of results.
-v –verbose print some information about the processing of mpwt.
–delete delete a specific PGDB inside the ptools-local folder.
–clean clean ptools-local folder, before any other operations.
Possible uses of mpwt:
mpwt -f path/to/folder/input
Create PGDBs of studied organisms inside ptools-local.
mpwt -f path/to/folder/input -d
Create PGDBs of studied organisms inside ptools-local and create dat files.
mpwt -f path/to/folder/input -o path/to/folder/output
Create PGDBs of studied organisms inside ptools-local. Then extract the files inside the output folder.
mpwt -f path/to/folder/input -o path/to/folder/output -d
Create PGDBs of studied organisms inside ptools-local and create dat files. Then extract the dat files inside the output folder.
mpwt can be used in a python script with an import:
import mpwt folder_input = "path/to/folder/input" folder_output = "path/to/folder/output" mpwt.multiprocess_pwt(folder_input, folder_output, dat_extraction=optional_boolean, size_reduction=optional_boolean, number_cpu=10, verbose=optional_boolean)
folder_input: folder containing sub-folders with Genbank file inside.
folder_output: output folder where all the result of Pathway-Tools will be moved. This argument is optional. If you don’t enter an argument, results will be stored in a folder named output inside the sub-folders containg Genbank file.
number_cpu: int or nothing. Number of cpu to use for the multiprocessing.
dat_extraction: True or nothing. If True, mpwt will only return dat files of the PGDB.
size_reduction: True or nothing. If True, after moving the data to the output folder, mpwt will delete files in ptools-local. This to decrease the size of the results.
verbose: True or nothing. If true, mpwt will be verbose.
- multiprocess_pwt(folder_input, folder_output, dat_extraction=optional_boolean, size_reduction=optional_boolean, verbose=optional_boolean)
Run the multiprocess Pathway-Tools on input folder.
Delete all the previous PGDB and the metadata files.
This can also be used with a command line argument:
If you use clean and the argument -f input_folder, it will delete input files (‘script.lisp’, ‘pathologic.log’, ‘genetic-elements.dat’ and ‘organism-params.dat’).
mpwt --clean -f input_folder
With this command, it is possible to delete a specified db, where pgdb_name is the name of the PGDB (ending with ‘cyc’). It can be multiple pgdbs, to do this, put all the pgdb IDs in a string separated by a ‘,’.
And as a command line:
mpwt --delete mydbcyc1,mydbcyc2
Return the path to ptools-local.
If you encounter errors (and it is highly possible) there is some tips that can help you resolved them.
For error during PathoLogic inference, a log is created where you launch the command. The log contains the summary of the build and the error for each species. There is also a pathologic.log in each sub-folders.
If the build passed you have also the possibility to see the result of the inference with the file resume_inference.tsv. For each species, it contains the number of genes/proteins/reactions/pathways/compounds in the metabolic network.
For others errors, currently nothing is made to help you. Maybe in the future.