TOPAS-pipeline
Automated (phospho)proteomics processing pipeline for large patient cohorts providing cohort as well as patient-specific insights.
The results of the pipeline can be explored on the web-based TOPAS portal: https://github.com/kusterlab/TOPAS-portal.git A public instance of the portal can be found here: https://topas-portal.kusterlab.org/
Supported inputs
- MaxQuant (TMT, LFQ)
- SIMSI-Transfer (TMT)
Runtime overview
Runtimes exclude processing time of SIMSI-Transfer.
| Dataset | #channels | #samples | #patients | #cores | runtime (h) | max memory (GB) |
|---|---|---|---|---|---|---|
| CPTAC UCEC | 170 | 153 | 95 | 8 | 1 | 8 |
| CPTAC BRCA | 170 | 153 | 122 | 8 | 1.1 | 10 |
| CPTAC LUAD | 250 | 225 | 110 | 8 | 1.7 | 11 |
| MTB cohort | 2068 | 1504 | 1284 | 8 | 8.3 | 50 |
Configuration
The pipeline needs configurations input from a JSON file format.
Examples can be found in config.json (full version) and config_minimal.json (only required configs).
Both relative and absolute paths are allowed.
Input parameters:
| Parameter | Required | Description | Example | Default |
|---|---|---|---|---|
| results_folder | yes | Path to the folder where results will be written. | "results/example_run" |
N/A |
| sample_annotation | yes | Path to the sample annotation file (CSV). | "example/annotation.csv" |
N/A |
| metadata_annotation | yes | Path to the metadata annotation file (Excel). | "example/METADATA_UCEC.xlsx" |
N/A |
| raw_file_folders | yes | List of raw file folders for proteomics and phosphoproteomics data. | ["example/raw_fp", "example/raw_pp"] |
N/A |
| data_types | List of data types to process: "fp" for proteome and "pp" for phosphoproteome. | ["fp", "pp"] |
["fp", "pp"] |
|
| simsi | ||||
| run_simsi | Boolean indicating whether to run SIMSI analysis. | true |
true |
|
| simsi_folder | yes | Path to the folder for writing SIMSI-Transfer results. | "results/SIMSI" |
N/A |
| tmt_ms_level | MS level for TMT quantification. | "ms2" |
"ms2" |
|
| stringencies | Stringency value for MaRaCluster. | 10 |
10 |
|
| tmt_requantify | Boolean indicating whether to requantify TMT data. | false |
false |
|
| maximum_pep | Maximum posterior error probability in percent for peptide ID propagation. | 1 |
1 |
|
| num_threads | Number of threads to use for SIMSI-Transfer. | 8 |
8 |
|
| correction_factor_mapping_file | Path to file mapping experiment names to correction factor files (TSV). | "example/correction_factor_mapping_file.tsv |
"" |
|
| preprocessing | ||||
| raw_data_location | yes | Path to the folder containing MaxQuant search result folders. | "example/CPTAC_searches" |
N/A |
| fasta_file | yes | Path to the FASTA file for protein sequences. | "example/uniprot_human.fasta" |
N/A |
| picked_fdr | False discovery rate threshold for protein groups. | 0.01 |
0.01 |
|
| fdr_num_threads | Number of threads to use in MaxLFQ computation. | 8 |
8 |
|
| imputation | Perform data imputation within batch on phosphoproteome level. | true |
true |
|
| debug | Run in debug mode. | false |
false |
|
| run_lfq | Input is from LFQ experiments. | false |
false |
|
| normalize_to_reference | Normalize channel intensities to the reference channel. | false |
false |
|
| clinic_proc | ||||
| pspFastaFile | yes | Path to the PSP FASTA file. | "PSP_annotations/Phosphosite_seq.fasta" |
N/A |
| pspKinaseSubstrateFile | yes | Path to the PSP kinase-substrate dataset. | "PSP_annotations/Kinase_Substrate_Dataset" |
N/A |
| pspAnnotationFile | yes | Path to the PSP phosphorylation site dataset. | "PSP_annotations/Phosphorylation_site_dataset" |
N/A |
| pspRegulatoryFile | yes | Path to the PSP regulatory sites file. | "PSP_annotations/Regulatory_sites" |
N/A |
| prot_baskets | yes | Path to the annotation file for TOPAS scores and proteins of interest. | "TOPASscores_POI_AS_250307.xlsx" |
N/A |
| extra_kinase_annot | Path to the annotation file with custom kinase-substrate relations. | "" |
"" |
|
| report | ||||
| samples_for_report | Which samples to include in the report. | "all" |
"all" |
|
| portal | ||||
| update | Automatically update the TOPAS portal once the run has finished | false |
false |
|
| cohort | Specifies the cohort name that should be updated in the TOPAS portal. | "" |
"" |
|
| url | URL of the TOPAS portal. | "" |
"" |
|
| config | Configuration file for the TOPAS portal. | "" |
"" |
|
| slack | ||||
| webhook_url | URL for the Slack webhook. | "https://hooks.slack.com/services/xxx/yyy/zzz" |
"" |
|
| channel | Name of the slack channel to post pipeline updates. | "#pipeline_channel" |
"" |
Install webhook for slack (optional)
If you want the pipeline to post update messages (finished runs, error messages) to your slack channel, follow these steps:
- Create a new slack app here: https://api.slack.com/apps?new_app=1, use the
from scratchoption. - Select your slack workspace and pick an appropriate name for the app, e.g.
topas-pipeline. - Navigate to
Incoming webhooksin the left menu. - Set
Activate Incoming webhookstoOnif this was not already the case. - Click on
Add New Webhook to Workspaceat the bottom of the page. - Select the channel you want to post messages in and add it to your config file as the
slack.channelproperty (N.B. do not forget to include the#symbol before the channel name!). - Copy the generated
Webhook URLto your config file as theslack.webhook_urlproperty.
Source: https://api.slack.com/messaging/webhooks
Running the pipeline
With Docker (recommended)
Requirements:
- git
- docker
- make
- Clone this repository
git clone https://github.com/kusterlab/TOPAS-pipeline.git - Build the docker image
make build - Create a config file named
config_patients.jsonin the repository with your configurations (see sectionConfiguration) - Start the pipeline:
You can also use a custom config file (works only with relative paths) and adjust the memory and cores (default: 100GB, 8 cores):make docker_allCONFIG_FILE=./path/to/config.json MEMORY_LIMIT=300gb CPU_LIMIT=16 make docker_all
With conda and poetry
Requirements:
- git
- Python ">=3.9, <=3.11"
- poetry
- make
- conda
- Create environment and install required packages from poetry.lock file:
conda create --name topas-pipeline python=3.9.12 conda activate topas-pipeline - Clone this repository
git clone https://github.com/kusterlab/TOPAS-pipeline.git - Install dependencies and start a poetry shell
poetry install poetry shell - Adjust the file paths in
config_minimal.jsonand run:make all
Note that it is also possible to run individual pipeline modules, e.g.:
# run simsi
python -m topas_pipeline.simsi -c config.json
# run whole pipeline following simsi
python -m topas_pipeline.main -c config.json
# run clinical annotation
python -m topas_pipeline.clinical_annotation -c config.json
Example
An example of the project folder setup and configuration file can be found in the /example folder.
Check the ReadMe in the /example folder for details.
Pipeline result files
The pipeline creates a folder with multiple output files:
| Output file | Description | Used on portal |
|---|---|---|
| configs.json | Copy of the configuration file in JSON format used for this pipeline run as described above. | |
| sample_annot_filtered.tsv | Subset of sample annotation/metadata after filtering out QC failed samples | |
| meta_input_file_{data_type}.tsv | Location per batch of search folder input, raw files and TMT correction factor file | |
| {data_type}_qc_numbers.csv | Per sample count of peptides, median intensities and summed intensities | |
| {data_type}_qc_batch_wise.csv | Per batch median and summed intensities | |
| {data_type}_in_batch_correction_factors.csv | Per sample correction factors for in-batch median centering | |
| {data_type}_ms1_correction_factors.csv | Per batch correction factors for MS1 median centering | |
| evidence.txt | Precursor level input to Picked Protein Group FDR in MaxQuant evidence.txt format | |
| pickedGeneGroups.txt | Gene-level output from Picked Protein Group FDR | |
| pickedGeneGroups_with_quant.txt | Gene-level output from Picked Protein Group FDR including quantification with MaxLFQ | |
| preprocessed_{data_type}.csv | Data matrix with patients as columns and normalized abundances of proteins or phosphopeptides as rows | |
| annot_{data_type}.csv | Same as preprocessed_{data_type}.csv but with gene and p-site functional annotations | |
| {data_type}_measures_rank.tsv | Data matrix with patients as columns and in-cohort rank per protein or phosphopeptide as rows | |
| {data_type}_measures_fc.tsv | Same as {data_type}_measures_rank.tsv but with fold changes | |
| {data_type}_measures_z.tsv | Same as {data_type}_measures_rank.tsv but with z-scores | X |
| {data_type}_measures_p.tsv | Same as {data_type}_measures_rank.tsv but with p-values derived from the z-scores | |
| basket_scores_4th.tsv | Data matrix with patients as rows and TOPAS scores as columns | X |
| basket_scores_4th_gen_zscored.tsv | Data matrix with patients as rows and Z-scored (across kinases) TOPAS scores as columns | X |
| subbasket_scores_{topas_rtk}.tsv | Data matrix with patients as rows and TOPAS subscores as columns for each TOPAS RTK | |
| kinase_results/kinase_scores.tsv | Data matrix with patients as columns and TOPAS substrate phosphorylation scores as rows | X |
| kinase_results/scored_peptides.tsv | Data matrix with patients as columns and weights and weighted z-scores for phosphopeptides as rows | |
| protein_results/protein_scores.tsv | Data matrix with patients as columns and TOPAS protein phosphorylation scores as rows | X |
| Reports/{patient_id}_proteomics_results.xlsx | Patient-specific reports | (X) |
| Pipeline_log.txt | Log messages printed by the pipeline |
Metadata
Release files for topas-pipeline 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| topas_pipeline-0.1.0.tar.gz | 86.7 kB | Without provenance | Details for topas_pipeline-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| topas_pipeline-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for topas_pipeline-0.1.0-py3-none-any.whl |
Total release size: 200.6 kB
Release files / topas_pipeline-0.1.0.tar.gz
| Download URL | topas_pipeline-0.1.0.tar.gz |
|---|---|
| Size | 86.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
34bb57ab87cb38835b4f8443948b3275375258349e5d38245632edc4bab61bb0
|
|
BLAKE2b-256 checksum How to use checksums |
784a5063f2e875e5e23173ffeade72ffa4c36d84b7523960a1689c00365c0b4e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.3 CPython/3.8.10 Linux/5.15.0-139-generic
|
Release files / topas_pipeline-0.1.0-py3-none-any.whl
| Download URL | topas_pipeline-0.1.0-py3-none-any.whl |
|---|---|
| Size | 113.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2a486a15b37ecd358c80a12783e0be384cb1eb1017c808dc10847b17261770b1
|
|
BLAKE2b-256 checksum How to use checksums |
49b17a7a53e1b67c6eee84e724b43747e7da57fe215d86ba2b02d66f9f4414f2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.3 CPython/3.8.10 Linux/5.15.0-139-generic
|
Security for topas-pipeline 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.