fasttenet 0.1.13

FastTENET

Indroduction

• FastTENET is an accelerated TENET algorithm based on manycore computing.

Installation

• :snake: Anaconda is recommended to use and develop FastTENET.
• :penguin: Linux distros are tested and recommended to use and develop FastTENET.

Anaconda virtual environment

After installing anaconda, create a conda virtual environment for FastTENET. In the following command, you can change the Python version (e.g.,python=3.7 or python=3.9).

conda create -n fasttenet python=3.9

Now, we can activate our virtual environment for FastTENET as follows.

conda activate fasttenet

Install from PyPi

pip install fasttenet

• Default backend framework of the FastTENET is PyTorch Lightning.
• You need to install other backend frameworks such as CuPy, Jax, and TensorFlow

Install from GitHub repository

First, clone the recent version of this repository.

git clone https://github.com/cxinsys/fasttenet.git

Now, we need to install FastTENET as a module.

cd fasttenet
pip install -e .

• Default backend framework of the FastTENET is PyTorch Lightning.

Install backend frameworks

FastTENET supports several backend frameworks including CuPy, JAX, TensorFlow, PyTorch and PyTorch-Lightning.
To use frameworks, you need to install the framework manually

• PyTorch Lightning

PyTorch Lightning is a required dependency library for FastTENET and is installed automatically when you install FastTENET.
If the library is not installed, you can install it manually via pip.

python -m pip install lightning

• PyTorch: Installing custom PyTorch version

PyTorch is a required dependency library for FastTENET and is installed automatically when you install FastTENET.
If the library is not installed, you can install it manually.

conda install pytorch torchvision torchaudio pytorch-cuda=xx.x -c pytorch -c nvidia (check your CUDA version)

• CuPy: Installing CuPy from Conda-Forge with cudatoolkit

Install Cupy from Conda-Forge with cudatoolkit supported by your driver

conda install -c conda-forge cupy cuda-version=xx.x (check your CUDA version)

• JAX: Installing JAX refer to the installation guide in the project README

Install JAX with CUDA > 12.x

pip install -U "jax[cuda12]"

Use 'XLA_PYTHON_CLIENT_PREALLOCATE=false' to disables the preallocation behavior
(https://jax.readthedocs.io/en/latest/gpu_memory_allocation.html)

• TensorFlow: Installing TensorFlow refer to the installation guide

Install TensorFlow-GPU with CUDA

python3 -m pip install tensorflow[and-cuda]

Tutorial

FastTENET class

Create FastTENET instance

FastTENET class requires data path as parameter

parameters

• dpath_exp_data: expression data path, required
• dpath_trj_data: trajectory data path, required
• dpath_branch_data: branch(cell select) data path, required
• dpath_tf_data: tf data path, required
• spath_result_matrix: result matrix data path, optional, default: None
• make_binary: if True, make binary expression and node name file, optional, default: False

import fasttenet as fte

worker = fte.FastTENET(dpath_exp_data=dpath_exp_data,
                           dpath_trj_data=dpath_trj_data,
                           dpath_branch_data=dpath_branch_data,
                           dpath_tf_data=dpath_tf_data,
                           spath_result_matrix=spath_result_matrix,
                           make_binary=True)

• aligned_data: when directly using rearranged data with expression data, trajectory data and branch data, optional
• node_name: 1d array of node names, required when using data directly
• tf: 1d array of tf names, optional when using data directly

import fasttenet as fte

node_name, exp_data = fte.load_exp_data(dpath_exp_data, make_binary=True)
trajectory = fte.load_time_data(dpath_trj_data, dtype=np.float32)
branch = fte.load_time_data(dpath_branch_data, dtype=np.int32)
tf = np.loadtxt(dpath_tf_data, dtype=str)

aligned_data = fte.align_data(data=exp_data, trj=trajectory, branch=branch)
        
worker = fte.FastTENET(aligned_data=aligned_data,
                       node_name=node_name,
                       tfs=tf,
                       spath_result_matrix=spath_result_matrix) # Optional

Run FastTENET

parameters

• backend: optional, default: 'cpu'
• device_ids: list or number of devcies to use, optional, default: [0] (cpu), [list of whole gpu devices] (gpu)
• procs_per_device: The number of processes to create per device when using non 'cpu' devices, optional, default: 1
• batch_size: Required
• kp: kernel percentile, optional, default: 0.5
• binning_method: discretization method for expression values, optional, 'FSBW-L' is recommended to achieve results similar to TENET.

result_matrix = worker.run(backend='gpu',
                                device_ids=8,
                                procs_per_device=4,
                                batch_size=2 ** 16,
                                kp=0.5,
                                binning_method='FSBW-L',
                                )

Run FastTENET with YAML config file

• Before run tutorial_config.py, batch_size parameter must be modified to fit your gpu memory size
• You can set parameters and run FastTENET via a YAML file
• The config file must have values set for all required parameters

Usage

python tutorial_config.py --config [config file path]

Example

python tutorial_config.py --config ../configs/config_tuck_sub.yml

Output

TE_result_matrix.txt

Run FastTENET with tutorial_notf.py

• Before run tutorial_notf.py, batch_size parameter must be modified to fit your gpu memory size

Usage

python tutorial_notf.py --fp_exp [expression file path] 
                        --fp_trj [trajectory file path] 
                        --fp_br [cell select file path] 
                        --backend [name of backend framework]
                        --num_devices [number of devices]
                        --batch_size [batch size]
                        --sp_rm [save file path]

Example

python tutorial_notf.py --fp_exp expression_dataTuck.csv 
                        --fp_trj pseudotimeTuck.txt 
                        --fp_br cell_selectTuck.txt 
                        --backend lightning
                        --num_devices 8
                        --batch_size 32768
                        --sp_rm TE_result_matrix.txt

Output

TE_result_matrix.txt

Run FastTENET with tutorial_tf.py

• Before run tutorial_tf.py, batch_size parameter must be modified to fit your gpu memory size

Usage

python tutorial_tf.py --fp_exp [expression file path] 
                      --fp_trj [trajectory file path] 
                      --fp_br [cell select file path] 
                      --fp_tf [tf file path] 
                      --backend [name of backend framework]
                      --num_devices [number of devices]
                      --batch_size [batch size]
                      --sp_rm [save file path]

Example

python tutorial_tf.py --fp_exp expression_dataTuck.csv 
                      --fp_trj pseudotimeTuck.txt 
                      --fp_br cell_selectTuck.txt 
                      --fp_tf mouse_tfs.txt 
                      --backend lightning
                      --num_devices 8
                      --batch_size 32768
                      --sp_rm TE_result_matrix.txt

Output

TE_result_matrix.txt

Run make_grn.py

parameters

• fp_rm: result matrix, required
• fp_tf: tf list file, optional
• fdr: specifying fdr, optional, default: 0.01
• t_degrees: specifying number of outdegrees, optional, generate final GRNs by incrementally increasing the fdr
  value until the total number of outdegrees is greater than the parameter value.
• trim_threshold: trimming threshold, optional, default: 0

Usage

When specifying an fdr

python make_grn.py --fp_rm [result matrix path] --fp_tf [tf file path] --fdr [fdr]

Example

python make_grn.py --fp_rm TE_result_matrix.txt --fp_tf mouse_tf.txt --fdr 0.01

Output

TE_result_matrix.byGRN.fdr0.01.sif, TE_result_matrix.byGRN.fdr0.01.sif.outdegrees.txt
TE_result_matrix.byGRN.fdr0.01.trimIndirect0.sif, TE_result_matrix.byGRN.fdr0.01.trimIndirect0.sif.outdegrees.txt

Usage

When specifying the t_degrees

python make_grn.py --fp_rm [result matrix path] --fp_tf [tf file path] --t_degrees [number of outdegrees]

Example

python make_grn.py --fp_rm TE_result_matrix.txt--fp_tf mouse_tf.txt --t_degrees 1000

Output

TE_result_matrix.byGRN.fdr0.06.sif, TE_result_matrix.byGRN.fdr0.06.sif.outdegrees.txt
TE_result_matrix.byGRN.fdr0.06.trimIndirect0.sif, TE_result_matrix.byGRN.fdr0.06.trimIndirect0.sif.outdegrees.txt

TODO

• add 'JAX' backend module
• add 'PyTorch Lightning' backend module
• add 'TensorFlow' backend module