Use fast FFT-based mutual information screening for large datasets. Works well on MRI brain imaging data. Developed by Kai Yang, [GPG Public key Fingerprint: CC02CF153594774CF956691492B2600D18170329](https://keys.openpgp.org/vks/v1/by-fingerprint/CC02CF153594774CF956691492B2600D18170329)
Project description
fastHDMI -- fast High-Dimensional Mutual Information estimation
Kai Yang
<kai.yang2 "at" mail.mcgill.ca>
License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
GPG Public key Fingerprint: CC02CF153594774CF956691492B2600D18170329
This packages uses FFT-based mutual information screening and accelerated gradient method for important variables from (potentially very) high-dimensional large datasets.
Consider the sizes of the datafiles, the most commonly-used functions are the functions run in parallel -- all functions running in parallel will has _parallel
suffix; and they all have arguments:
core_num
: number of CPU cores used for multiprocessing; the default option is to use all the cores available, considering the job is most likely running on a server instead of a PCmultp
: job multiplier, the job to be run in parallel will be first divided intocore_num * multp
sub-jobs -- as equal as possible, then at each time, one core will take one subjob.verbose
: how verbal the function will be, with0
being least verbal and increases wrt. the number decalred here
The function implementing our propsoed FFT-based mutual information estimation will have the following arguments:
N
: the grid size for 1-D FFT; withN=500
as the default valuea_N
,a_N
: similar to above, the grid size for 2-D FFT; with300
as the default valueskernel
andbw
specify the kernel and bandwidth used for KDEnorm
is the norm used for KDE -- this option only takes effects for 2-D KDE
The screening functions and their arguments:
- For
plink
files:
- arguments
bed_file
,bim_file
,fam_file
are the location of the plink files; - arguments
outcome
,outcome_iid
are the outcome values and the iids for the outcome. For genetic data, it is usual that the order of SNP iid and the outcome iid don't match. While SNP iid can be obtained from the plink1 files, outcome iid here is to be declared separately.outcome_iid
should be a list of strings or a one-dimensional numpy string array. continuous_screening_plink
,continuous_screening_plink_parallel
for screening on continuous outcomes with continuous covariatesbinary_screening_plink
,binary_screening_plink_parallel
for screening on binary outcomes with continuous covariatesclump_plink_parallel
for clumping -- starting from the first covariate (i.e., the first column on the left of the datafile), clumping will remove all subsequent covariates with a mutual information higher than what theclumping_threshold
declares with the one it looks at
- For
csv
files:
- argument
_usecols
is a list of column labels to be used, the first element should be the outcome. Returned mutual information calculation results match_usecols
. - Note that it is assumed the left first column should be the outcome; if not, use
_usecols
to set the first element to be the outcome column label. csv_engine
can usedask
for low memory situations, orpandas
'sread_csv
engine
s, orfastparquet
engine for a createdparquet
file for faster speed. Iffastparquet
is chosen, declareparquet_file
as the filepath to the parquet file; ifdask
is chosen to read very large CSV, it might need to specify a largersample
.binary_screening_csv
,binary_screening_csv_parallel
for screening on binary outcomes with continuous covariatesbinary_skMI_screening_csv_parallel
,continuous_skMI_screening_csv_parallel
for screening using mutual information estimation provided byskLearn
, i.e.,sklearn.metrics.mutual_info_score
,sklearn.feature_selection.mutual_info_classif
Pearson_screening_csv_parallel
for screening using Pearson correlationcontinuous_screening_csv
,continuous_screening_csv_parallel
for screening on continuous outcomes with continuous covariatesclump_continuous_csv_parallel
similar to above
A share_memory
option is added for multiprocess computing. As a feature, it can be applied on large .csv
data in parallel in a memory-efficient manner and use FFT for KDE to estimate the mutual information extremely fast. A tqdm progress bar is now added to be more useful on cloud computing platforms. verbose
option can take values of 0,1,2
, with 2
being most verbal; 1
being only show progress bar, and 0
being not verbal at all.
- For DataFrame files:
binary_screening_dataframe
,binary_screening_dataframe_parallel
for screening on binary outcomes with continuous covariatesbinary_skMI_screening_dataframe_parallel
,continuous_skMI_screening_dataframe_parallel
for screening using mutual information estimation provided byskLearn
, i.e.,sklearn.metrics.mutual_info_score
,sklearn.feature_selection.mutual_info_classif
Pearson_screening_dataframe_parallel
for screening using Pearson correlationcontinuous_screening_dataframe
,continuous_screening_dataframe_parallel
for screening on continuous outcomes with continuous covariatesclump_continuous_dataframe_parallel
similar to above
- For
numpy
arrays:
binary_screening_array
,binary_screening_array_parallel
for screening on binary outcomes with continuous covariatescontinuous_screening_array
,continuous_screening_array_parallel
for screening on continuous outcomes with continuous covariatesbinary_skMI_array_parallel
,continuous_skMI_array_parallel
for screening using mutual information estimation provided byskLearn
, i.e.,sklearn.metrics.mutual_info_score
,sklearn.feature_selection.mutual_info_classif
continuous_Pearson_array_parallel
for screening using Pearson correlation
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.