a python package to split machine learning data sets using graph partitioning
Project description
mlNODS
split machine learning data sets using graph partioning
Appropriate assessments require appropriate splits of training and evaluation data sets and this in turn requires clustering. For many problems single-linkage clustering suffices toward this end. Encountering a problem that could not be solved by such a standard procedure, we developed a simple graph-based tool for the creation of unique data sets.
mlNODS is a graph-based method which allows to split original data sets into non-overlapping sets that cannot be grouped without removing some of the data. mlNODS optimizes the following constraints: (1) retain as many data points as possible, and (2) remove any overlap between two split sets. The nodes of our graph are the original data points, and the connections are measures for the similarity between nodes (e.g. sequence similarity for protein sets). The method begins by building the full graph and proceeds by removing nodes in order to optimally fit the constraints to the similarity table. mlNODS is applicable to any prob- lem and has the additional benefit of allowing overlap within one set (i.e. training on homologues) while it is disal- lowed between two sets (i.e. training and testing do not overlap).
usage: mlnods [-h] -s SPLITS -c CUTOFF [-l LIMIT] -e EDGES_FILE
[-f EDGES_FORMAT] -n NODES_FILE [-a] [-r RANDOM]
[-o OUTFOLDER] [-v] [-q] [--version]
This is a script that will create independent sets of data
Version: 1.0 [03/14/20]
optional arguments:
-h, --help show this help message and exit
-s SPLITS, --splits SPLITS
number of splits required
-c CUTOFF, --cutoff CUTOFF
similarity cutoff in the units of link scores
-l LIMIT, --limit LIMIT
limit on the number of links for each node (default=0, infinity)
-e EDGES_FILE, --edges EDGES_FILE
file containing a table of instances with link scores for each pair
-f EDGES_FORMAT, --format EDGES_FORMAT
format of the table file
blast : takes a list of -m 9 formated blast files and builds a table based on seqID
hssp : takes a list of -m 9 formated blast files, runs HSSP scoring script and builds an HSSP distance table
self<int> : space/tab separated table file, similarity score in column <int>
eg "ID1 ID2 similarity_score" will be addressed as self3 (default=self5)
-n NODES_FILE, --nodes NODES_FILE
instance file containing IDs of all instances being considered
IDs are case-independent (eg ABC = abc)
IDs are always preceeded by ">" and followed by a white space.
No white spaces are allowed in an ID.
If score is provided for an ID, it should be surrounded by spaces and directly follow the ID
(eg. >abl1_human 10 gene associated with ....)
Everything between two IDs is printed in the junction files, but not considered in evaluation
-a, --abundance the option to score
false : score retrieved from instance file, range [0-100], default=50 when missing
true : score approximated by actual number of times an ID appears in the instance file
-r RANDOM, --random RANDOM
set a fixed random seed to generate consistent partitions
-o OUTFOLDER, --outfolder OUTFOLDER
path to output folder (default=<current directory>
-v, --verbose set verbosity level
-q, --quiet no logging to stdout
--version show program's version number and exit
If an ID is present in the instance file, but not in the table file the ID is considered to not be linked to anything else
If an ID is present in the table file but not in the instance file, it is ignored
mlnods was developed by Yana Bromberg and refactored by Maximilian Miller.
Feel free to contact us for support at services@bromberglab.org.
This software is licensed under [NPOSL-3.0](http://opensource.org/licenses/NPOSL-3.0)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.