Various methods of feature selection from Text Data
Project description
DocumentFeatureSelection
what’s this?
This is set of feature selection codes from text data. (About feature selection, see here or here)
The Feature selection is really important when you use machine learning metrics on natural language data. The natural language data usually contains a lot of noise information, thus machine learning metrics are weak if you don’t process any feature selection. (There is some exceptions of algorithms like Decision Tree or Random forest . They have feature selection metric inside the algorithm itself)
The feature selection is also useful when you observe your text data. With the feature selection, you can get to know which features really contribute to specific labels.
Please visit project page on github.
If you find any bugs and you report it to github issue, I’m glad.
Any pull-requests are welcomed.
Supporting methods
This package provides you some feature selection metrics. Currently, this package supports following feature selection methods
TF-IDF
Pointwise mutual information (PMI)
Strength of Association (SOA)
Bi-Normal Separation (BNS)
Contribution of this package
Easy interface for pre-processing
Easy interface for accessing feature selection methods
Fast speed computation thanks to sparse matrix and multi-processing
Overview of methods
TF-IDF
This method, in fact, just calls TfidfTransformer of the scikit-learn.
See scikit-learn document about detailed information.
PMI
PMI is calculated by correlation between feature (i.e. token) and category (i.e. label). Concretely, it makes cross-table (or called contingency table) and calculates joint probability and marginal probability on it.
To know more, see reference
In python world, NLTK and Other package also provide PMI. Check them and choose based on your preference and usage.
SOA
SOA is improved feature-selection method from PMI. PMI is weak when feature has low word frequency. SOA is based on PMI computing, however, it is feasible on such low frequency features. Moreover, you can get anti-correlation between features and categories.
In this package, SOA formula is from following paper,
Saif Mohammad and Svetlana Kiritchenko, "Using Hashtags to Capture Fine Emotion Categories from Tweets", Computational Intelligence, 01/2014; 31(2).
SOA(w, e)\ =\ log_2\frac{freq(w, e) * freq(\neg{e})}{freq(e) * freq(w, \neg{e})}
Where
freq(w, e) is the number of times w occurs in an unit(sentence or document) with label e
freq(w,¬e) is the number of times w occurs in units that does not have the label e
freq(e) is the number of units having the label e
freq(¬e) is the number of units having NOT the label e
BNS
BNS is a feature selection method for binary class data. There is several methods available for binary class data, such as information gain (IG), chi-squared (CHI), odds ratio (Odds).
The problem is when you execute your feature selection on skewed data. These methods are weak for such skewed data, however, BNS is feasible only for skewed data. The following paper shows how BNS is feasible for skewed data.
Lei Tang and Huan Liu, "Bias Analysis in Text Classification for Highly Skewed Data", 2005
or
George Forman, "An Extensive Empirical Study of Feature Selection Metrics for Text Classification",Journal of Machine Learning Research 3 (2003) 1289-1305
Requirement
Python 3.x(checked under Python 3.5)
Setting up
install
python setup.py install
Note
You might see error message during running this command, such as
We failed to install numpy automatically. Try installing numpy manually or Try anaconda distribution.
This is because setup.py tries to instal numpy and scipy with pip, however it fails. We need numpy and scipy before we install scikit-learn.
In this case, you take following choice
You install numpy and scipy manually
You use anaconda python distribution. Please visit their site.
Examples
See scripts in examples/
Change log
For your reference I checked performance under following environment,
MacBookPro (late 2015) 3.1 GHz Intel Core i7, 16 GB 1867 MHz DDR3
input data has 98,600 feature dimensions.
0.6 2016/04/02
supports PMI and TF-IDF under Python3.x
0.7 2016/04/03
Added SOA under Python3.x
0.8 2016/04/03
Added BNS under Python3.x
0.9 2016/04/10
Removed a bug when calling n_gram method of DataConverter
1.0 2016/08/22
Refactored some modules. (I changed some module names. Sorry if you have problems…)
Added interface script
1.1 2016/9/16
Resolved bottleneck point in pre-processing
Fixed a bug which n_jobs parameter does not work in interface
PMI takes around 6 minutes (with both of multiprocessing and multithreading)
1.2 2016/9/16
A bug in calculating TF-IDF score, this bug was resolved.
1.3 2016/9/28
Resolved bottleneck poins in pre-processing
Introduced dict-vectorising in ScikitLearn
Introduced Cython in calculating PMI & SOA. You can call them with use_cython=True flag. See examples/basic_example.py
Performance
Cython PMI takes 11.87 sec.
Python multiprocessing PMI takes 513.541 sec. (8.55 min.)
1.3.2 2016/11/29
You can put persisted-dict-object on disk-drive instead of dict-object on memory.
You can put huge dict object as data-source of interface.run_feature_selection()
See example examples/huge_data_example.py
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for DocumentFeatureSelection-1.3.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | deee5827d5618bdbd293644d59d4a50df7fcb5e0fe88993d1d80deea2d956648 |
|
MD5 | 70f4ef6ac7bcb7cc6f0e3252c6ee440b |
|
BLAKE2b-256 | 8259fb283de01b5f6000646a872b043963224fd22526084fc8331188b71d9ad7 |
Hashes for DocumentFeatureSelection-1.3.2-py3.5-macosx-10.6-x86_64.egg
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4ca6529f2b7341452105a14d40cfd7c03f50df79174aab5a8033ae2860d191e |
|
MD5 | 306b8acfe34e8e9c156d82ef0653a108 |
|
BLAKE2b-256 | 562d2e7ce56ea4146e14a614336d0dad9f94a5679eae444c9559d558332cf4e3 |