installation

You either download the code from here and include the nlputils folder in your $PYTHONPATH or install (the library components only) via pip:

$ pip install nlputils

nlputils library components

dependencies: numpy, scipy, unidecode, matplotlib

• dict_utils.py: various helper functions to manipulate dictionaries, e.g. to invert them on various levels (for example transform a dict with {document: {word: count}} into {word: {document: count}}).

• features.py: this contains code to preprocess texts and transform them into tf-idf features. It’s somewhat similar to the sklearn TfidfVectorizer, but based on (sparse) dictionaries instead of sparse vectors. These dictionary based document features are the main input used for other parts of this library. But there is also a features2mat function to transform the dictionaries into a sparse feature matrix, which can be used with sklearn classifiers, for example.

• simcoefs.py: this has one main function compute_sim, which gets as input the tf-idf feature dictionaries of two documents and then computes their similarity. Concerning the type of similarity to compute between the documents, you can chose from a large variety of similarity coefficients, kernel functions, and distance measures, implemented based on [RIE08].

• simmat.py: this contains wrapper functions for simcoefs.py to speed up the computation of the similarity matrix for a whole corpus.

• ml_utils.py: helper function to perform a cross-validation.

• knn_classifier.py: based on a similarity matrix, perform k-nearest-neighbors classification.

• embedding.py: based on a similarity matrix, project data points to 2D with classical scaling or t-SNE.

• visualize.py: helper functions to create a plot of the dataset based on the 2D embedding. This can also create a json file, which can be used with d3.js to create an interactive visualization of the data.

examples

additional dependencies: sklearn

In the iPython Notebook at examples/examples.ipynb are several examples on how to use the above described library components.

If you have any questions please don’t hesitate to send me an email and of course if you should find any bugs or want to contribute other improvements, pull requests are very welcome!