quick start

To install, either download the code from here and include the textcatvis folder in your $PYTHONPATH or install (the library components only) via pip:

$ pip install textcatvis

If you have text data available as a collection of .txt files either in a single folder or in multiple folders (in case of texts already sorted in different categories), you can call the script analyze_relevantwords.py with the path to the folder (or parent directory of multiple folders) to load this data and create word clouds for it.

textcatvis library components

dependencies: numpy, scipy, matplotlib, sklearn, wordcloud, nlputils

• data_utils.py: contains a function to load a text dataset (organized in a folder with subdirectories for each class containing .txt documents) in the form required by the other functions.

• cluster.py: contains a function to cluster a collection of text documents with the DBSCAN algorithm from sklearn.

• check_query.py: contains functions to formulate queries and check how often a term occurs in texts of a given category.

• vis_utils.py: contains functions to create the word clouds and highlight relevant words in individual texts.

• distinctive_words.py: contains code to examine a text dataset and identify “distinctive words” by comparing how often a word occurs in one category compared to all others.

• visualize_relevantwords.py: contains 3 functions to generate word clouds and highlight words in individual documents based on tf-idf features, distinctive words, as well as the classification scores obtained with a linear SVM.

examples

• analyze_relevantwords.py: can be called with a path to a dataset to carry out the analysis for this dataset, i.e. create word clouds for different classes etc.

• in experiments_cancer.py, the above mentioned tools are tested on the cancer papers dataset to create the results reported in the paper. (You need to download this dataset first.)

• in experiments_nytimes.py, the above mentioned tools are tested on articles downloaded with the NYTimes API. (Make sure you have an API key stored in nytimes_apikey.txt.)

If you have any questions please don’t hesitate to send me an email and of course if you should find any bugs or want to contribute other improvements, pull requests are very welcome!