retfidf 0.0.2

This package provides a simple method to retrieve documents from large text corpora. For use in social sciences.

Document retrieval for social sciences

This package provides a simple method to retrieve documents from large text corpora. For use in social sciences.
It accompanies the paper [Finding democracy in big data: document retrieval using word-embedding] and is an
implementation of the algorithm described in the paper.

Manual

To create tfidf metrics, first pass the documents in list form to the class DocumentRetrieval at initialization.
You can either pass a created word2vec model to the class, or let the algorithm train word embeddings for you.
Then use the function calculate_similarity() with a keyword fo your choice to calculate the similarity between
the documents and the query. This function will return a list of scores for each of the documents passed to the
class at initialization.

You can use a multiple words query by simply passing it in a string format to the calculate_similarity() function.

Proposed usage:

from retfidf import doc_retrieval

df = pd.read_csv('corpus.csv')

emb_sim = doc_retrieval(df['text'])

query = 'democracy'

df['democracy_metrics'] = emb_sim.calculate_similarity(query)  

GITHUB

https://github.com/hplisiecki/document-retrieval-for-social-sciences