skifts 0.1.0

Search for the most relevant documents containing words from a query

skifts

Search for the most relevant documents containing words from the query.

query = ['A', 'B']

documents = [
    ['N', 'A', 'M'],  # matching features: 'A'
    ['C', 'B', 'A'],  # matching features: 'A', 'B'  
    ['X', 'Y']  # no matching features
]

The search with return ['C', 'B', 'A'] and ['N', 'A', 'M'] in that particular order.

It's not necessarily about text. Words are just any str instances. Documents are unordered collections of these str. We search for documents considering frequency, rarity and match accuracy.

Install

pip3 install git+https://github.com/rtmigo/skifts_py#egg=skifts

Use for full-text search

Finding documents that contain words from the query.

from skifts import SkiFts

# three documents, one per row
documents = [
    ["wait", "mister", "postman"],
    ["please", "mister", "postman", "look", "and", "see"],
    ["oh", "yes", "wait", "a", "minute", "mister", "postman"]
]

fts = SkiFts(documents)

# find and print the most relevant documents:
for doc_index in fts.search(['postman', 'wait']):
    print(documents[doc_index])

Words inside the documents list are considered ready-made feature identifiers. If your text needs preprocessing or stemming, this should be done separately.

The ranking takes into account the frequency of words in the document and the rarity of words in the corpus. The word order in the document and the distance between words do not matter.

Implementation details

The search uses the scikit-learn library, which ranks documents using tf-idf and cosine similarity.

See also

The gifts package implements the same search, but in pure Python with no binary dependencies.