textsearchpy 1.2.0

In memory text search engine written in python

TextSearchPy

TextSearchPy is a high performance, easy to use, in-memory text search engine library written in Python.

Install

pip install textsearchpy

Quickstart

Default Index is created with SimpleTokenizer and LowerCaseNormalizer

from textsearchpy import Index, Document

# create index with default settings
index = Index()

# add to index
index.append(["The quick brown fox"])

# alternatively, create Document object and append
doc = Document(text="jumps over the lazy dog")
index.append([doc])

# search with query string, returns list of Document objects
print(index.search("fox"))
print(index.search("fox OR dog"))
print(index.search("fox AND dog"))
print(index.search("fox NOT quick"))

# shows how input text is tokenized & normalized
print(index.text_to_index_tokens("The quick brown fox"))

Query Syntax

Query can be written in string format (shown in quickstart) or by creating different Query objects

query = "fox"

query = "fox AND dog NOT quick"

query = "(fox OR dog) AND (quick OR lazy)"

# proximity search with distance of 1
query = '"quick fox"~1'

Supported Query Types:

TermQuery - query with single token

from textsearchpy.query import TermQuery

query = "fox"
query = TermQuery(term="fox)

BooleanQuery - sub queries with boolean condition

Clause - "SHOULD", "MUST", "MUST_NOT"

BooleanClause accept any query type, thus it is possible to create a tree of grouped boolean queries

from textsearchpy.query import TermQuery, BooleanClause, BooleanQuery

query = "(fox OR dog) AND (quick NOT lazy)"

term_1 = TermQuery(term="fox")
term_2 = TermQuery(term="dog")
query_left = BooleanQuery(
    clauses=[
        BooleanClause(query=term_1, clause="SHOULD"),
        BooleanClause(query=term_2, clause="SHOULD"),
    ]
)

term_3 = TermQuery(term="quick")
term_4 = TermQuery(term="lazy")
query_right = BooleanQuery(
    clauses=[
        BooleanClause(query=term_3, clause="SHOULD"),
        BooleanClause(query=term_4, clause="MUST_NOT"),
    ]
)

final_query = BooleanQuery(
    clauses=[
        BooleanClause(query=query_left, clause="MUST"),
        BooleanClause(query=query_right, clause="MUST"),
    ]
)

PhraseQuery - multi term query with option to set proximity distance

by default, terms in phrase is not order sensitive

the distance represents a term based edit distance, i.e. how many other terms are allowed within the boundry of the search terms

from textsearchpy.query import PhraseQuery

query = '"brown fox"'
query = PhraseQuery(terms=["brown", "fox"], distance=0)
# to enforce order sensitivity
query = PhraseQuery(terms=["brown", "fox"], distance=0, ordered=True)


query = '"jumps dog"~3'
query = PhraseQuery(terms=["jumps", "dog"], distance=3)
# to enforce order sensitivity
query = PhraseQuery(terms=["jumps", "dog"], distance=3, ordered=True)

# any two additional terms are allowed between these three terms for searching
query = '"jumps lazy dog"~2'

WildcardQuery - query with wildcard character to match Note: current implementation scans through entire index for simplicity, which can be slow

from textsearchpy.query import WildcardQuery

# allow a single wildcard character, i.e. brown brawn
query = "br?wn"
query = WildcardQuery(term="br?wn")

# allow many wildcard character to match, i.e. bun, barn, brown
query = "b*n"
query = WildcardQuery(term=term)

Benchmark

see ./benchmark for more info

Index Performance

index created with SimpleTokenizer and LowerCaseNormalizer

terms - searchable tokens after tokenization and normalization

dataset	# of docs	# of unique terms	# of terms total	index time	raw data size	memory use
Reuters	10788	29188	1330266	2.7s	8.84 MB	~135 MiB
Gutenberg	1000	424612	68132774	120s	396.30 MB	~3700 MiB

Query Performance

calculated as the average latency of few different preset queries

dataset	TermQuery	BooleanQuery	PhraseQuery	WildcardQuery
Reuters	0.08ms	0.11ms	0.06ms	8.46 ms
Gutenberg	0.07ms	0.10ms	55ms	104.6 ms