pocketsearch 0.7.1

A pure-Python full text indexing search engine based on sqlite and the FTS5 extension.

pocketsearch

pocketsearch is a pure-Python full text indexing search engine based on sqlite and the FTS5 extension. It provides

• Support for full text search
• A simple API (inspired by the ORM layer of the Django web framework) for defining schemas and searching
• Support for multi-field indices including text, numeric and date search

It does not have any external dependencies other than Python itself. pocketsearch has been tested on Python 3.8, Python 3.9, Python 3.10 and Python 3.11.

pocketsearch is currently being tested on data from Wikipedia, indexing more than 6 million abstracts. If you are interested in preliminary performance tests, have a look at https://github.com/kaykay-dv/pocketsearch/tree/development/tests.

Status

The package is currently in Beta status.

Installation

Run

pip install pocketsearch

to install the package.

Getting started

By default, pocketsearch creates an in-memory database using a default search index schema containing only one field called 'text':

from pocketsearch import PocketSearch

pocket_search = PocketSearch()
pocket_search.insert(text="Hello World !")
print(pocket_search.search(text="hello")[0].text)
Hello World !

From a database perspective, the new document will be immediately available to the search index, as each insert is followed by a database commit.

Be aware that the search methods limits results to 10 by default. Results are ordered by the rank of the search result which is calculated by the FTS extension in sqlite (see https://www.sqlite.org/fts5.html#the_bm25_function for more details) showing how relevant a document is to a given query.

The API also supports iteration:

for document in pocket_search.search(text="hello"):
    print(document.text)

There is also supported for slicing:

pocket_search.search(text="hello")[1:3]

AND/OR queries

The FTS5 engines supports AND/OR queries. By default they are disabled in the API, if you want to make boolean queries, you have to use a lookup parameter in your query:

from pocketsearch import PocketSearch

pocket_search = PocketSearch()
pocket_search.insert(text="Hello World !")
print(pocket_search.search(text__allow_boolean="hello OR world")[0].text)
Hello World !

Please note, that AND as well as OR are case-sensitive in this context.

Counting results

By invoking the count method you get the number of search results:

print(pocket_search.search(text__allow_boolean="hello OR world").count())
1

Prefix queries

If you want to search for substrings, you can use prefix queries, by providing the allow_prefix lookup:

print(pocket_search.search(text__allow_prefix="hel*")[0].text)

Phrase queries

If you want to search for phrases, use quotation marks:

pocket_search.search(text='"this is" "a phrase"').count()

Combining lookups

Lookups can also be combined:

print(pocket_search.search(text__allow_prefix__allow_boolean="hel* OR wor*")[0].text)
Hello World !

Ordering results

By invoking the order method you can influence how your results are sorted. By default search results are sorted by relevance to the query.

# Order by text in ascending order
pocket_search.search(text__allow_boolean="hello OR world").order_by("text")
# This is equivalent to the previous call:
pocket_search.search(text__allow_boolean="hello OR world").order_by("+text")
# Order by text in descending order
pocket_search.search(text__allow_boolean="hello OR world").order_by("-text")

Schemas

A search index may have an arbitrary list of fields that can be searched. Schemas are defined through Schema classes:

from pocketsearch import Schema, PocketSearch
from pocketsearch import Text, Int, Real, Numeric, Blob, Date, Datetime

class FileContents(Schema):

    text = Text(index=True)
    filename = Text(is_id_field=True)

# create pocketsearch instance and provide schema
pocket_search.PocketSearch(schema=FileContents)
pocket_search.insert(text="Hello world",filename="a.txt")

Fields

Following fields are available:

Field	SQLite data type
Text	TEXT
Int	INTEGER
Real	REAL
Numeric	Numeric
Blob	Blob
Date	Date
Datetime	Datetime

Following options are available for fields:

• index - if the field is a Text field, a full text search index is created, otherwise a standard sqlite3 index is created
• is_id_field - a schema can only have one IDField. It is used by the .insert_or_update method to decide if a document should be inserted or an existing document should be updated.

With respect to naming your fields following restrictions apply:

• Fields may not start with an underscore.
• Fields may not contain double underscores.

NOTE: While not explicitly set, pocketsearch automatically adds an "id" field to the schema (using the INTEGER data type plus the AUTOINCREMENT option of sqlite). It is used as the primary key for each document.

Once the schema is created, you can query multiple fields:

# Searches field text for "world"
pocket_search.search(text="world")
# Searches documents that contain "world" in text AND have "a.txt" is a filename.
# Please note: as "filename" has not set its index option, only exact matches 
# will be considered.
pocket_search.search(text="world",filename="a.txt")

NOTE: When using multiple fields in search, the default boolean operation is AND. Currently, there is no way to express OR queries in the .search method.

However, you can join 2 queries (resulting in a UNION statement in SQL):

q = pocket_search.search(text="world") | pocket_search.search(filename="a.txt")
for result in q:
    print(result.text)

The result will contain all documents containing either "world" or where the filename is "a.text".

This option is currently experimental and still has issues, espcially when accessing results through indexing.

Inserting, updating and deleting data

Handling updates and deletes

Using the id of a document, you can run updates:

pocket_search.update(rowid=1, text="The updated text.")

If want to update more fields, simply provide them as keyword arguments.

To delete a document, use:

pocket_search.delete(rowid=1)

Using index readers to insert data

Normally we have a data source at hand (e.g. files in a file system or a source database) that we use to read data from. IndexReader classes can be used to build an index from such a data source. Assume, you want to index text files from a directory, we first define a schema:

class FileSchema(Schema):

        text = Text(index=True)
        filename = Text(is_id_field=True) 

Next, we create a PocketSearch and

from pocketsearch import FileSystemReader
pocket_search = PocketSearch(schema=FileSchema)
reader = FileSystemReader(base_dir="/some/directory", file_extensions=[".txt"])
pocket_search.build(reader)

This will build the index. If a document has already been seen it will be updated, a new document will be inserted otherwise.

Currently, the FileSystemReader is the only implementation provided, however you can easily implement your own by implementing the abstract class IndexError implementing a .read method. The .read method should return an iterable containing dictionaries whereas the dictionary's keys correspond to schema fields and its values the data you want to insert for the document.

Optimizing the index for query performance

If you have inserted a large volume of new documents, it might be sensible to optimize the index for query performance. This can be achieved by running VACUUM ANALYSE on the database, pocketsearch has a convenience method for this, that can be run e.g. after the indexing process is complete:

pocket_search = PocketSearch(db_name="my_db.db",writeable=True)
pocket_search.optimize()

Note, that this will close the current database connection and establish a new one.

More search options

Searching numeric data

You can also search for numeric data:

class Product(Schema):

    price = Int()
    description = Text(index=True) # Full text index
    category = Text()  # not part of FT index

pocket_search = PocketSearch(schema=Product)
# Create some sensible test data before proceeding ...
# Matches products with price=3
pocket_search.search(price=3)
# Matches products with price greater than 3
pocket_search.search(price__gt=3)
# Matches products with price lower than 3
pocket_search.search(price__lt=3)
# Matches products with price lower than equal 3
pocket_search.search(price__lte=3)
# Matches products with price greater than equal 3
pocket_search.search(price__gte=3)
# Matches products with price greater than equal 3 AND where the description contains "apple".
pocket_search.search(price__gte=3,description="apple")

You can also provide an index for numeric data by setting ...

price = Int(index=True)

... to speed up queries.

Searching date fields

pocketsearch also provides some (experimental) support for searching dates:

class AllFields(Schema):

    published=Datetime()

pocket_search = PocketSearch(schema=Product)
# Search documents published in year 2023
pocket_search.search(published__year=2023)
# Search document published after 2020
pocket_search.search(published__year__gt=2023)
# Search documents published in month 6
pocket_search.search(published__month=6)
# Search documents published on 21/6/2023:
pocket_search.search(published__month=21,published__month=6,published_year=2023)

NOTE: In search results, datefields are automatically converted to datetime and date objects respectivley.

Making your database persistent

The previous examples use an in-memory sqlite database. If you want to actually store the database, you have to provide a name:

pocket_search = PocketSearch(db_name="my_db.db",writeable=True)
# now, all operations will be done on the my_db database that is stored in the 
# current working directory.

When working with search indices that are stored on disk, it is important to provide the writeable argument, as any PocketSearch instance that works with file sqlite databases, is in read-only mode by default (unlike their in-memory counterpart.).

Behind the scenes: how searching works

pocketsearch uses the FTS5 extension of sqlite. More information can be found here: https://www.sqlite.org/fts5.html

Internally, it:

• Creates two tables, one named "document" and one virtual table "document_idx" - the latter holds the full-text-search enabled files.
• The document_idx table is populated through triggers on the document table.
• It uses the unicode61 tokenizer as default.

If you want to change the tokenizer, you can do so by overriding the Meta class of a schema:

from pocketsearch import Schema, PocketSearch

class FileContents(Schema):

    class Meta:
        '''
        Additional options for setting up FTS5
        See https://www.sqlite.org/fts5.html for more information.
        If a value is set to None, we leave it up to sqlite to
        set proper defaults.
        '''
        sqlite_tokenize = "unicode61" # change to available tokenizer of your choice
        sqlite_remove_diacritics = None
        sqlite_categories = None
        sqlite_tokenchars = None
        sqlite_separators = None    

    text = Text(index=True)
    filename = Text(is_id_field=True)

Multiple indicies in one database

You can have multiple indicies in one database (only databases written to disk) by setting the "index_name" option:

pocket_search = PocketSearch(db_name="my_db.db",index_name="Product",schema=Product)

Contribute

Pull requests are welcome. If you come across any issues, please report them at https://github.com/kaykay-dv/pocketsearch/issues