A hint-enabled search engine framework for biomedical classification systems
Project description
Cateye
A hint-enabled search engine framework for biomedical classification systems
Features
- Hint: Show hints for search terms which can fast narrow down the results
- Automatic fallback: If no result satisfying the query, the system automatically eliminates less important search terms.
- Spelling correction: Build-in spelling correction
- Abbreviation expansion: Pre-defined abbreviation list will be automatically applied during the search
Installation
$ python setup.py install
Usage
1. Prepare your data:
Folders overview:
- data: The data source for the search engine, all information in this subfolders using the document id as their filenames.
- data/token: The tokens of the documents, after lemmatization
- data/snippet: The HTML snippets of the documents, which will be shown on the search results
- data/hint: The hints for each document
- data/spelling.txt: The formal spelling of your tokens (before normalization). If possible, sort the tokens with the frequency of usage, the most common word the first.
- data/abbreviation.txt: The abbreviations, one line for one abbreviation pair, using tab to separate the short form and long form
Cateye include some very basic text processing tools: tokenizer (basic_functions.tokenize) and lemmalizer (basic_functions.lemmalize)
The tokenize function will be used in two places: the first place is to cut your documents into tokens, and the second place is to cut your query into tokens.
The lemmatizing function will normalize your tokens. If you wish to build a case-insensitive search engine, you may use lowercase lemmatizer on the tokens.
2. Check the configuration.py:
The configuration.py setups the paths of token_folder, index_url, snippet_folder, hint_folder, abbreviation_file, and spelling_correction_file.
The index_url is used in the Shove object, which can be a remote URL starts with s3:// or a local URL starts with file:// please check the document of Shove.
3. Build the index:
Run command in the command line
$ cateye buildindex
This command read the files in token_folder and build an on-disk index in the index_url. It takes times depends on the size of your data
4. Run the application:
$ FLASK_APP=app.py flask run
License
- Free software: MIT license
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.