A Sphinx extension for tokenize japanese query word with TinySegmenter.js
This extension tweaks searchtools.js of sphinx-generated html document to tokenize Japanese composite words.
Since Japanese is an agglutinative language, query word for document search usually becomes composite form like ‘システム標準’ (system standard). This makes difficult to search pages containing phrase such as ‘システムの標準’, ‘標準システム’, because TinySegmenter.py (Sphinx’s default Japanese index tokenizer) tokenizes ‘システム’ and ‘標準’ as indexes.
sphinx-tsegsearch patches searchtools.js to override query tokinization step so that query input is re-tokenized by TinySegmenter.js (original JavaScript implementation of TinySegmenter). As a result, roughly say, this tiny hack improves recall of Japanese document search in exchange of precision.
Usage:
Add ‘sphinx_tsegsearch’ in conf.extensions
Rebuild document.
1 maintainer
Release files for sphinx-tsegsearch 1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sphinx-tsegsearch-1.2.tar.gz | 15.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sphinx_tsegsearch-1.2-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size: 30.0 kB
Release files / sphinx-tsegsearch-1.2.tar.gz
| Download URL | sphinx-tsegsearch-1.2.tar.gz |
|---|---|
| Size | 15.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4d879b84c3a20491090025c91fb5a061d5840ba8850a15e842d1f9e6d454e5db
|
|
BLAKE2b-256 checksum How to use checksums |
c42cbb367b0294fa4a639fd5099fda5fdcc3b31e56dfe19edaf26f7a4634095c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.15 CPython/3.10.2 Darwin/21.3.0
|
Release files / sphinx_tsegsearch-1.2-py2.py3-none-any.whl
| Download URL | sphinx_tsegsearch-1.2-py2.py3-none-any.whl |
|---|---|
| Size | 14.7 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
ac9f1dc6ff87341884e024c9e3d4ad3e7499d914139776f25fda266971c78d9a
|
|
BLAKE2b-256 checksum How to use checksums |
1b6ca6764dbf5290bea12814293cf83c3cf7b67e27a315ebc4ca69e320b55753
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.15 CPython/3.10.2 Darwin/21.3.0
|
