Lexikanon
Korean and English tokenizers, text normalizers, and stopword lists. Lexikanon is a plain Python library; it also ships HyFI 2.0 config groups, so the same tokenizers can be selected and run from HyFI workflows.
Lexikanon 1.0 drops the HyFI 1.x API. Pin lexikanon<1 to keep it.
Install
uv add lexikanon # or: pip install lexikanon
pip install 'lexikanon[datasets]' # tokenize_dataset / extract_tokens on datasets.Dataset
pip install 'lexikanon[similarity]' # find_similar_docs_by_clustering
Python 3.11 or newer. Korean morphological analysis uses MeCab through eKoNLPy (fugashi and mecab-ko-dic wheels). On Python 3.15, fugashi builds from source and needs the MeCab library (apt-get install libmecab-dev or brew install mecab).
Use it as a library
from lexikanon.normalizers import Normalizer
from lexikanon.tokenizers import MecabTokenizer
tokenizer = MecabTokenizer(noun_postags=["NNG", "NNP"])
tokenizer("금통위는 물가안정과 금리정책을 밝혔다.")
# ['금통위/NNG', '는/JX', '물가/NNG', '안정/NNG', '과/JC', ...]
tokenizer.nouns("금통위는 물가안정과 금리정책을 밝혔다.")
Normalizer(hanja2hangle=True)("今年에도 持續하고 있다")
# '금년에도 지속하고 있다'
Importing lexikanon never imports HyFI.
Use it from HyFI
Installing lexikanon registers its config groups through the hyfi.plugins entry point:
| group | options |
|---|---|
tokenizer |
default, simple, mecab, nltk, nltk_universal |
normalizer |
default, formal_ko, informal_ko, formal_en, formal_en_parantheses |
stopwords |
default |
workflow |
lexikanon_tokenize_text, lexikanon_tokenize_dataset, lexikanon_find_similar_docs |
hyfi plugins
hyfi run workflow=lexikanon_tokenize_text tokenizer=mecab
import hyfi
tokenizer = hyfi.instantiate(hyfi.compose_group("tokenizer", "mecab"))
See Usage for workflow steps and presets.
Citation
@software{lee_2023_lexikanon,
author = {Young Joon Lee},
title = {Lexikanon: A HyFI-based library for Tokenizers},
year = 2023,
publisher = {GitHub},
url = {https://github.com/entelecheia/lexikanon}
}
License
1 maintainer
Metadata
1 maintainer
Release files for lexikanon 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| lexikanon-1.0.0.tar.gz | 1.2 MB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| lexikanon-1.0.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.0 MB
Release files / lexikanon-1.0.0.tar.gz
| Download URL | lexikanon-1.0.0.tar.gz |
|---|---|
| Size | 1.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
502ef9ed2f68682947b63a89e35b86027391fce0b203ab12b0c7ec08cf34c654
|
|
BLAKE2b-256 checksum How to use checksums |
1dea825d37c0e177174245212e167deaf7b983c7ec647086c21e289bc5408453
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / lexikanon-1.0.0-py3-none-any.whl
| Download URL | lexikanon-1.0.0-py3-none-any.whl |
|---|---|
| Size | 853.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
68dfab794ecbf556835b2e8779b7ad66af636a9014bbbd233f89e847cad10da7
|
|
BLAKE2b-256 checksum How to use checksums |
696d2c21dfc30e5e04d26cc32922c0c1fb4a8c8131aafba504b72ac4938aecec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logSecurity for lexikanon 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for lexikanon-1.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
