# tokenizer-hub
yoctol 乂卍oO煞氣ㄟtokenizerOo卍乂
Tokenizers have the same interface of Jieba:
`python from tokenizer_hub import XXX_tokenizer tokenizer = XXX_tokenizer() tokenizer.lcut('我来到北京清华大学') > ['我', '来到', '北京', '清华大学'] `
2 maintainers
Metadata
2 maintainers
Release files for tokenizer-hub 0.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tokenizer-hub-0.0.1.tar.gz | 7.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tokenizer_hub-0.0.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 20.9 kB
Release files / tokenizer-hub-0.0.1.tar.gz
| Download URL | tokenizer-hub-0.0.1.tar.gz |
|---|---|
| Size | 7.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3c82fb39074b93324ef95bd6b642976558bf85aa0ef802da3f3897eecbf56e79
|
|
BLAKE2b-256 checksum How to use checksums |
45c244cf09f4f98f949cd472d694df079d3c092999536b2fbb9d95efe06fec68
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.0.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.5.2
|
Release files / tokenizer_hub-0.0.1-py3-none-any.whl
| Download URL | tokenizer_hub-0.0.1-py3-none-any.whl |
|---|---|
| Size | 13.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d22b00d0890736d21b983b201ba4b25532c982fdb64bf610281ede7802872812
|
|
BLAKE2b-256 checksum How to use checksums |
ac7b09978044a1f55d4ba74068473c57399d8d3ca11d08698dcd988d835f77ca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.0.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.5.2
|
