Rust Extension For Language Technology Platform(Python).
Project description
Language | version |
---|---|
Python | |
Rust |
LTP extension For Python
LTP for Rust 对 Python 的绑定,用于提升 LTP 的速度,以及加入传统机器学习算法实现的中文信息处理工具。
method | ltp 3.0(c++) | ap(1) | ap(8) | pa | pa-i(0.5) | pa-ii(0.5) |
---|---|---|---|---|---|---|
cws | 97.83 | 97.93 | 97.67 | 97.90 | 97.90 | 97.93 |
pos | 98.35 | 98.41 | 98.30 | 98.39 | 98.39 | 98.38 |
ner | 94.17 | 94.28 | 93.42 | 94.02 | 94.06 | 93.95 |
features
- 分句
- 任务
- 中文分词(cws)
- 对数字、英文、网址、邮件的处理
- 支持自定义词典
- 词性标注(pos)
- 支持自定义词典
- 命名实体识别(ner)
- 中文分词(cws)
- 算法
- 平均感知机(ap)
- 单线程平均感知机
- 多线程平均感知机
- 被动攻击算法(pa)
- 平均感知机(ap)
- 模型量化
- 在线学习
- 增量学习
性能测试
评测环境
- Python 3.10
- MacBook Pro (16-inch, 2019)
- 处理器: 2.6 GHz 六核Intel Core i7
- 内存: 16 GB 2667 MHz DDR4
注: 速度测试文件大小为 33.85 MB / 305041 行
分词
我们选择Jieba、Pkuseg、Thulac等国内代表分词软件与 LTP 做性能比较,根据第二届国际汉语分词测评发布的国际中文分词测评标准,对不同软件进行了速度和准确率测试。
在第二届国际汉语分词测评中,共有四家单位提供的测试语料(Academia Sinica、 City University 、Peking University(PKU) 、Microsoft Research(MSR)), 在评测提供的资源icwb2-data 中包含了来自这四家单位的训练集(icwb2-data/training)、测试集(icwb2-data/testing), 以及根据各自分词标准而提供的相应测试集的标准答案(icwb2-data/gold).在icwb2-data/scripts目录下含有对分词进行自动评分的perl脚本score。
我们在统一测试环境下,对若干流行分词软件和 LTP 进行了测试,使用的模型为各分词软件自带模型。在PKU和MSR测试集评测结果如下:
Algorithm | Speed(KB/s) | PKU(F1) | MSR(F1) |
---|---|---|---|
Jieba | 982.49 | 81.8 | 81.3 |
Pkuseg | 109.72 | 93.4 | 87.3 |
Thulac | 48.13 | 94.0 | 87.9 |
Thulac[Fast] | 1133.21 | 同上 | 同上 |
LTP 3(pyltp) | 451.20 | 95.3 | 88.3 |
LTP legacy(1) | 1603.63 | 95.2 | 87.7 |
LTP legacy(2) | 2869.42 | 同上 | 同上 |
LTP legacy(4) | 4949.38 | 同上 | 同上 |
LTP legacy(8) | 6803.88 | 同上 | 同上 |
LTP legacy(16) | 7745.16 | 同上 | 同上 |
注:括号内为线程数量
注2:Jieba的词表是在人民日报数据集上统计的
流水线
除了分词以外,我们也测试了 LTP 三个任务(分词、词性标注、命名实体识别)流水线的速度:
Algorithm | Speed(KB/s) |
---|---|
LTP 3(pyltp) | 153.10 |
LTP legacy(1) | 508.74 |
LTP legacy(2) | 899.25 |
LTP legacy(4) | 1598.03 |
LTP legacy(8) | 2267.48 |
LTP legacy(16) | 2452.34 |
注:括号内为线程数量
注2:速度数据在人民日报命名实体测试数据上获得,速度计算方式均为所有任务顺序执行的结果。
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ltp_extension-0.1.7-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b90cf24b6eed5ef831a025a197e01f09adfeecb6f13eb4ab8a04cc6032af3420 |
|
MD5 | 2d4699446cdb9cfc17181e34ba6ac957 |
|
BLAKE2b-256 | 2bbc45f9ef12a6098dbc749bb7719954c162804c65431484c42f0e0a58a87f11 |
Hashes for ltp_extension-0.1.7-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8829b6238b408a02fbecaac59bf79e5a9f83f2b1dcfb526c7f435830e3792af7 |
|
MD5 | 0f0fd2cfa544565cc84ad7253f2364f9 |
|
BLAKE2b-256 | 98631cd4557591fbd6a1123310845b05ac5be237d5cbbf9fce4eb33da204e514 |
Hashes for ltp_extension-0.1.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c89922c19c144b7fa7f57fb6df7c898f892f3517832803e342c635970c6149ae |
|
MD5 | 1ea725174cb4bb01fadc52cd66a9de81 |
|
BLAKE2b-256 | 4efb8684c7158bf3acadf8ca0fde1b23412693b7c96dc3199f447c5f4f021f33 |
Hashes for ltp_extension-0.1.7-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1e8d90780db62e93169b5ea31c0195f93d3db41419670250b6b5baa5ba40a2c |
|
MD5 | a7abca51c6e963e79eb59b70944a900a |
|
BLAKE2b-256 | 66dcaec4193a2730cb58394069cad0a6804bf58667f45b0f8c18486e9a084f54 |
Hashes for ltp_extension-0.1.7-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a924ce7bbf3cd422f46980df6dc10ed06a38263ded85ab74adea165204d689dc |
|
MD5 | cf38dc93840166db1b6209aeac24e4d6 |
|
BLAKE2b-256 | 13ea45d221d42ae6ef6360c9a15653a388c67493f7c7bbe5fb02facac2f88c6e |
Hashes for ltp_extension-0.1.7-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 280f4ba04cc4783811d39d2b42d45c019549b4cafab8cbd5a6aff99794b3a7b0 |
|
MD5 | a65d9be56347c73402d3f1d1d546e956 |
|
BLAKE2b-256 | 9f7b9fe52eea8e964973f6138dacdc72d2654447a4177b69dd11005d63b96dfb |
Hashes for ltp_extension-0.1.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1058eb8dc4f12b6141184ec4c9c9d3aed3460b48cad73bf87951fd54c87d1eb0 |
|
MD5 | b9a676c6b03b58c72bfaf0ecc5183ca8 |
|
BLAKE2b-256 | f2b95fe3bb1cb87312477dad1c0eb70636eb72ff7d820653d6ae23c44aeb7aae |
Hashes for ltp_extension-0.1.7-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aff64721a4ea1c10c66e03fbe8a22af395cf9adc62f96c5bbac5d37e377a161d |
|
MD5 | 7fb094d88a47789039392c93d2ad770d |
|
BLAKE2b-256 | b237aef74f6d4edb11546e18890f73f4cf4c554cdd2d7a4ef196bf8fe8f7a664 |
Hashes for ltp_extension-0.1.7-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0c3fb175d92003e3bae0bcebad57422dbba7f3914572293a98d8f3715ecaec42 |
|
MD5 | 6cf7e702781696cf6173261292e9c16e |
|
BLAKE2b-256 | 75b0e571706f59b7a7a8728080e1de731a501af84cd95018df8fbbc2bc49587e |
Hashes for ltp_extension-0.1.7-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17dc6fcfab0531cc57f3722ed9e949baf945dd20ab625f6f5d768707d4e903ca |
|
MD5 | 024887f306118012b5bd45a68013070b |
|
BLAKE2b-256 | a0e1ec92a63a4c7d7c61f715eb8e7422db357719a3c36c727f79112a723ecae8 |
Hashes for ltp_extension-0.1.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f6103d7293057161abfeacf0edfa8f6a037e0221da1947e2e5b797ef8d612459 |
|
MD5 | 68def47080a6ccac6f70d73a8589a102 |
|
BLAKE2b-256 | 40aaeb9faf20df4c96739b012a0f65062327a77840e83ff95805911af8666a60 |
Hashes for ltp_extension-0.1.7-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de1fa6e971864fcf56d39b78e5fd4dd855a98e1b5cedc61e73f21c5aaafa5f6b |
|
MD5 | c8e8e4e20db499a04eb75f6734d13962 |
|
BLAKE2b-256 | 11acbec6cf59f24d552f1f224b3a17cb6df6d05a40496d79e45b4df33563ab91 |
Hashes for ltp_extension-0.1.7-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35873270d3e0ecbefbd3e943a88bf9540c749f120a441e3c75123ccd57f73610 |
|
MD5 | bc463610af16fde28bb4a033a9b6afa4 |
|
BLAKE2b-256 | e372ad77718563d03def0376a58b2480065b7e0728fe0214b524365da8fed6cc |
Hashes for ltp_extension-0.1.7-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d604e302d566d42ea43e56786bf53481be08d0af79bb704ea6fa99c5d03d617a |
|
MD5 | e33a2b75269376864aec8e6e251a0e22 |
|
BLAKE2b-256 | de1ccd6602299995122227e1cf781e7a4548e0091593fb9f66f7b2dfb9de56de |
Hashes for ltp_extension-0.1.7-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa80e0353ab2c6a663d56133eeec8f50623e82e6d6102bd79e4c29475b649cf9 |
|
MD5 | 6f5b2fbf0686980b44393fe05fa24b3b |
|
BLAKE2b-256 | c6e9a807e426088720f4a3e950907aae54b1e45971651446f90a7a1f59f118d5 |
Hashes for ltp_extension-0.1.7-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42fa5caa388599bc13fd66a76552d2c467d075d9d6b503e8b8e8d836be05c134 |
|
MD5 | 71c8203a43622daea5bbc7b63f03cffc |
|
BLAKE2b-256 | fbe74b62587be9b16feda1a94e9d4903ffcb92110321ca06e9c9a4b249a52ab7 |
Hashes for ltp_extension-0.1.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac6dddd299d5b2352d15261ea079020c369efd56dd6e90106f86558f03dc90fa |
|
MD5 | 32034b9dc4baafb443375310ec316fd5 |
|
BLAKE2b-256 | dd96888a179e8174930683160faf1bcf6dc76c7a995924bc12a0a1b398fb74df |
Hashes for ltp_extension-0.1.7-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc68661ddd6b7b9a5aa2604771cff35a27fe3da421c1bcd1bbba8b85a7ee1d8d |
|
MD5 | 90ad3ad5d78d2ae483bb61828ce1f2e4 |
|
BLAKE2b-256 | 9e0eadfddeef108616a60406abd754c1fa5c2696fd5d412e5d20d104e4ed0faa |
Hashes for ltp_extension-0.1.7-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0153cf042b27d6d1230775e58f41e8cbaf769db40fb6efb4df56be0782d63544 |
|
MD5 | 0f49303993149219cc419981a6e4a657 |
|
BLAKE2b-256 | 0e37100dfdd9deef030a8c998b4836684b00ad519c8e621090552eb911bf559f |
Hashes for ltp_extension-0.1.7-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df99f4a50146fd8837e67cbad754735a141ecd32ee2e55cb0d3e01c8c52d92ce |
|
MD5 | b586bdb8dcca08406911f6a363c2b939 |
|
BLAKE2b-256 | 7ba09447ec925efcfaf9b92268022a3e248a8750dd4d86dd98526fd6012138ed |
Hashes for ltp_extension-0.1.7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 702465a0625a11433aac75ae6b1897dd4f9be656fdfe494c8f14b6a5ecb0b501 |
|
MD5 | a981c01cacf4153cd6a9452981b85ba3 |
|
BLAKE2b-256 | bd5e2eff6c5a4769191562c62319cd6d698a2ce0014e4e96ac04ab79478b4b03 |
Hashes for ltp_extension-0.1.7-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c0d41ee7ff733d20fa13e8c74fffbd7f1aa629f1d4087bec259725bf905a316 |
|
MD5 | c71468c45928f007ec93e831424246e9 |
|
BLAKE2b-256 | 07d06c1dc1820c9a2252b0d3c8e6c4d0a9bb6b4af0dc92cfa2a083d2edf5d054 |
Hashes for ltp_extension-0.1.7-cp37-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1741a01f0607823d3719520775c5743d9108581f1315f3f1638a17c39856e8b3 |
|
MD5 | 87be0b74a30a19dc46889e2975fd5158 |
|
BLAKE2b-256 | f644a5502b540f42a3daeb6e86dc53e12d39d8b99aa7fa0a87c047ddd00a0b9a |
Hashes for ltp_extension-0.1.7-cp37-abi3-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1f76e45cd81266a154756575076665307b95ff532468e14001cc6adb389176e |
|
MD5 | 350662bcf143fe55aee5fc1b23ae07de |
|
BLAKE2b-256 | a81ad19050d59b70382d787da8226c8410211cc86c4920fa374290032e83321c |
Hashes for ltp_extension-0.1.7-cp37-abi3-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b15232aa93e62d1d5ae92eab6ff3c0a18ab1663f1f64c692685808e15b98a35c |
|
MD5 | d048d00f47a7601e6d3ddd681c3811b0 |
|
BLAKE2b-256 | 7df8dc519febc6d3b2af48c2c0e1fa6d60ac7aaa410f62112a1605c5a2846e22 |
Hashes for ltp_extension-0.1.7-cp37-abi3-musllinux_1_2_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8662fee7bcee2dae6e0f6335f80aaee7c75af0cbdc36292bfb5ddc4282ee4018 |
|
MD5 | afd110033428411cfaa85f6eae479cd5 |
|
BLAKE2b-256 | e15c07a2f5619f2c6d09fa093aaec5f927b993ed7205ed2ff5dd200cbfa60b8b |
Hashes for ltp_extension-0.1.7-cp37-abi3-musllinux_1_2_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 261336f47743f5d80b7f9a1e98f77f056ea17c5037814af576ed2f223e1abe16 |
|
MD5 | 1a5d2da5e742ef01957e72cc679a2618 |
|
BLAKE2b-256 | 471355f2f102ea980e1c6e6b9b60b816f03d89d6a51abe046445bd108100540e |
Hashes for ltp_extension-0.1.7-cp37-abi3-musllinux_1_2_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d8965a34d305c176a28f564eddca8131c5feb89c270d365268e83d9a0278103 |
|
MD5 | 8a1d19d1500c62c374499e72bbee5b32 |
|
BLAKE2b-256 | bd441d94a53b6c7bf22c4ca5c5f052fa5beba31a9c5237d0cbc7852a64f5c840 |
Hashes for ltp_extension-0.1.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3d2cbe05d61ae8a36eaa8b6c53bff2b94f73d526da0d9ae81f48c1f72094fb59 |
|
MD5 | 9f652f5beaa8c3d691fd58b67eea6a74 |
|
BLAKE2b-256 | b48492435f75c0b0a7c1ccda3822cdd61dd3e137e0cb0e8796ffe0e54e981cca |
Hashes for ltp_extension-0.1.7-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e722b5a1e07354a77d7510f4be7f435b8735ee0c1f021f2ff9a94c8f102bfab9 |
|
MD5 | 4d063ddda47d388e8cba43f4aab59974 |
|
BLAKE2b-256 | 2dc08dd3ce8025de4f97dae5a54d536549020dd89cff3a464e393be6b899d831 |
Hashes for ltp_extension-0.1.7-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 44d8cc413223639168d970a975be09b4c543e5c31dc3f3549eac8b40a9a14a59 |
|
MD5 | 5a7eeee32ed82cdc8970a1373fd9f05d |
|
BLAKE2b-256 | 1e8b1172d82f39ff3fb9a3528d5128fd5b1a499b4696b31ecf24cc23d94ec250 |
Hashes for ltp_extension-0.1.7-cp37-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 62d769973e2aa40dd2c53e8b88192534cc3d59ad87d5fb96fe4a336e31ffa1a9 |
|
MD5 | e9ec45297f265a5014c5748469721762 |
|
BLAKE2b-256 | 5b0c3459834a8cf242c283eed5252b5884c20817c332a755eaa3422affbe5d72 |
Hashes for ltp_extension-0.1.7-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fcbd0a3d7ef54335c7e56face973d5c3830bdb53db93896676fddff8cd69c631 |
|
MD5 | e7c2ea70b8c74f09bb8e21cbdafc9269 |
|
BLAKE2b-256 | 0966dbbcf6e83ab07fef56c4808722d235e27031f6bdd1b359771b03ef89d999 |
Hashes for ltp_extension-0.1.7-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bfcfdd7a3cf644440e684354dbb8772ab0e4b81029aa5b1fb5035d7ee9de18e9 |
|
MD5 | ec9d3a53dba39060dc6459ed8dc9a283 |
|
BLAKE2b-256 | e8cb06365958b16317c637a8fddeafbee9c0710390feada0e5a3131ca2059f2e |
Hashes for ltp_extension-0.1.7-cp37-abi3-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 021997060869073b8be552f23b3d7f3b2a190e5176258b758963eda82717f5d2 |
|
MD5 | b29149ff16f8b5665ad12df068d1b817 |
|
BLAKE2b-256 | 304ee49fa41a72c3270c3b7b816f1a41cd3299c579b95ba966d4f728771f98fd |
Hashes for ltp_extension-0.1.7-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a216181b73c6732dc83e76abb8838609c7c711a40a89d4af0124edfdff492b1a |
|
MD5 | 780c432807a755dc23d7a4571ee1c7c3 |
|
BLAKE2b-256 | 429863275847c96b4f4083e6ccaf003065c42a5c7524a31fa1dae4a58d890cb6 |
Hashes for ltp_extension-0.1.7-cp37-abi3-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a5efafe5047f7510d7fd50ec46b763b6262d5c89cf68301df0026968727b377 |
|
MD5 | 85ae9faed982ea2d75f5f82a88c06217 |
|
BLAKE2b-256 | 765b0cac5323381fd03412eb8db258b67c96ae2cd01233d4d6501c25436da5b9 |