Rust Extension For Language Technology Platform(Python).
Project description
Language | version |
---|---|
Python | |
Rust |
LTP extension For Python
LTP for Rust 对 Python 的绑定,用于提升 LTP 的速度,以及加入传统机器学习算法实现的中文信息处理工具。
method | ltp 3.0(c++) | ap(1) | ap(8) | pa | pa-i(0.5) | pa-ii(0.5) |
---|---|---|---|---|---|---|
cws | 97.83 | 97.93 | 97.67 | 97.90 | 97.90 | 97.93 |
pos | 98.35 | 98.41 | 98.30 | 98.39 | 98.39 | 98.38 |
ner | 94.17 | 94.28 | 93.42 | 94.02 | 94.06 | 93.95 |
features
- 分句
- 任务
- 中文分词(cws)
- 对数字、英文、网址、邮件的处理
- 支持自定义词典
- 词性标注(pos)
- 支持自定义词典
- 命名实体识别(ner)
- 中文分词(cws)
- 算法
- 平均感知机(ap)
- 单线程平均感知机
- 多线程平均感知机
- 被动攻击算法(pa)
- 平均感知机(ap)
- 模型量化
- 在线学习
- 增量学习
性能测试
评测环境
- Python 3.10
- MacBook Pro (16-inch, 2019)
- 处理器: 2.6 GHz 六核Intel Core i7
- 内存: 16 GB 2667 MHz DDR4
注: 速度测试文件大小为 33.85 MB / 305041 行
分词
我们选择Jieba、Pkuseg、Thulac等国内代表分词软件与 LTP 做性能比较,根据第二届国际汉语分词测评发布的国际中文分词测评标准,对不同软件进行了速度和准确率测试。
在第二届国际汉语分词测评中,共有四家单位提供的测试语料(Academia Sinica、 City University 、Peking University(PKU) 、Microsoft Research(MSR)), 在评测提供的资源icwb2-data 中包含了来自这四家单位的训练集(icwb2-data/training)、测试集(icwb2-data/testing), 以及根据各自分词标准而提供的相应测试集的标准答案(icwb2-data/gold).在icwb2-data/scripts目录下含有对分词进行自动评分的perl脚本score。
我们在统一测试环境下,对若干流行分词软件和 LTP 进行了测试,使用的模型为各分词软件自带模型。在PKU和MSR测试集评测结果如下:
Algorithm | Speed(KB/s) | PKU(F1) | MSR(F1) |
---|---|---|---|
Jieba | 982.49 | 81.8 | 81.3 |
Pkuseg | 109.72 | 93.4 | 87.3 |
Thulac | 48.13 | 94.0 | 87.9 |
Thulac[Fast] | 1133.21 | 同上 | 同上 |
LTP 3(pyltp) | 451.20 | 95.3 | 88.3 |
LTP legacy(1) | 1603.63 | 95.2 | 87.7 |
LTP legacy(2) | 2869.42 | 同上 | 同上 |
LTP legacy(4) | 4949.38 | 同上 | 同上 |
LTP legacy(8) | 6803.88 | 同上 | 同上 |
LTP legacy(16) | 7745.16 | 同上 | 同上 |
注:括号内为线程数量
注2:Jieba的词表是在人民日报数据集上统计的
流水线
除了分词以外,我们也测试了 LTP 三个任务(分词、词性标注、命名实体识别)流水线的速度:
Algorithm | Speed(KB/s) |
---|---|
LTP 3(pyltp) | 153.10 |
LTP legacy(1) | 508.74 |
LTP legacy(2) | 899.25 |
LTP legacy(4) | 1598.03 |
LTP legacy(8) | 2267.48 |
LTP legacy(16) | 2452.34 |
注:括号内为线程数量
注2:速度数据在人民日报命名实体测试数据上获得,速度计算方式均为所有任务顺序执行的结果。
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ltp_extension-0.1.6-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c9aee735d8ca04d7783768dfd20ec46138256925cc73c432d6b21c2ca36b8548 |
|
MD5 | f85755c77cf1ce03b97a22cbf9419358 |
|
BLAKE2b-256 | ce6f23a971873d487667ab3acf3264c4cd5b33326f50c0903d54f588660b496b |
Hashes for ltp_extension-0.1.6-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f3e7ae22f10b85e0fd1e0fa306af4e843f56f83fe1a24392cc5deefb2c3887b1 |
|
MD5 | 4da5a497e00b940e5300167820c77907 |
|
BLAKE2b-256 | 0f721edc5f138a678b3b46ec18c6f01d1df08d75ce9acd3016b9e9e73d70dda3 |
Hashes for ltp_extension-0.1.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 111ee27cdfc6f7d8d8843f60cd07627acf24d35a829de228ac7fce97f258d970 |
|
MD5 | 1d2e256b0c884f578516adefa606a26f |
|
BLAKE2b-256 | c1a6223cf29bfcab901f177a05ab039e8c2196b19cb85376bf837c225b454014 |
Hashes for ltp_extension-0.1.6-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0c5ae9dfb73a2e5f1423805a8d45e76bd2b1d0b20fb2da20af02b5ee3f8401c5 |
|
MD5 | 671d1fd68f70714204d66476093c09b6 |
|
BLAKE2b-256 | a0a8f5c320752bce7de8654646fe795225eafae5fad2d0f00b2e7d1ea1f8d031 |
Hashes for ltp_extension-0.1.6-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09a983ca0b188d15e25975ef89070005bdaf31df5fefd130a06ee1226630ded9 |
|
MD5 | 2f949c685cbc50971e584da0487b5a85 |
|
BLAKE2b-256 | 58a1773f5050648644ecb42e3c01f544a08fb94926d4d14d207e47c7ff017b70 |
Hashes for ltp_extension-0.1.6-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4557b20b5a84aac067378745c38e2e40337a21756b4ac00b4a687af13632c1b4 |
|
MD5 | 15edb19f3737f12f7bf35a93ddfa4a23 |
|
BLAKE2b-256 | 8a6dca245f223991d49d70ddeac3cb189f590a31a5c0821cb330d7d407e59ff9 |
Hashes for ltp_extension-0.1.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e80ce3d5c816ea2c5029764daa95cc79f5971f57289dcb29da9f4f0b9e2bb00 |
|
MD5 | c894b73b57831a3ee677457ba804f5dc |
|
BLAKE2b-256 | 92ecd072b92c5a726e90918d0bfd4691b9bba03ed4bd5978283d8f06dc9a1617 |
Hashes for ltp_extension-0.1.6-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | beba938f8094ee01ad4e68f8c70e8a710d151ba69c214b99d23b2b72959c43e0 |
|
MD5 | 0693b0876b43e5575ed966fa75266eca |
|
BLAKE2b-256 | 1f2f6812514b3dd6e796a96162354bcec843514cd22f48575983bae97badc2a6 |
Hashes for ltp_extension-0.1.6-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d697b63a0f0037fb1fda050326ef84e96ce3e537d0217c0e496c7a7990977a0 |
|
MD5 | ec8b5fde48c617d5ed2f4f450ccf8392 |
|
BLAKE2b-256 | 9f34364fbda5c806d21f82dac339b735a6464e305cb26041a2fd0eeff487f161 |
Hashes for ltp_extension-0.1.6-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e662d7cac25f9084be7ef15fdd33dcde916a048c650d95b9c1fb6beb62dc06a2 |
|
MD5 | f888d2ce6971453348f2a6b8c7b42371 |
|
BLAKE2b-256 | 52241093044229ab1c45bd7208d80651f9e33f88194dc9d9e33f2bc74c3617c7 |
Hashes for ltp_extension-0.1.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1bf8d32fd7f798f06d327087dc296c705a441ed41bb43369caa7b8aae72edf4 |
|
MD5 | bccdb49a87cdbfb01795993744da5fc5 |
|
BLAKE2b-256 | 6a5a24e2f2749f907581b9c21838e0a3f010ac5e8d0497e9f020a678f539279c |
Hashes for ltp_extension-0.1.6-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 96a52c4786f6873e1eb88c4ea2dc90810da44371357b72f8415bdbf1eb4faa8a |
|
MD5 | f6b585a406265b78226279ab00eccf6c |
|
BLAKE2b-256 | 704b873076d6f76fa5b76d132f9e92c46f0f4f94f94e97891de15b55ce477a29 |
Hashes for ltp_extension-0.1.6-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 571817676d14bcd78279c3a6bf1b5dd1a773d4f3b64c4f752a4c19d671da1dc2 |
|
MD5 | 89f2879ffcbd44b1eae31adf882aa2c2 |
|
BLAKE2b-256 | 824daeadf73cdee5e8fde72b8bc83aaf4da069615292a1c8e3e4407cfabadedc |
Hashes for ltp_extension-0.1.6-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 123f1327e4d689e0da56dd18c28fdc9f0d56a158851a395ba5f5edb82de94184 |
|
MD5 | 0c5822c1b593297b3aa2f7f41c32e756 |
|
BLAKE2b-256 | 1c129611fcc7ac4ccf2d097105a84b022ad43a48fa3dbb794fc30d3edd978f6d |
Hashes for ltp_extension-0.1.6-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0755c77755bac3deee34f38a7682179406f6ad8a4c70d1b7e0bbcd9d3303a604 |
|
MD5 | 59874bd982643504709aba060dc12f25 |
|
BLAKE2b-256 | df3806941b9488ba39c318cee75bb13ddba7a3d0e38f49ce71640e0cd1ff53c1 |
Hashes for ltp_extension-0.1.6-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 078be8d5e802502ea41aa946f4d3f8253302a7d49cace0f914ca557422c23249 |
|
MD5 | 29c4219e276a3e4856bebc7b6d25987f |
|
BLAKE2b-256 | df4c1d6dd09882002f628f8333e29644945416f9a68e16c9339fd6e7adfbc235 |
Hashes for ltp_extension-0.1.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a9139473ad17137897d992f2284f6bd23bbbdc7a3c6ee825e487a322bc0457a |
|
MD5 | cf4c64d5436b4ea0799a6e9513cbcdc7 |
|
BLAKE2b-256 | 10269985d931363842bde39e3aa585fb69728cec0ae515ec54dc8dd20dd02efd |
Hashes for ltp_extension-0.1.6-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 394fca63fc66022cc9f98005d8febcc1df706b5f3ada074e4c0549b6e678d592 |
|
MD5 | 4ea18128096f8c0d71df9116bc1fd97b |
|
BLAKE2b-256 | ceed12e396ead23f5b5605fbd9a95a77c5a5d8136183597e9cc819a9477c431c |
Hashes for ltp_extension-0.1.6-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d0b84863db4610031f77fe48fc5fa417ff2fc3153577a34bebc0a61aab1e1275 |
|
MD5 | 569211e9c74a56971b67f8134d24428f |
|
BLAKE2b-256 | 1fca348fa7aa6e705920ada41e295eafef68340d9425dae30ac2c5f3fcf4e0e2 |
Hashes for ltp_extension-0.1.6-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 45e45d9a228910b941a52f67aa8b0334ef05927ee91fe26014b5db12eca8cd84 |
|
MD5 | 94196778c1616d46535a720815061ca8 |
|
BLAKE2b-256 | 102bd629d1a6a9285ce3869c960b7e6bb091767944b898a6a1086cc713aa98e0 |
Hashes for ltp_extension-0.1.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 72634eaed0a47062e4c94b43531dcdf998d9bb81f329d9978a57e39af81808ec |
|
MD5 | 3203d3bfc0aedd3db89e7e09b04daf68 |
|
BLAKE2b-256 | 4ce2c20c0fa44e4802d38541d4c5b73fba47f2e4965293572e7a71ad529dd33c |
Hashes for ltp_extension-0.1.6-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cbcd8f546b6e0881924dd81e2760601bb7ad246132ba74202a4b8e4653c32046 |
|
MD5 | 9d5210eaa5eed8fa4700cc6300a8fa79 |
|
BLAKE2b-256 | b9e03569ee65271e05cd9ade43bea565b0b20c4b6b7fb6a9bacf6dbd5f7abbf0 |