Rust Extension For Language Technology Platform(Python).
Project description
Language | version |
---|---|
Python | |
Rust |
LTP extension For Python
LTP for Rust 对 Python 的绑定,用于提升 LTP 的速度,以及加入传统机器学习算法实现的中文信息处理工具。
method | ltp 3.0(c++) | ap(1) | ap(8) | pa | pa-i(0.5) | pa-ii(0.5) |
---|---|---|---|---|---|---|
cws | 97.83 | 97.93 | 97.67 | 97.90 | 97.90 | 97.93 |
pos | 98.35 | 98.41 | 98.30 | 98.39 | 98.39 | 98.38 |
ner | 94.17 | 94.28 | 93.42 | 94.02 | 94.06 | 93.95 |
features
- 分句
- 任务
- 中文分词(cws)
- 对数字、英文、网址、邮件的处理
- 支持自定义词典
- 词性标注(pos)
- 支持自定义词典
- 命名实体识别(ner)
- 中文分词(cws)
- 算法
- 平均感知机(ap)
- 单线程平均感知机
- 多线程平均感知机
- 被动攻击算法(pa)
- 平均感知机(ap)
- 模型量化
- 在线学习
- 增量学习
性能测试
分词
我们选择结巴分词、Pkuseg、Thulac等国内代表分词软件与 LTP 做性能比较。我们选择 MACOS 作为测试环境,根据第二届国际汉语分词测评发布的国际中文分词测评标准,对不同软件进行了速度和准确率测试。
在第二届国际汉语分词测评中,共有四家单位提供的测试语料(Academia Sinica、 City University 、Peking University 、Microsoft Research), 在评测提供的资源icwb2-data 中包含了来自这四家单位的训练集(training)、测试集(testing), 以及根据各自分词标准而提供的相应测试集的标准答案(icwb2-data/scripts/gold).在icwb2-data/scripts目录下含有对分词进行自动评分的perl脚本score。
我们在统一测试环境下,对若干流行分词软件和 LTP 进行了测试,使用的模型为各分词软件自带模型。在PKU和MSR测试集评测结果如下:
- File Size: 33.85 MB / 305041 lines
- Hard Ware:
- MacBook Pro (16-inch, 2019)
- 处理器: 2.6 GHz 六核Intel Core i7
- 内存: 16 GB 2667 MHz DDR4
- 图形卡: Intel UHD Graphics 630 1536 MB
Algorithm | Speed(KB/s) | PKU(F1) | MSR(F1) |
---|---|---|---|
Jieba | 982.49 | 81.8 | 81.3 |
Pkuseg | 109.72 | 93.4 | 87.3 |
Thulac | 48.13 | 94.0 | 87.9 |
Thulac[Fast] | 1133.21 | 同上 | 同上 |
LTP 3(pyltp) | 451.20 | 95.3 | 88.3 |
LTP legacy(1) | 1155.11 | 95.2 | 87.7 |
LTP legacy(2) | 2192.58 | 同上 | 同上 |
LTP legacy(4) | 3771.35 | 同上 | 同上 |
LTP legacy(8) | 5763.14 | 同上 | 同上 |
LTP legacy(16) | 7113.26 | 同上 | 同上 |
注:括号内为线程数量 注2:结巴分词的词表是在人民日报数据集上统计的
流水线
除了分词以外,我们也测试了 LTP 三个任务(分词、词性标注、命名实体识别)流水线的速度:
- File Size: 33.85 MB / 305041 lines
- Hard Ware:
- MacBook Pro (16-inch, 2019)
- 处理器: 2.6 GHz 六核Intel Core i7
- 内存: 16 GB 2667 MHz DDR4
- 图形卡: Intel UHD Graphics 630 1536 MB
Algorithm | Time(s) | Speed(KB/s) |
---|---|---|
LTP 3 | 226.40 | 153.10 |
LTP legacy(1) | 90.66 | 382.33 |
LTP legacy(2) | 49.43 | 701.23 |
LTP legacy(4) | 27.98 | 1238.76 |
LTP legacy(8) | 20.11 | 1723.72 |
LTP legacy(16) | 16.99 | 2040.26 |
注:括号内为线程数量
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ltp_extension-0.1.4-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dce05076f7d04d0e434392efbb0f17814cd7bab72684c37d7f51cc1081811f4d |
|
MD5 | 72a83f760b4b6522b62678ba8083a9d0 |
|
BLAKE2b-256 | 1e8a1115be6c6ccc1760a474f47cc451f149b4e1d4d690e0421e446dc66a4745 |
Hashes for ltp_extension-0.1.4-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 75ffb23fede5f13ee1cf5889240e58430ba01fdf704888c03185e4f877584f35 |
|
MD5 | 8a3a2dfe13a9753830ece24661a8fa5e |
|
BLAKE2b-256 | a181f87984743e7a5fd3e7016b74cfd4d53846a54ee69839078b584a095c54c7 |
Hashes for ltp_extension-0.1.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2b415d1d4dc2b5e40b46ffeb328cb82f91f79b2b111e80866e32b71a92c6a7e7 |
|
MD5 | 2ed8b2a297783735ecb2463e7774e648 |
|
BLAKE2b-256 | eebab774e4d3d2bc5f2fa4bed1da1135cb53e80bc459ba1016d70ce4204eecac |
Hashes for ltp_extension-0.1.4-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f003feae5a41a1787f70fc0cc0c9a99d48a9dec8e552b2607dfd3b8050da131e |
|
MD5 | 19a755e98745336fb426b88255acf4a9 |
|
BLAKE2b-256 | 4fe8ef76514444df5046a99df3664958de68a3fb8da0a435e0d2c24764c4de5f |
Hashes for ltp_extension-0.1.4-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b8f88e50e076343c65c4b83468f0828cf6bdce3a163bd84863c0937a6b1c5e4c |
|
MD5 | c3d48bbf92ff5f0bd092e0d4ec69b332 |
|
BLAKE2b-256 | 0653551fcf93d2f0682822ea4b67dc28b61a23b65c752b5c6601c3b4cd962dae |
Hashes for ltp_extension-0.1.4-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc7863cb40793c6b16212e7fedc5058cae1a294fa05ac88a92c3ae252bc1b7f7 |
|
MD5 | 5bc9ff1c15a7ea391e8e16c5a2de0ecb |
|
BLAKE2b-256 | 9d975553df94f1b5d58eeaf914ef07120cfdcb73ec952586ea0a67945c9b59c7 |
Hashes for ltp_extension-0.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd4271ca12248b748b165d5b7a4c7084fb98ea27e87379b405ade6344b5974e7 |
|
MD5 | d0c297e543b59182ebf1a6c6dea1f975 |
|
BLAKE2b-256 | 2f811b0a06f6fb18b782c7b7eb277bb5862e3816b78842eb8e1786346938bdf5 |
Hashes for ltp_extension-0.1.4-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c721cd6b37e5c64b41c1846cca12b90fc4fe307e996940fd945c5cd6722974c4 |
|
MD5 | 6e02af43f4921baa4cc851bdb26f7afa |
|
BLAKE2b-256 | df6ccba5a167f6021fbb9e21f65b587a523892d0970ad6fa20401a1fca64da05 |
Hashes for ltp_extension-0.1.4-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2d9595a1f076c888d0942f0657ff453774f358f7b608f7fac8fb31142a93095 |
|
MD5 | a97012529caa9384df34aad095fdf257 |
|
BLAKE2b-256 | 91530208eef238b40fb66eacb99e21fbd392675ead3cf09acfb17948cade649b |
Hashes for ltp_extension-0.1.4-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f9433bd693a6c65cac8b826a6ebe0fc268b51a1be06b5270167be2741f88554 |
|
MD5 | 4bc735c2331ef0eed06998ce856f82bb |
|
BLAKE2b-256 | ef06197f2011df0092515abacc17782e0a5d9f0f3de6d9642c8d636ceab26d8e |
Hashes for ltp_extension-0.1.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9663ba6268380daad923acca1e510de02c319a497f786b792a70d3df5d77b25d |
|
MD5 | 082a3f476f35fbccc9fa4ca0e576de03 |
|
BLAKE2b-256 | 70a200872efbec50fb7903c0803675e64c6a35833f9d883d1234a0362f973a39 |
Hashes for ltp_extension-0.1.4-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 949227fd731d9ca856c7fd833c17ba02238319f99cad7a4c473c064b1ece5db5 |
|
MD5 | 8c0934e98c1df661127693a38d625131 |
|
BLAKE2b-256 | a238594d84c95cfc83fb49599deecee284a6230803ca583bb29a244fcfc15619 |
Hashes for ltp_extension-0.1.4-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd1443ba20f2cbe62f1075ad89a63760920819404b3fb445b3888e7fcccaafb2 |
|
MD5 | 16c392fe38957c9f9557822045ce38f0 |
|
BLAKE2b-256 | 7146360c54f9843ce7210b110d8d834ffb41618c8b99b730d0ba2a79f33a06c3 |
Hashes for ltp_extension-0.1.4-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d0df6e9f2f8ea1b0e3fac00b4540843b637c178aa316b990707f9d20d1a0ba0d |
|
MD5 | f1394ff2a2eaec8d7109e87c6047e335 |
|
BLAKE2b-256 | e5ae2e3264a466ccadb3432aa6ca3b123a6d4c23e027f17f2bb97d46b46f3e52 |
Hashes for ltp_extension-0.1.4-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9111f95f75e5404cbdcb75904d54dc0172d4b15e1525d4da3a53d42db3edcd64 |
|
MD5 | 61eec7618466d9f53398d21d6c68db0c |
|
BLAKE2b-256 | efa8adeb325e64fca7181c66846a98d1671d366f3a11d3052fc79eae7496ca67 |
Hashes for ltp_extension-0.1.4-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a518767cb1b0217b06708e5236151f227ccc956f48a243b8304ba32709cb8ea |
|
MD5 | a8777b45eff9c051ddd5d972ab9a0bb6 |
|
BLAKE2b-256 | e2bbce751a47e2e17709b41e42d5ea3bcbf4c63aba453bdfd2703ae0e7dce8d8 |
Hashes for ltp_extension-0.1.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af718f8c835083a4062b81e75fb35849741c1efa2a9710be8cfe044b38d0b2ec |
|
MD5 | 800baf1dfe1b0cbda23502617d5ab404 |
|
BLAKE2b-256 | 8a82e3df21072d23c5ac31642ddfc1620efab1a5f633b651b5305cc2e22862f9 |
Hashes for ltp_extension-0.1.4-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 892cf854a2e677e2f8da3a1f33bfa3277b082406dd67123efe61b7d322f94e42 |
|
MD5 | 1c21c9bd5f8409ea4469729c23c307e1 |
|
BLAKE2b-256 | 27a87f1447d68aa59714d2175bb175078bd3b4a4713e2ba135a5967878542de2 |
Hashes for ltp_extension-0.1.4-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4dc580077cd752dc59d8180d1e3d26df20faa2e14a0db77a3d6f2523d09ad513 |
|
MD5 | 240128285550a63dfbe6206386624ef0 |
|
BLAKE2b-256 | 6ad57f282ecb4b56b386429b77d107ab826e5d7d3d18a50d06ec2b9fcc408c10 |
Hashes for ltp_extension-0.1.4-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5db6d363905c7e3a6d4e359e6f4109c3d362cdaacd5849fa5ee11adfd5b67dcf |
|
MD5 | 2866bdf8f60cea8d2105cc4eb7a2e308 |
|
BLAKE2b-256 | 7b5ef23ff205d36de5598e73b18e51dfbecb7345cc3c26e24d46932804ce1f7e |
Hashes for ltp_extension-0.1.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 372c2300b93e1226a470ea96fd0e137f9add723e6e7b2832eec5fe0146733b23 |
|
MD5 | 369a5f869c64c5c1530b62d3a778cfda |
|
BLAKE2b-256 | adfdb336f68257e366e1841b9db025d870c9f3c00053fd2df55cde3284892f62 |
Hashes for ltp_extension-0.1.4-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 289c8bf6fb78984ef13dae67f7b6b0ea028bd91e07187b52410f644a957376b2 |
|
MD5 | b6d73f4bd01b8847fe720d8a7e0a9524 |
|
BLAKE2b-256 | 05f186ed1f6e689dde09925617e3a7932c403c6d74f64eb9cbae13131b295027 |