Rust Extension For Language Technology Platform(Python).
Project description
Language | version |
---|---|
Python | |
Rust |
LTP extension For Python
LTP for Rust 对 Python 的绑定,用于提升 LTP 的速度,以及加入传统机器学习算法实现的中文信息处理工具。
method | ltp 3.0(c++) | ap(1) | ap(8) | pa | pa-i(0.5) | pa-ii(0.5) |
---|---|---|---|---|---|---|
cws | 97.83 | 97.93 | 97.67 | 97.90 | 97.90 | 97.93 |
pos | 98.35 | 98.41 | 98.30 | 98.39 | 98.39 | 98.38 |
ner | 94.17 | 94.28 | 93.42 | 94.02 | 94.06 | 93.95 |
features
- 分句
- 任务
- 中文分词(cws)
- 对数字、英文、网址、邮件的处理
- 支持自定义词典
- 词性标注(pos)
- 支持自定义词典
- 命名实体识别(ner)
- 中文分词(cws)
- 算法
- 平均感知机(ap)
- 单线程平均感知机
- 多线程平均感知机
- 被动攻击算法(pa)
- 平均感知机(ap)
- 模型量化
- 在线学习
- 增量学习
性能测试
评测环境
- Python 3.10
- MacBook Pro (16-inch, 2019)
- 处理器: 2.6 GHz 六核Intel Core i7
- 内存: 16 GB 2667 MHz DDR4
注: 速度测试文件大小为 33.85 MB / 305041 行
分词
我们选择Jieba、Pkuseg、Thulac等国内代表分词软件与 LTP 做性能比较,根据第二届国际汉语分词测评发布的国际中文分词测评标准,对不同软件进行了速度和准确率测试。
在第二届国际汉语分词测评中,共有四家单位提供的测试语料(Academia Sinica、 City University 、Peking University(PKU) 、Microsoft Research(MSR)), 在评测提供的资源icwb2-data 中包含了来自这四家单位的训练集(icwb2-data/training)、测试集(icwb2-data/testing), 以及根据各自分词标准而提供的相应测试集的标准答案(icwb2-data/gold).在icwb2-data/scripts目录下含有对分词进行自动评分的perl脚本score。
我们在统一测试环境下,对若干流行分词软件和 LTP 进行了测试,使用的模型为各分词软件自带模型。在PKU和MSR测试集评测结果如下:
Algorithm | Speed(KB/s) | PKU(F1) | MSR(F1) |
---|---|---|---|
Jieba | 982.49 | 81.8 | 81.3 |
Pkuseg | 109.72 | 93.4 | 87.3 |
Thulac | 48.13 | 94.0 | 87.9 |
Thulac[Fast] | 1133.21 | 同上 | 同上 |
LTP 3(pyltp) | 451.20 | 95.3 | 88.3 |
LTP legacy(1) | 1603.63 | 95.2 | 87.7 |
LTP legacy(2) | 2869.42 | 同上 | 同上 |
LTP legacy(4) | 4949.38 | 同上 | 同上 |
LTP legacy(8) | 6803.88 | 同上 | 同上 |
LTP legacy(16) | 7745.16 | 同上 | 同上 |
注:括号内为线程数量
注2:Jieba的词表是在人民日报数据集上统计的
流水线
除了分词以外,我们也测试了 LTP 三个任务(分词、词性标注、命名实体识别)流水线的速度:
Algorithm | Speed(KB/s) |
---|---|
LTP 3(pyltp) | 153.10 |
LTP legacy(1) | 508.74 |
LTP legacy(2) | 899.25 |
LTP legacy(4) | 1598.03 |
LTP legacy(8) | 2267.48 |
LTP legacy(16) | 2452.34 |
注:括号内为线程数量
注2:速度数据在人民日报命名实体测试数据上获得,速度计算方式均为所有任务顺序执行的结果。
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ltp_extension-0.1.5-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb47c2abfc2fb4a28cbc1f3b8b8afdee1c83959a356cad9696e7b97935398777 |
|
MD5 | a9c5886ab6ec6fb8c5b83d8a1de225b5 |
|
BLAKE2b-256 | 163862be2b4b92f4936e67852198d0cd62b35f28cb71e85f441e09c24225f37b |
Hashes for ltp_extension-0.1.5-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 368a5c1fdc4d05d0cff5d55b7eebef311cb3fb4214403398ce7c92dc56d5d09d |
|
MD5 | bc79a7be2c61c7387f00d9f8f18d3534 |
|
BLAKE2b-256 | 064bd05b4fcd00d3e4d738799afd8329a38b298d507da6771686e232bfba60e0 |
Hashes for ltp_extension-0.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6cb7d80e3f0ddfbbcdd822485c3e2ce6bfe25bd11aac7c5a53382e51b9f572bf |
|
MD5 | c0903be12e9cd17cb5ed103ed1a9a629 |
|
BLAKE2b-256 | dd5d9aa11eb0bbbdbf715de6d22bbef99a6ef595d6b63afbb4273c44d65ca59d |
Hashes for ltp_extension-0.1.5-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 565f1164512080f61a00ba6c5e9df055b31971fc0e5b79cf0fabcf60512c3887 |
|
MD5 | 804e101fafb257ba7d871099e57fb5cd |
|
BLAKE2b-256 | 2accd10261fcf21c1d1c4e8402e27f627a009fb07cbbfebff6b69c32c253d412 |
Hashes for ltp_extension-0.1.5-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f44930a4e51cfba90ad683f9544775305663cb2fd93a4df297a8681e92f4138 |
|
MD5 | a1e1e421b6323a33a24def6acb738883 |
|
BLAKE2b-256 | 26105e3a8e569f4eb7ea3f8b69a84be97035e83f9df4c931045a164eb39b7f41 |
Hashes for ltp_extension-0.1.5-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e036d5ba6698c03529b8b7718c728024f47dfb11dd82f984e10cced8db75aaf7 |
|
MD5 | 449fd2f67d7dee081d09178ee74ae22a |
|
BLAKE2b-256 | 7693413ecc7aae42e1d40b66c2d17764c3c254a0d140cedbf8ddca39fe643716 |
Hashes for ltp_extension-0.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d13d21663d83142ab33fdcb19e89cd34a5c91cd4a497c43cdaacf6f03c865c41 |
|
MD5 | 2298ad864b81b13ef10842598de15199 |
|
BLAKE2b-256 | 6eaf03164f2689f486ac87ddd34840a660dcda1166a9cacfe4223e0027ad8282 |
Hashes for ltp_extension-0.1.5-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69da596f08ac9f6e9dd4b39ee6f888665bb20a0e7d2356d5a807e5d69a824698 |
|
MD5 | e5a8599de3c5f582d4d7a4c48b72c585 |
|
BLAKE2b-256 | 1b33a927c85a1873426a3bc15122bd6467ef6f32118a534cdfac50be8bf0b2ac |
Hashes for ltp_extension-0.1.5-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c569927c4c7e4272ba84243a0567ecc341f1ffaebf779af7696977eff55d8689 |
|
MD5 | aeb449ac1ff02020c60223382b34bc83 |
|
BLAKE2b-256 | 8dc77cc73b9bf14a1e74465019b571eec042d2318acf5179b798d8983c825e66 |
Hashes for ltp_extension-0.1.5-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 70e761e71baf4b39af09e73825759553d816c4dfd60dc5c2771fc74ea12fbdf5 |
|
MD5 | e9a50f7c8ca2a081a33bb2bc984db355 |
|
BLAKE2b-256 | f037b2e8ca0836a3bcdd4be4237f52f613cd5eaa0a28bf0fd914e6ad9c526ffd |
Hashes for ltp_extension-0.1.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 72c5f8f800baf1aafcbdc6a68297f3af8fcc56c631ad5a86a839ac8ede3eabcd |
|
MD5 | b4fb000a15ae8e9aca37894ef080e3a8 |
|
BLAKE2b-256 | aef4d5e89406b337d140b598f47d5b7922bc17fc2d5c708315b7c8ad22edd768 |
Hashes for ltp_extension-0.1.5-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afdbf46d5de3792585a6638250597969d754b8c06015647bb32940a2a5897e0b |
|
MD5 | b686b1718a76f1175af5aa9b736faacf |
|
BLAKE2b-256 | 508edb339adc4884eeb78f482c5f4768f0a3790f4b48fabb425e6257c65f1207 |
Hashes for ltp_extension-0.1.5-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0debd912f276f217604f37f6c3870264464e7270823899613a729631ee0f5844 |
|
MD5 | e84ff2e3f2d6e1b98accb3e224da402c |
|
BLAKE2b-256 | 576893e259d05d26af65084a03df2c94c29dc432594e27d45248bdc6028fb0f9 |
Hashes for ltp_extension-0.1.5-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a180f38793ba0441db346ace07ddf426050f532061f0fffb8b75c899f8fcd49 |
|
MD5 | 09e780a71b8bc45af1e005684d2d6ae6 |
|
BLAKE2b-256 | 9e6334068730b3007e046278a899dd0e6b7b9c3c0e33447a78636ff9043a1236 |
Hashes for ltp_extension-0.1.5-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d052d05d8f4676fa3ee5fd93460679ee20998f8324bab47cde29bb5efd242721 |
|
MD5 | f6bb26b82abb24c062a51508003ea2cc |
|
BLAKE2b-256 | 0bab0a6376a38062833db8ed4f0ea0957fdf02fc6b85cd9f3ab5f0ab6b257a06 |
Hashes for ltp_extension-0.1.5-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04ac6cf4ec9bee3494d5ee2d716b6d8ab7567c4b2454226cbc884400b9b1edbb |
|
MD5 | 0096eac5222ccf138bdc8e2ff2b4b366 |
|
BLAKE2b-256 | 481cb298caba9d7f9851f3929d789833750717bb4094691d15c653c9afed82da |
Hashes for ltp_extension-0.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a027ec18bb5f2c81402866ddadde994c154492f942b7a58b0103fcd23f26680 |
|
MD5 | a3c2bd127803d2f52bc8e5d5196c1d76 |
|
BLAKE2b-256 | 4d6e0403af3d94f4f0252aec29fd64aa7498325763b0ae100b77e7b8adbd5ebc |
Hashes for ltp_extension-0.1.5-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 883a088bca003cba0b87ff5d11b0e94095b8fe0677b852b81351b0885ac5ac54 |
|
MD5 | cd6b3b3ecbc896c5e6e21d6da0d1037f |
|
BLAKE2b-256 | 0eeef2a891ee0fa90f4e08f69ccc49e6178adbd84d4027e3a225896995549c56 |
Hashes for ltp_extension-0.1.5-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 192fab47e2226ec977f95efed00a9f140eaa6eea3790fce81d26470e71711842 |
|
MD5 | 6c66ce0b9aef240d4491602a0fd688bf |
|
BLAKE2b-256 | 7345ff6bbbac8d7e437f514d761436056744b4f591dffbc1dc0025764ffff181 |
Hashes for ltp_extension-0.1.5-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d20d59ea258888a71adcfd7cb4bfc7bce4a7b056be019ff895648ce64c293438 |
|
MD5 | 6d47e0f5a51727a4778a246c6f00d276 |
|
BLAKE2b-256 | 1bc410e3c279219d2b22e1de22933bdbe377ea100d4efbf76c37a46851af4a43 |
Hashes for ltp_extension-0.1.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14f9aa3766283c7c9f1677c26074236a0c56394da09850a81ec00c380c086fc4 |
|
MD5 | 4f5371dae2b1bb2b3bd73919ec5491ce |
|
BLAKE2b-256 | 5bfb91ae430aceff570edcd99c854b305704c808ef8cfd7dd8d983ede6477f10 |
Hashes for ltp_extension-0.1.5-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 766e9a923af2eb80f2bc20e76b038b49679a8d907b9eccbbb467b9bc4cf9b4b5 |
|
MD5 | 70709a720ad3de6aaf1bee58b4663dd1 |
|
BLAKE2b-256 | 64b93b61a18cabd35266d7cdbd76f6af6986020f16d573b0176785ca7a8e23bf |