character span label to tokenized base label for Japanese text
Project description
noyaki
Converts character span label information to tokenized text-based label information.
Installation
$ pip install noyaki
Usage
Pass the tokenized text and label information as arguments to the convert function.
import noyaki
label_list = noyaki.convert(
['明日', 'は', '田中', 'さん', 'に', '会う'],
[[3, 5, 'PERSON']]
)
print(label_list)
# ['O', 'O', 'U-PERSON', 'O', 'O', 'O']
If you want to remove the subword symbol (eg. ##), specify the subword
argument.
import noyaki
label_list = noyaki.convert(
['明日', 'は', '田', '##中', 'さん', 'に', '会う'],
[[3, 5, 'PERSON']],
subword="##"
)
print(label_list)
# ['O', 'O', 'B-PERSON', 'L-PERSON', 'O', 'O', 'O']
Note
Only Japanese is supported.
Only BILOU supports the tag format. (BIO may support in the future)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
noyaki-0.1.1.tar.gz
(2.9 kB
view hashes)
Built Distribution
Close
Hashes for noyaki-0.1.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb04f1f1c507a99cb5ac4ba890089a6184e80ed54cdbf6c05d859f7d5473ca24 |
|
MD5 | b344f4a91bd6c9236ab7ba348cdbb7c9 |
|
BLAKE2b-256 | 0e3f64c9f76314b5c71fe7edbf92c1c2c66bd065f4cb90b35de3fe4611746119 |