jptranstokenizer 0.2.0

Japanese tokenizer with transformers library

jptranstokenizer: Japanese Tokenzier for transformers

This is a repository for japanese tokenizer with HuggingFace library.
You can use JapaneseTransformerTokenizer like transformers.BertJapaneseTokenizer.
issue は日本語でも大丈夫です。

Documentations

Documentations are available on readthedoc.

Install

pip install jptranstokenizer

Quickstart

This is the example to use jptranstokenizer.JapaneseTransformerTokenizer with sentencepiece model of nlp-waseda/roberta-base-japanese and Juman++.
Before the following steps, you need to install pyknp and Juman++.

>>> from jptranstokenizer import JapaneseTransformerTokenizer
>>> tokenizer = JapaneseTransformerTokenizer.from_pretrained("nlp-waseda/roberta-base-japanese")
>>> tokens = tokenizer.tokenize("外国人参政権")
# tokens: ['▁外国', '▁人', '▁参政', '▁権']

Note that different dependencies are required depending on the type of tokenizer you use.
See also Quickstart on Read the Docs

Citation

There will be another paper. Be sure to check here again when you cite.

This Implementation

@misc{suzuki-2022-github,
  author = {Masahiro Suzuki},
  title = {jptranstokenizer: Japanese Tokenzier for transformers},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/retarfi/jptranstokenizer}}}

Related Work

• Pretrained Japanese BERT models (containing Japanese tokenizer)
  • Autor NLP Lab. in Tohoku University
  • https://github.com/cl-tohoku/bert-japanese
• SudachiTra
  • Author Works Applications
  • https://github.com/WorksApplications/SudachiTra
• UD_Japanese-GSD
  • Author megagonlabs
  • https://github.com/megagonlabs/UD_Japanese-GSD
• Juman++
  • Author Kurohashi Lab. in Universyti of Kyoto
  • https://github.com/ku-nlp/jumanpp