jptranstokenizer 0.0.4

Japanese tokenizer with transformers library

jptranstokenizer: Japanese Tokenzier for transformers

This is a repository for japanese tokenizer with HuggingFace library.

issue は日本語でも大丈夫です。

Table of Contents

1. Usage
2. Roadmap
3. Citation
   • This Implementation
4. Licenses
5. Related Work

Usage

To be added

Roadmap

See the open issues for a full list of proposed features (and known issues).

Citation

There will be another paper for this pretrained model. Be sure to check here again when you cite.

This Implementation

@misc{suzuki-2022-github,
  author = {Masahiro Suzuki},
  title = {jptranstokenizer: Japanese Tokenzier for transformers},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/retarfi/jptranstokenizer}}}

Licenses

The codes in this repository are distributed under the Apache License 2.0.

Related Work

• Pretrained Japanese BERT models (containing Japanese tokenizer)
  • Autor NLP Lab. in Tohoku University
  • https://github.com/cl-tohoku/bert-japanese
• SudachiTra
  • Author Works Applications
  • https://github.com/WorksApplications/SudachiTra
• UD_Japanese-GSD
  • Author megagonlabs
  • https://github.com/megagonlabs/UD_Japanese-GSD
• Juman++
  • Author Kurohashi Lab. in Universyti of Kyoto
  • https://github.com/ku-nlp/jumanpp