kanasplit 1.0.0

A Japanese text tokenizer with POS tagging and Jisho.org integration.

KanaSplit - Japanese Text Tokenizer

A powerful and efficient Japanese text tokenizer with POS tagging and Jisho.org integration.

📌 Overview

KanaSplit is a Japanese text tokenizer designed to break down Japanese sentences into meaningful tokens while providing part-of-speech (POS) tagging. It integrates with Jisho.org to fetch additional lexical data for individual words. The tool is built using MeCab, a popular morphological analyzer for the Japanese language.

🚀 Features

• 🔹 Tokenization: Splits Japanese sentences into words and morphemes.
• 🔹 POS Tagging: Provides grammatical category for each token.
• 🔹 Furigana Support: Extracts readings for kanji words.
• 🔹 Jisho.org API Integration: Retrieves word meanings and definitions.
• 🔹 Command-Line Interface (CLI): Allows easy text tokenization from the terminal.
• 🔹 Error Logging: Captures and logs API failures for debugging.

🔧 Installation

KanaSplit requires Python 3.6+ and MeCab to be installed.

1️⃣ Install Dependencies

pip install -r requirements.txt

2️⃣ Install MeCab (if not installed)

For Windows:

choco install mecab mecab-ipadic

For macOS:

brew install mecab mecab-ipadic

For Linux (Ubuntu/Debian):

sudo apt install mecab mecab-ipadic

3️⃣ Install KanaSplit

pip install .

📖 Usage

1️⃣ Tokenize Japanese Text

kanasplit-cli "私はお寿司を食べたいです。"

Output:

- 私 (名詞)
- は (助詞)
- お (接頭詞)
- 寿司 (名詞)
- を (助詞)
- 食べ (動詞)
- たい (助動詞)
- です (助動詞)
- 。 (記号)

2️⃣ Fetch Word Definitions from Jisho.org

kanasplit-cli -w "寿司"

Output:

Word: 寿司
Reading: すし
Meanings: ['sushi', 'range of dishes made with vinegared rice combined with fish, vegetables, egg, etc.']

🛠 API Usage

You can also use KanaSplit in your Python scripts:

from tokenizer import tokenize_text_with_pos, fetch_word_from_jisho

text = "ハサミを買いたいんですが、文房具売り場は何回ですか？"
tokens = tokenize_text_with_pos(text)
print(tokens)

word_data = fetch_word_from_jisho("寿司")
print(word_data)

📝 Configuration

KanaSplit logs errors in errors.log. You can configure logging settings in tokenizer.py.

🏗️ Development & Contribution

Want to contribute? Feel free to fork this repository and submit a pull request!

git clone https://github.com/byteMe394/KanaSplit.git
cd KanaSplit

📜 License

This project is licensed under the MIT License.

👨‍💻 Author

José Trujillo
GitHub | Email