sentence-struct

Analyze text into a learner-friendly structure:

document → sentences[] → { tokens[], chunks[] }

Chunks are siblings of tokens and reference them via tokenIndices (not nested).

Japanese (ja) via GiNza / Sudachi (furigana + POS coloring).

Other languages tokenize with transparent posGroup/typeGroup "_" (one chunk per token). Some also fill reading:

Install

pip install sentence-struct
# or
uv add sentence-struct

English IPA needs the spaCy small model:

python -m spacy download en_core_web_sm

(Local uv: uv sync --group en-model.)

Optional larger Sudachi dictionary:

pip install "sentence-struct[ja-full]"

Requires Python ≥ 3.11. First install pulls spaCy / GiNza / Sudachi (hundreds of MB).

Usage

from sentence_struct import analyze

doc = analyze("秋が近づくにつれ、朝晩の涼しさが心地よい。", language="ja")
print(doc["sentence_count"], doc["token_count"], doc["chunk_count"])
print(doc["sentences"][0]["tokens"][0])
print(doc["sentences"][0]["chunks"][0])

en = analyze("I refuse the refuse.", language="en")
ko = analyze("안녕하세요", language="ko")
th = analyze("สวัสดีครับ", language="th")
zh = analyze("你好，世界。", language="zh")

CLI:

sentence-struct "秋が近づく。"
sentence-struct -f essay.txt -o out.json

Schema (abbreviated)

{
  "language": "ja",
  "text": "...",
  "sentences": [
    {
      "index": 1,
      "text": "...",
      "tokens": [
        {
          "text": "散歩する",
          "posGroup": "動詞",
          "reading": "さんぽする",
          "segments": [
            {"text": "散歩", "reading": "さんぽ"},
            {"text": "する"}
          ]
        }
      ],
      "chunks": [
        {"type": "NP", "typeGroup": "PHRASE", "tokenIndices": [0, 1], "text": "秋が", "role": "nsubj"}
      ]
    }
  ]
}

Develop (uv)

uv sync
uv run pytest -m "not integration"
uv run pytest -m integration
uv run sentence-struct "今日は良い天気です。"

Release

See RELEASING.md. Tag vX.Y.Z → GitHub Actions builds with uv and publishes to PyPI via Trusted Publishing.

License

MIT