sentence-struct

Analyze text into a learner-friendly structure:

document → sentences[] → { tokens[], chunks[] }

Chunks are siblings of tokens and reference them via tokenIndices (not nested).

Japanese (ja) via GiNza / Sudachi. English (en) is tokenize-only (spacy.blank("en")): tokens with posGroup/typeGroup "_", one chunk per token.

Install

pip install sentence-struct
# or
uv add sentence-struct

Optional larger Sudachi dictionary:

pip install "sentence-struct[ja-full]"

Requires Python ≥ 3.11. First install pulls spaCy / GiNza / Sudachi (hundreds of MB).

Usage

from sentence_struct import analyze

doc = analyze("秋が近づくにつれ、朝晩の涼しさが心地よい。", language="ja")
print(doc["sentence_count"], doc["token_count"], doc["chunk_count"])
print(doc["sentences"][0]["tokens"][0])
print(doc["sentences"][0]["chunks"][0])

CLI:

sentence-struct "秋が近づく。"
sentence-struct -f essay.txt -o out.json

Schema (abbreviated)

{
  "language": "ja",
  "text": "...",
  "sentences": [
    {
      "index": 1,
      "text": "...",
      "tokens": [
        {"text": "秋", "pos": "名詞", "posGroup": "名詞", "lemma": "秋", "reading": "あき"}
      ],
      "chunks": [
        {"type": "NP", "typeGroup": "PHRASE", "tokenIndices": [0, 1], "text": "秋が", "role": "nsubj"}
      ]
    }
  ]
}

Develop (uv)

uv sync
uv run pytest -m "not integration"
uv run pytest -m integration
uv run sentence-struct "今日は良い天気です。"

Release

See RELEASING.md. Tag vX.Y.Z → GitHub Actions builds with uv and publishes to PyPI via Trusted Publishing.

License

MIT