Centering-Lgram

Discourse cohesion analysis based on Centering Theory (Grosz, Joshi, and Weinstein, 1983). Measures how entities and topics flow across sentences — pronouns, repetitions, entity continuity.

Note: Cohesion (bağdaşıklık) = surface grammatical/lexical links. Coherence (tutarlılık) = deeper semantic unity. Centering Theory models cohesion.

⚠️ What This Tool Does NOT Measure

• Factual accuracy — a high cohesion score does NOT mean the content is correct
• Hallucination — a fluent-sounding LLM output can still be completely false
• Overall quality — cohesion is ONE dimension of text quality, not the whole picture

Correct positioning: Use centering-lgram as a complementary evaluator alongside faithfulness/accuracy checkers. It measures "how smoothly does this read?", not "is this correct?".

Quick Start

pip install centering-lgram
python -m spacy download en_core_web_sm

from lgram import TextAnalyzer

ta = TextAnalyzer()
r = ta.analyze("AI helps doctors. It speeds up diagnosis. These tools save lives.")
print(r.overall_cohesion)  # 0.0-1.0 (exact value depends on the spaCy model)
print(r.quality)           # "high"

For best results, use the medium model:

ta = TextAnalyzer("en_core_web_md")

Genre Calibration

Empirically derived transition patterns. Method: Tukey's fence (p75 + 1.5×IQR).

Brown Corpus (1960s)

NLTK Brown: 1.1M words, 500 files, 15 categories. n=30/genre, HIGH confidence.

Modern Corpus (2020s) — n=30/genre, HIGH confidence

Note: Single-author corpus. Stylistic homogeneity may narrow distributions.

Cross-Validation: Wikipedia (multi-author, 2024)

n=12, MEDIUM confidence. Small sample — treat as observational, not conclusive.

Wikipedia's 52% flag threshold is broadly consistent with Brown's 58%. Direction is as expected (multi-author corpora cluster together, single-author is the outlier). However, at n=12 this is an observation, not a validated claim. Two caveats: (1) Wikipedia's strict editing guidelines may make it a distinct sub-genre, not representative of general 2020s expository writing. (2) The 6% gap could be measurement noise, temporal change, or genre artifact — the current data cannot distinguish between these explanations.

Findings

1. Rough-Shift >50% is abnormal — all corpora agree. Flag thresholds: 51-63%.
2. Wikipedia cross-validates the multi-author finding — Brown (58%) and Wikipedia (52%) cluster together, while the single-author corpus (63%) diverges. Consistent with the stylistic homogeneity hypothesis. However, n=12 precludes strong conclusions about temporal stability.
3. Wikipedia is not "general 2020s writing" — its strict editing guidelines may constitute a distinct sub-genre. More diverse modern sources needed.

Calibration is reproducible: python -m lgram.brown_calibration

⚠️ Reliability note: Scores depend on the underlying embedding model. For production use, pick ONE model (recommended: en_core_web_md) and standardize on it. Compare texts only within the same genre — cross-genre comparison is meaningless because different genres have different natural transition patterns.

Core Concepts

Centering Theory tracks three discourse centers per utterance:

Five transition types between utterances:

API — TextAnalyzer (High-Level)

from lgram import TextAnalyzer

ta = TextAnalyzer()                    # default: en_core_web_sm
ta = TextAnalyzer("en_core_web_md")    # better vectors
ta = TextAnalyzer(use_sentence_transformers=True)  # best quality

Core Analysis

Cohesion Metrics

Segmentation & Quality

Export

API — EnhancedCenteringTheory (Low-Level)

from lgram import EnhancedCenteringTheory
import spacy

nlp = spacy.load("en_core_web_sm")
ct = EnhancedCenteringTheory(nlp)

state = ct.analyze_utterance("John went to the store.")
ct.update_discourse("He bought milk.")

result = ct.evaluate_cohesion(["John went.", "He bought milk.", "The store was busy."])

Key methods: compute_forward_centers, compute_backward_center, determine_transition, extract_clauses, detect_boundaries, validate_sequence, visualize, compare_texts, stream_start/feed/flush, save/load, reset.

CLI

centering-lgram analyze --text "John went to the store. He bought milk."
centering-lgram clauses --text "She left because she was tired."
centering-lgram full --text "John left. He was tired because he worked late."
centering-lgram score --text "Alice met Bob. She greeted him."
centering-lgram info
centering-lgram version

How It Works

Salience Ranking

Cf ordered by: grammatical role (S=4 > O=3 > other=2 > poss=1) + POS (PRON=3 > PROPN=2 > NOUN=1) + position + entity type (PERSON/ORG/GPE bonus) + pronoun antecedent bonus.

Backward Center (5-level cascade)

1. Possessive scan — "his"/"her" → matched person entity
2. Direct match — entity appears in both Cf lists
3. Pronoun resolution — gender-aware (he→male, she→female)
4. Coreference — entity type matching + vector similarity fallback
5. Compound plural — multiple persons → "they"

Gender-Aware Pronoun Resolution

120+ name gender map (English + Turkish) + title detection (Mr/Mrs) + suffix heuristics. Male pronoun "he" does NOT match female entity "Alice".

Clause Detection

Dependency parse: main, conj, advcl, ccomp, acl, relcl. Separator tokens (commas, conjunctions) assigned to following clause.

Architecture

lgram/
  __init__.py              # Package exports
  analyzer.py        924   # TextAnalyzer (17 methods)
  benchmark.py       290   # CohesionBenchmark (4 tests)
  cli.py             238   # 6 CLI commands
  core.py              7   # Re-export hub
  utils.py            20   # Logging
  models/
    __init__.py         7   # Sub-package exports
    centering_theory.py 1122 # Core engine
tests/
  test_lgram.py       209   # 15 core tests
  test_edges.py       312   # 34 edge case tests
docs/
  RESEARCH.md                # Literature survey
  IMPLEMENTATION_PLAN.md     # Implementation plan

Dependencies: spacy>=3.4.0 only. Optional: sentence-transformers for MiniLM.

Use Cases

License

MIT — see LICENSE. Copyright (c) 2025 Ilker Atagun.