cotstruct 0.1.0

Chain-of-thought reasoning topology analysis toolkit — deterministic, LLM-free.

cotstruct

Chain-of-Thought Reasoning Topology Analysis

Deterministic, LLM-free toolkit for analyzing the topological structure of chain-of-thought reasoning.

cotstruct extracts and compares path signatures from CoT text — capturing step-type sequences, mathematical entities, reasoning depth, and branching structure — with zero LLM dependency. Every component is fully deterministic: same input → same output, on any machine, at any time.

Eight CoT solutions to the same problem: structured prompts (top-left block) produce consistent reasoning topology; freeform prompts scatter. Generated by cotstruct.visualization — no LLM judge involved.

---

Quick Start

Install from PyPI (Python >= 3.10):

pip install cotstruct

# Optional: classifier validation via Cohen's kappa
pip install cotstruct[validation]

>>> import cotstruct
>>> cotstruct.__version__
'0.1.0'

from cotstruct import StepClassifier, extract_path_signature, compute_sim_struct

# 1. Classify a chain-of-thought into step types
classifier = StepClassifier()
classified = classifier.classify(
    "Let x be the unknown. We calculate 2x + 3 = 11. "
    "Subtract 3: 2x = 8. Therefore, x = 4."
)
print([s.step_type.value for s in classified])
# ['DEF', 'CAL', 'CAL', 'DED']

# 2. Extract the full path signature P(c) = (S, E, D, B)
signature = extract_path_signature(
    "By Pythagorean theorem, a^2 + b^2 = c^2. "
    "Substitute a = 3, b = 4. Calculate c^2 = 9 + 16 = 25. "
    "Therefore, c = 5."
)
print(f"Steps={signature.num_steps} Depth={signature.depth} Branches={signature.branching}")
print(f"Entities: {signature.entity_chain.names()}")
# Entities: ['Pythagorean theorem']

# 3. Compare two signatures with SimStruct
sig1 = extract_path_signature("By Pythagorean theorem, a^2+b^2=c^2. Calculate c^2=9+16=25. So c=5.")
sig2 = extract_path_signature("Use the formula. Plug in values. Compute. Answer is 5.")
similarity = compute_sim_struct(sig1, sig2)
print(f"SimStruct = {similarity:.4f}")

Going further:

• Multi-label classification (HybridClassifier) — see Multi-Label Classification
• Coarse-grained routing (RouterClassifier) — see API Reference
• Batch processing & benchmark evaluation — see API Reference and Running Evaluation
• End-to-end stability analysis demo — see examples/stability_analysis.py

---

Architecture

flowchart TD
    CoT["CoT Text"] --> CLS["Step Classifier<br/>(3-phase rule / Router / Hybrid)"]
    CoT --> ENT["Entity Extractor<br/>(130+ math patterns)"]
    CoT --> DEP["Dependency Graph<br/>(25 connector patterns)"]

    CLS --> S["S — Step Sequence"]
    ENT --> E["E — Entity Chain"]
    DEP --> D["D — Depth"]
    DEP --> B["B — Branching"]

    S --> P["Path Signature<br/>P(c) = (S, E, D, B)"]
    E --> P
    D --> P
    B --> P

    P --> SIM["SimStruct<br/>Similarity"]
    P --> STA["Stability Metrics<br/>IQV · AD · Entropy"]
    P --> EVA["Evaluation<br/>Report"]

    style CoT fill:#1f6feb,color:#fff
    style P fill:#8957e5,color:#fff
    style SIM fill:#238636,color:#fff
    style STA fill:#238636,color:#fff
    style EVA fill:#238636,color:#fff

Three classification modes — all fully deterministic (MLP uses 45 rule-derived features with frozen numpy weights):

Mode	Strategy	Output
Rule-only	3-phase deterministic (regex → keyword → prototype)	10 fine types, single label
Router	Rule high-confidence → direct; low-confidence → MLP	4 coarse types, single label
Hybrid	Rule primary label + MLP complementary labels	10 fine types, multi-label

Modules

Module	Description
cotstruct.classifier	3-phase rule classifier + MLP router + Hybrid multi-label + coarse/fine taxonomies
cotstruct.signature	Path signature extraction: step sequence, entity chain (130+ patterns), depth, branching
cotstruct.metrics	SimStruct weighted topological similarity + stability metrics (IQV, AD, entropy)
cotstruct.data	100-sample benchmark corpus across 7 mathematical domains
cotstruct.evaluation	Full evaluation pipeline: per-class F1, confusion matrix, Cohen's kappa, confidence calibration
cotstruct.visualization	6 plot functions: distribution, confusion matrix, similarity matrix, dependency graph, entity timeline, stability
cotstruct.utils	Logging system, configuration, SimStruct weight presets

Step Type Taxonomy (10 Types)

The classifier assigns each sentence at least one step type from this closed taxonomy. In single-label mode, the highest-confidence type is chosen; in multi-label (Hybrid) mode, complementary labels (e.g. RET + DEF, CAL + DED) are added where MLP signals agreement.

Type	Name	Signal Words	Example
CAL	Calculation	calculate, compute, +, *, =	"2 + 3 = 5, then 5 × 4 = 20."
DED	Deduction	therefore, thus, hence, implies	"Therefore, x = 7."
RET	Retrieval	by theorem, recall, we know	"By the Pythagorean theorem..."
SUB	Substitution	plug in, replace, substitute	"Plug x = 3 into the equation."
VER	Verification	check, verify, make sure	"Let's check: 4² = 16, correct."
BCK	Backtrack	wait, wrong, try again, hmm	"Wait, that's wrong. Let me try again."
HYP	Hypothesis	suppose, assume, case 1	"Suppose n is an integer. Case 1: n is even."
SYN	Synthesis	in conclusion, answer is, therefore the	"In conclusion, the answer is 42."
DEF	Definition	let X be, define, denote	"Let x be the unknown variable."
EXP	Explanation	in other words, because, note that	"In other words, this works because..."

Classification Pipeline

flowchart LR
    SENT["Sentence"] --> P1["Phase 1<br/>Regex patterns<br/>conf 0.6–0.9"]
    P1 -->|no match| P2["Phase 2<br/>Keyword overlap<br/>conf ≤ 0.5"]
    P2 -->|no match| P3["Phase 3<br/>Prototype TF cosine<br/>conf ≤ 0.45"]
    P1 --> DIS["Disambiguation<br/>CAL vs DED · SYN detection"]
    P2 --> DIS
    P3 --> DIS
    DIS --> POS["Position-aware refinement<br/>tail→SYN · start→DEF · BCK→HYP"]
    POS --> OUT["StepType + confidence"]

    style SENT fill:#1f6feb,color:#fff
    style OUT fill:#238636,color:#fff

Multi-Label Classification (Hybrid Rule+MLP)

Many CoT sentences serve multiple reasoning functions simultaneously. For example, "By the Pythagorean theorem, a^2 + b^2 = c^2" is both a definition (introducing variables) and a retrieval (invoking a theorem). The Hybrid classifier captures this via selective MLP augmentation:

final_labels = {rule_primary} ∪ {mlp_labels where prob ≥ per-label threshold}

Selective augmentation strategy: MLP only adds complementary labels for the types where it showed the largest F1 gains over Rule on the training split (RET, SYN, DEF). For CAL/DED (high-frequency, Rule handles well) and rare labels (VER/BCK/HYP/EXP, MLP unreliable), the Rule's primary label is preserved as-is. Note that on the held-out test split the augmentation is roughly F1-neutral versus Rule-only — see Benchmark & Evaluation for honest numbers.

Component	Role
Rule (3-phase)	High-precision primary label, anchors subset accuracy
MLP (sigmoid)	Per-label probability, 10 independent binary classifiers
Per-label thresholds	Tuned on the training split only (e.g. RET=0.35, SYN=0.20, DEF=0.35)
Selective gating	MLP labels used only for {RET, SYN, DEF}

from cotstruct.classifier import HybridClassifier

hybrid = HybridClassifier()
results = hybrid.classify(cot_text)

for cs in results:
    print(f"{cs.labels}  ← {cs.sentence}")
    # ['DEF', 'RET'] ← By Pythagorean theorem, a^2 + b^2 = c^2.
    # ['CAL']       ← 9 + 16 = 25.
    # ['DED']       ← Therefore, c = 5.

# Per-label thresholds (auto-loaded from trained weights)
print(hybrid.per_label_thresholds)
# {'DED': 0.5, 'CAL': 0.4, 'RET': 0.35, 'SYN': 0.2, ...}

Coarse-Grained Taxonomy (4 Types)

For higher-accuracy applications, the 10 fine types collapse into 4 functional groups:

Coarse Type	Fine Types	Description
COMPUTE	CAL + SUB	Numerical/symbolic manipulation
CONCLUDE	DED + SYN	Logical inference and synthesis
REFER	RET + DEF + EXP	External knowledge invocation
META	VER + BCK + HYP	Meta-reasoning about the process

from cotstruct.classifier import CoarseStepType

step = StepType.CAL
print(step.to_coarse())  # CoarseStepType.COMPUTE

Path Signature P(c) = (S, E, D, B)

Component	Description	Details
S — Step Sequence	Ordered list of 10 step types	BoW similarity with positional weighting
E — Entity Chain	Named theorems, formulas, heuristics	130+ mathematical patterns, Jaccard similarity
D — Depth	Longest path in implicit dependency DAG	Built from 25 connector patterns + type heuristics
B — Branching	Backtracks, case splits, rejected sub-goals	7 conditional branch + 7 rejected subgoal patterns

The dependency graph is built deterministically from explicit signals only — no unconditional adjacent edges. A chain with no detectable dependencies has depth 1.

SimStruct

SimStruct(P1, P2) = w_s·Sim_S + w_e·Sim_E + w_d·Sim_D + w_b·Sim_B

Default weights: w_s=0.40, w_e=0.40, w_d=0.10, w_b=0.10

• Sim_S: Step-sequence similarity (BoW + position-weighted bigram overlap)
• Sim_E: Entity-chain similarity (Jaccard over named entities)
• Sim_D: Depth similarity (min/max ratio, normalized to [0,1])
• Sim_B: Branching similarity (min/max ratio, normalized to [0,1])

Weight sensitivity validated via grid search over w_s, w_e in [0.30, 0.50] at 0.05 increments.

Benchmark & Evaluation

A 100-sample hand-labeled corpus across 7 mathematical domains provides ground-truth for both single-label and multi-label classifier evaluation. The corpus contains 525 sentences with an average of 1.17 ground-truth labels per sentence (81 multi-label cases).

Evaluation protocol: the corpus is split into a stratified-by-domain 80/20 train/test partition (80 samples / 417 sentences for training, 20 samples / 108 sentences held out, fixed seed). MLP weights and per-label thresholds are fitted on the training split only; all tables below report performance on the held-out test split unless noted otherwise. Reproduce any split with the --split flag on the evaluation scripts.

Corpus limitations: 20 held-out samples (108 sentences) is a small test set — rare labels (SUB, BCK, EXP) have zero or near-zero test support, so their per-class numbers are not meaningful. All samples are English mathematical CoT; performance on other domains or languages is unvalidated.

Corpus Composition

Domain	Samples
Algebra	20
Arithmetic	15
Geometry	15
Calculus	15
Probability	15
Logic	10
Word Problems	10

Difficulty distribution: 44 easy, 49 medium, 7 hard.

Single-Label Classification (10 Fine Types, Rule-only, test split)

Metric	Value
Micro Accuracy	49.1%
Macro Accuracy	34.8%
Cohen's Kappa	0.312 (fair agreement)
Confidence Gap	+0.200 (well-calibrated)

Step Type	F1-Score	Support
DED	0.625	18
CAL	0.598	56
RET	0.375	9
VER	0.364	6
DEF	0.333	11
HYP	0.200	7
SYN	0.000	1
SUB	—	0
BCK	—	0
EXP	—	0

Multi-Label Classification: Rule vs MLP vs Hybrid (test split)

Metric	Rule-only	MLP-tuned	Hybrid
Hamming loss ↓	0.1019	0.2352	0.1204
Subset accuracy ↑	0.4444	0.0463	0.3611
F1 micro ↑	0.5417	0.4009	0.5149
F1 macro ↑	0.4092	0.3499	0.4009

Label	Support	Rule F1	MLP F1	Hybrid F1
DED	30	0.800	0.720	0.800
CAL	56	0.598	0.746	0.598
VER	10	0.533	0.214	0.533
DEF	13	0.400	0.286	0.333
RET	11	0.333	0.350	0.341
HYP	7	0.200	0.133	0.200
SYN	5	0.000	0.000	0.000

Honest takeaway: on the held-out test split, Rule-only is the strongest multi-label baseline. The MLP's apparent advantage in earlier revisions came from evaluating on its own training data. Hybrid stays close to Rule (F1 macro −0.008) while adding recall on RET, but does not beat it. With only ~50 training sentences per rare label, the MLP overfits — more labeled data is the path to real gains.

4-Class Coarse-Grained: Rule vs MLP vs Router (test split)

Metric	Rule-Only	MLP-Only	Router
Micro Accuracy	56.5%	69.4%	66.7%
Macro Accuracy	52.1%	58.5%	59.5%
Cohen's Kappa	0.349	0.506	0.483

Coarse Type	Rule F1	MLP F1	Router F1
COMPUTE	0.642	0.760	0.726
CONCLUDE	0.642	0.732	0.727
REFER	0.389	0.684	0.650
META	0.286	0.125	0.211

The coarse 4-class task is where the MLP genuinely helps: +13 points micro accuracy over Rule-only on held-out data (5-fold CV on the training split: 66.0% ± 5.9%). The Router routes ~51% of sentences to Rule (high confidence) and ~49% to MLP, trading a little accuracy for interpretable per-sentence provenance. MLP uses 45 rule-derived features with frozen numpy weights, fully deterministic at inference.

Per-Domain Accuracy (Rule-Only, test split)

Domain	Accuracy
Probability	61.5%
Calculus	53.8%
Word Problems	53.3%
Arithmetic	50.0%
Logic	50.0%
Geometry	42.9%
Algebra	40.0%

Note: All results from the deterministic rule+MLP system — no LLM dependency. Trained components use only the 80-sample training split; tables above are the 20-sample held-out test split.

Running Evaluation

# Run full evaluation and generate Markdown report on the held-out test split
python scripts/run_evaluation.py --split test

# Multi-label evaluation (Rule vs MLP vs Hybrid)
python scripts/eval_multilabel.py --split test
python scripts/eval_hybrid.py --split test

# Coarse 4-class routing comparison
python scripts/eval_router.py --split test

# Evaluate on the full corpus or the training split
python scripts/run_evaluation.py --split all
python scripts/run_evaluation.py --split train --output my_report.md

Visualization

Six plot functions for exploratory analysis — the figures in this README are generated by scripts/gen_readme_figures.py using this module:

from cotstruct.visualization import (
    plot_step_distribution,    # Grouped bar chart of step-type frequencies
    plot_confusion_matrix,     # Heatmap of classifier confusion matrix
    plot_similarity_matrix,    # Pairwise SimStruct heatmap
    plot_dependency_graph,     # NetworkX DAG of reasoning dependencies
    plot_entity_timeline,      # Gantt-style entity first-invocation chart
    plot_stability_metrics,    # SR / AD / H across sampling temperatures
)

API Reference

Classifier

classifier = StepClassifier()

# Single sentence
step_type, confidence = classifier.classify_sentence("Therefore, x = 7.")

# Full CoT text
classified = classifier.classify(cot_text)
# Returns list[ClassifiedStep] with .sentence, .step_type, .index, .confidence

# Batch
results = classifier.classify_batch([text1, text2, ...])
sequences = classifier.classify_batch_step_sequences([text1, text2, ...])

# Validation
kappa = classifier.compute_kappa(predictions, ground_truth)

MLP Classifier

from cotstruct.classifier import NumpyMLP, get_coarse_mlp

# Pre-trained coarse-grained MLP (10.7 KB weights, pure numpy)
mlp = get_coarse_mlp()

# Requires features from the rule system
from cotstruct.classifier.feature_extractor import FeatureExtractor
extractor = FeatureExtractor()
feats = extractor.extract_sequence(sentences)  # shape: (n, 45)
labels, confs = mlp.predict(feats)

# Or train your own: python scripts/train_mlp.py

Router (Rule + MLP)

from cotstruct.classifier import RouterClassifier

router = RouterClassifier()
results = router.classify(cot_text)
# Returns list[RouterResult] with .step_type, .coarse_type, .confidence, .source

# Routing stats
print(router.stats)  # {'rule': 55, 'mlp': 53, 'fallback': 0}

# Coarse type sequences
coarse_seq = router.get_coarse_sequence(results)
# ['COMPUTE', 'COMPUTE', 'CONCLUDE']

Hybrid Multi-Label Classifier

from cotstruct.classifier import HybridClassifier

hybrid = HybridClassifier()

# Multi-label classification
results = hybrid.classify(cot_text)
# Returns list[ClassifiedStep] with .sentence, .step_type (primary), .labels (all)

for cs in results:
    print(f"{cs.labels}  ← {cs.sentence[:60]}")
    # ['DEF', 'RET'] ← By Pythagorean theorem, a^2 + b^2 = c^2.
    # ['CAL', 'SYN'] ← Therefore, 9 + 16 = 25.

# Batch
batch_results = hybrid.classify_batch([text1, text2, ...])

# Per-label thresholds
print(hybrid.per_label_thresholds)
# {'DED': 0.45, 'CAL': 0.5, 'RET': 0.35, ...}

# Custom augment labels
hybrid2 = HybridClassifier(augment_labels={"RET", "SYN", "DEF", "CAL"})

Path Signature

extractor = PathSignatureExtractor()

# Single text
sig = extractor.extract(cot_text)

# Batch
sigs = extractor.extract_batch([text1, text2, ...])

# Convenience function
sig = extract_path_signature(cot_text)

# Dependency graph (public)
G = build_dependency_graph(classified_steps)

Metrics

metric = SimStruct(weights=(0.40, 0.40, 0.10, 0.10))

# Full decomposition
result = metric.compute(sig1, sig2)
# result.value, result.components.sim_s, .sim_e, .sim_d, .sim_b

# Scalar only
value = metric.compute_value(sig1, sig2)
# Or: compute_sim_struct(sig1, sig2)

# Weight sensitivity
ws = weight_sensitivity_analysis(sig1, sig2)
# ws.mean, ws.range, ws.min, ws.max

Stability Metrics

from cotstruct.metrics import (
    compute_iqv,                # Inter-quartile variation
    compute_absolute_dispersion,
    compute_centroid_signature,
    compute_path_space_entropy,
    compute_stability_ratio,
)

Use Cases

• LLM researchers: Analyze reasoning quality across models, prompts, and temperatures
• Prompt engineers: Debug how prompt changes affect reasoning structure
• AI safety: Detect "fake reasoning" or reasoning template overfitting
• Model evaluation: Compare reasoning diversity across model families
• Education: Analyze student solution strategies in mathematical problem-solving

See examples/ for a runnable end-to-end demo comparing reasoning stability across prompting styles.

Requirements

• Python ≥ 3.10
• numpy ≥ 1.24.0
• matplotlib ≥ 3.7.0
• networkx ≥ 3.1

Optional: scikit-learn ≥ 1.2.0 (for Cohen's kappa in classifier validation)

Contributing

Contributions are welcome! See CONTRIBUTING.md for setup, testing, and the project's design constraints (determinism, zero heavy dependencies). Release history lives in CHANGELOG.md.

Citation

If you use cotstruct in your research, please cite it (see CITATION.cff):

@software{cotstruct,
  title  = {cotstruct: Chain-of-Thought Reasoning Topology Analysis},
  author = {Feng, Rru},
  year   = {2026},
  url    = {https://github.com/Fengrru/cotstruct},
  version = {0.1.0}
}

License

MIT License.