RTSA

Reasoning Trace Structure Analysis — study Chain-of-Thought reasoning as a structured graph; no white-box model access required.

Documentation · API Reference · Comparison · Changelog

Motivation

Long reasoning traces are expensive to generate, opaque to inspect, and hard to verify — yet most tooling treats them as flat strings. RTSA parses CoT text into a typed DAG (Retrieve / Transform / Verify / Branch / Backtrack / Compare) and makes structural analysis practical: where the redundancy is, which step is likely wrong, whether structure predicts correctness, and who wrote the trace. Everything is computed from the text alone, so it works with any API-only model, requires no annotations, and every analysis is reproducible through a versioned experiment entrypoint.

Example Extraction

Real extraction (rule-based) of a MATH problem's human solution: 6 nodes, 8 edges. Each node type carries a distinct color; the graph is the input to every downstream analysis.

Quickstart

pip install rtsa

from rtsa.extractors import RuleBasedExtractor
from rtsa.core.ngs_validator import NGSValidator
from rtsa.analysis.prune import RedundancyAnalyzer, PruneConfig

text = "Retrieve x=3. Transform: x*2=6. Verify: 6 is even."
graph = RuleBasedExtractor().extract(text, trace_id="demo_001")

valid, violations = NGSValidator().validate(graph)
report = RedundancyAnalyzer(config=PruneConfig()).analyze(graph, apply_pruning=True)
print(report.summary())

rtsa extract cot.txt --extractor rbe --output graph.json
rtsa validate graph.json
rtsa prune graph.json --apply --output pruned.json

What RTSA Answers

Capabilities

Maturity levels: Stable (battle-tested, covered by tests) · Evolving (functional, API may shift) · Experimental (proof of concept, optional deps).

Pipeline

raw CoT text (JSONL / HuggingFace datasets)
    |  extractors: RBE (rule) · SBE (syntax) · LLM · random baselines
    v
ReasoningTraceGraph (typed DAG)
    |
    +--> validate    NGS structural rules + failure-mode taxonomy
    +--> analyze     graph metrics, motifs, TSI/JSD, structure<->correctness
    +--> prune       redundancy regions -> pruned graph (DAG-preserving)
    +--> classify    per-step error probability (GradientBoosting)
    +--> benchmark   GCP · NGS pass rate · TSI · authorship fingerprint

Related Work

A capability-by-capability matrix is maintained in docs/comparison.md.

Reproducible Experiments

python -m experiments.run extract     --dataset gsm8k --max-traces 50
python -m experiments.run correlation --synthetic

Every run lands in rtsa/experiments/results/runs/<command>_<timestamp>/ with a manifest.json recording git commit, Python version, arguments, and UTC timestamp. See the full CLI for all subcommands.

Results

Selected numbers from the built-in validation and real-data runs (reproducible via the commands above):

Documentation

• docs/api.md — public API reference and module layout
• docs/comparison.md — capability matrix vs. LLM-MindMap / CRV / CoT2Graph
• docs/failure_modes.md — NGS failure-mode taxonomy
• rtsa/experiments/notebooks/end_to_end.ipynb — end-to-end walkthrough
• CHANGELOG.md — version history (Keep a Changelog)

Tests

python -m pytest tests/ -q

Citation

@software{rtsa2026,
  title={RTSA: Reasoning Trace Structure Analysis Toolkit},
  author={Fengrru},
  year={2026},
  url={https://github.com/Fengrru/rtsa}
}

Contributing & License

• Contributing: CONTRIBUTING.md
• License: MIT