cot-coherence

Detect silent incoherence in AI chain-of-thought reasoning.

Most evaluation tools check if each reasoning step is correct. cot-coherence checks if the logical progression between steps holds together — schema-level coherence, not step-level accuracy.

Why?

Chain-of-thought reasoning in LLMs silently degrades. Recent research (arXiv, Feb-Mar 2026) shows:

• CoT faithfulness decays at 70-85% of chain length ("Reasoning Horizon")
• Reasoning tokens have a negative effect past this horizon
• No existing tool detects this — current tools check RAG grounding, linguistic flow, or safety. None check logical coherence between steps.

cot-coherence fills this gap.

Install

pip install cot-coherence

With CLI support:

pip install cot-coherence[cli]

With LLM-powered detection:

pip install cot-coherence[llm]

Quick Start

import cot_coherence

report = cot_coherence.analyze("""
Step 1: The user asks about Python performance.
Step 2: Python is interpreted, so it's generally slower than compiled languages.
Step 3: Let me discuss the history of JavaScript frameworks.
Step 4: Therefore, Python is definitely the fastest language available.
""", original_question="Is Python fast?")

print(report.overall_score)  # 0.43
print(report.is_coherent)    # False
print(len(report.flags))     # 3+ (scope_creep, conclusion_drift, confidence_inflation)

What It Detects

5 Incoherence Patterns

Reasoning Horizon

Detects the point in a chain where quality starts to degrade — the "Reasoning Horizon" described in recent research. Reports the estimated horizon position and degradation signals.

CLI

# Analyze a trace file
cot-coherence check trace.txt -q "What is quantum computing?"

# Pipe from stdin
echo "Step 1: ..." | cot-coherence check

# JSON output
cot-coherence check trace.txt --json-output

# Adjust sensitivity (0.0=lenient, 1.0=strict)
cot-coherence check trace.txt -s 0.8

# Disable horizon analysis
cot-coherence check trace.txt --no-horizon

LLM Mode (v0.2+)

Rule-based detection is free and works offline. For higher accuracy, enable LLM-powered detection using Claude Haiku as a second-pass verifier:

export ANTHROPIC_API_KEY=your-key-here

# CLI
cot-coherence check trace.txt -q "your question" --use-llm

# Python
from cot_coherence import analyze, CoherenceConfig

config = CoherenceConfig(use_llm=True)
report = analyze(trace_text, original_question="...", config=config)

The LLM:

• Validates rule-based flags (reduces false positives)
• Discovers issues the rules missed (reduces false negatives)
• Enriches flag summaries with better explanations

Cost: ~$0.001 per trace (single API call). If the API fails, rule-based results are returned automatically.

Configuration

from cot_coherence import analyze, CoherenceConfig, IncoherenceType

config = CoherenceConfig(
    sensitivity=0.7,                    # 0.0=lenient, 1.0=strict
    enabled_detectors={                 # Enable specific detectors
        IncoherenceType.PREMISE_ABANDONMENT,
        IncoherenceType.CONFIDENCE_INFLATION,
    },
    analyze_horizon=True,               # Enable horizon analysis
    weights={                           # Custom pattern weights
        IncoherenceType.PREMISE_ABANDONMENT: 2.0,
        IncoherenceType.CONFIDENCE_INFLATION: 1.5,
    },
)

report = analyze("Step 1: ...", config=config)

Trace Formats

Auto-detects 4 formats:

# Numbered (default for most LLMs)
"Step 1: First\nStep 2: Second"
"1. First\n2. Second"

# XML (Claude-style thinking)
"<step>First</step><step>Second</step>"
"<thinking>Reasoning here</thinking>"

# Markdown
"## Step 1\nFirst\n## Step 2\nSecond"

# Newline-separated (fallback)
"First block\n\nSecond block"

Or pass pre-split steps:

report = analyze(steps=["First step", "Second step"])

Custom Parsers

from cot_coherence import register_parser
from cot_coherence.models import ReasoningStep

def my_parser(text):
    parts = text.split("|||")
    return [ReasoningStep(index=i, text=p.strip(), raw_text=p)
            for i, p in enumerate(parts) if p.strip()]

register_parser("pipe", my_parser)
report = analyze("First ||| Second ||| Third", trace_format="pipe")

The CoherenceReport

report = analyze(trace)

report.overall_score    # float 0.0-1.0
report.is_coherent      # True if score >= 0.7
report.steps            # list[ReasoningStep]
report.flags            # list[IncoherenceFlag]
report.critical_flags   # flags with CRITICAL severity
report.horizon          # HorizonAnalysis or None
report.pattern_scores   # dict[IncoherenceType, float]

# Each flag contains:
flag.type               # IncoherenceType enum
flag.severity           # LOW, MEDIUM, HIGH, CRITICAL
flag.confidence         # 0.0-1.0
flag.step_range         # (start_step, end_step)
flag.summary            # Human-readable description
flag.evidence           # Specific evidence
flag.suggestion         # How to fix it

How It Works

v0.1 uses rule-based heuristics — zero API cost, works offline:

• Premise Abandonment: Extracts premise markers ("given", "assuming"), checks if key entities appear in subsequent steps
• Conclusion Drift: Identifies conclusion markers ("therefore", "thus"), compares topic overlap via Jaccard similarity
• Confidence Inflation: Tracks hedging vs. certainty word ratios, flags unjustified jumps without evidence markers
• Scope Creep: Measures content-word overlap between each step and the original question
• Circular Return: Computes content-word fingerprints, flags high similarity between non-adjacent steps

Scoring: each pattern starts at 1.0, penalties applied per flag based on severity and confidence. Overall score is the weighted average.

Dependencies

Required: pydantic>=2.0

Optional:

• [cli] — click, rich (for terminal interface)
• [llm] — anthropic (for future LLM-powered detection)
• [dev] — pytest, pytest-cov, ruff, mypy

License

Apache 2.0