contextscope 0.1.0

Attention-level diagnostics for LLM context windows.

contextscope

Attention-level diagnostics for LLM context windows.

Long-context models degrade silently past ~100K tokens. Langfuse, Galileo, and Braintrust track output-level metrics — cost, latency, quality scores. None expose which parts of the context the model actually attended to. contextscope fills that gap.

import contextscope
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct", attn_implementation="eager"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

probe = contextscope.wrap(model, tokenizer=tokenizer)
result = probe.generate(messages=[{"role": "user", "content": "..."}])

print(result.diagnostics.attention_by_segment)
# {"system": 0.03, "early": 0.08, "middle": 0.42, "recent": 0.47}

What it detects

Three context-degradation patterns, each with a score from 0 (healthy) to 1 (degraded):

Pattern	What it means
lost_system_prompt	System prompt receives less attention than its uniform share
recency_bias	Last 10% of context absorbs disproportionate attention
ignored_early_context	First 10% of context receives near-zero attention

Install

# Core — zero runtime dependencies
pip install contextscope

# HuggingFace adapter (real attention probing)
pip install contextscope[hf]

# OpenAI adapter (logprobs proxy)
pip install contextscope[openai]

# Anthropic adapter (perturbation probe)
pip install contextscope[anthropic]

# All providers
pip install contextscope[all]

Three tiers, one interface

Tier 1 — HuggingFace (real attention)

Full mechanistic probing via output_attentions=True. The model must be loaded with attn_implementation="eager" — SDPA and Flash Attention backends silently return None for attention weights. contextscope raises immediately if a non-eager model is passed rather than silently producing empty diagnostics.

from transformers import AutoModelForCausalLM, AutoTokenizer
import contextscope

model = AutoModelForCausalLM.from_pretrained("...", attn_implementation="eager")
tokenizer = AutoTokenizer.from_pretrained("...")

probe = contextscope.wrap(model, tokenizer=tokenizer)
result = probe.generate(messages=[...], max_new_tokens=200)
print(result.diagnostics)

Power-user knobs for attention aggregation:

# Use only last 4 layers and first 8 heads (e.g. for retrieval-focused analysis)
result = probe.generate(messages=[...], layers=[-4, -3, -2, -1], heads=list(range(8)))

Tier 2 — OpenAI (logprobs proxy)

Real attention is not available for closed models. contextscope computes Shannon entropy per output token from top_logprobs and returns entropy-based heuristics for recency bias and ignored early context.

Limitations: lost_system_prompt is always None (not detectable from output-side logprobs). Degradation scores for the other two patterns are entropy heuristics — always severity="info". For definitive attention-level analysis, use the HF adapter.

from openai import OpenAI
import contextscope

client = contextscope.wrap(OpenAI())
response = client.chat.completions.create(model="gpt-4o-mini", messages=[...])
print(response.contextscope)  # Diagnostics attached to the response

Tier 3 — Anthropic (perturbation probe, opt-in)

Anthropic exposes no logprobs or attention weights. contextscope re-runs the call with three modified contexts and measures output divergence.

Cost: 3 extra API calls per diagnosis (~$0.10–0.30 at Sonnet rates, 10–30s extra latency). Opt-in via perturbation_probe=True.

import anthropic, contextscope

client = contextscope.wrap(anthropic.Anthropic(), perturbation_probe=True)
response = client.messages.create(model="claude-sonnet-4-6", messages=[...])
print(response.contextscope)

Without perturbation_probe=True, the adapter returns minimal provenance diagnostics with all scores at 0.

Diagnostics schema

@dataclass(frozen=True)
class Diagnostics:
    lost_system_prompt: float | None   # None if no system prompt or not detectable
    recency_bias: float                # 0-1
    ignored_early_context: float       # 0-1

    attention_by_segment: dict[str, float] | None  # HF only
    output_entropy: list[float] | None              # HF + OpenAI

    warnings: list[Warning]            # structured, filterable by .code and .severity
    provider: str                      # "huggingface" | "openai" | "anthropic"
    method: str                        # "attention" | "logprobs" | "perturbation"
    context_token_count: int
    model_name: str

@dataclass(frozen=True)
class Warning:
    code: str      # "LOST_SYSTEM_PROMPT" | "RECENCY_BIAS" | "IGNORED_EARLY_CONTEXT"
    severity: str  # "info" | "warn" | "high"
    message: str
    evidence: dict[str, float]

Thresholds

Warning thresholds are named constants in contextscope.config.Thresholds. Defaults are calibration-in-progress — override them per-call:

from contextscope.config import Thresholds

thresholds = Thresholds(
    lost_system_prompt_warn=0.3,
    lost_system_prompt_high=0.7,
)
probe = contextscope.wrap(model, tokenizer=tokenizer, thresholds=thresholds)

Development

git clone https://github.com/luoojason/contextscope
cd contextscope
pip install -e ".[all,dev]"
pytest -m "not slow"      # fast tests (mocked, no network)
pytest -m slow            # slow tests (downloads tiny HF model)
ruff check src/ tests/

Status

v0.1.0 — calibration in progress. Threshold defaults will be updated as benchmark data accumulates (planned: MMLongBench-Doc suite).

Roadmap:

• Calibrated thresholds from benchmark suite
• CLI (contextscope diagnose)
• PyPI publish after TestPyPI dry-run