tsfm-lens

Layered, architecture-agnostic comparison of time-series foundation model internals. Built to answer, for a pair like TimesFM (decoder-only, 32-step patch tokens) and Chronos (T5 encoder-decoder, per-timestep quantized tokens): where do these models compute similar things, where do they differ, on which kinds of data — and which of those findings survive statistical scrutiny and a held-out benchmark?

It consumes labeled benchmarks produced by the companion tsfm_benchmark package (sealed corpora with per-sample provenance and a public/private split) and produces one self-contained interactive HTML report.

The layered method

No single measurement supports the claim "these models share structure." Each level asks a stronger question and exists because of the previous level's limitation:

Level	Question	Method	Limitation it inherits
L0	Who is better, where?	MASE / sMAPE / pinball per family, paired bootstrap tests, Holm-corrected	Behavioral only
Screen	Which of this model's own layers are worth further analysis?	Residual-trajectory work+bend geometry (default), or coverage/factor-emergence	Cheap proxy for interestingness, not interestingness itself
Profile	What is in each model?	Effective dimensionality, family-probe decodability, CKA-to-input per layer	Per-model, descriptive
Lens	Where in depth does the forecast form?	Skip lens (layer-l states patched into the final block, decoded with the model's own head) + tuned ridge readout; crystallization depth	Depth-resolved, not component-resolved
L1	Do representations share geometry?	Linear CKA (global + per-family) with series-bootstrap CIs, RSA	Correlational
L2	Is the shared geometry linearly translatable beyond the input?	Ridge stitching vs an input-feature baseline; CI on the gain	Still not causal
L3	Where is structure causally carried?	Corruption sensitivity fingerprints + within-model activation patching (per layer and per window)	Within-model causality, compared across models
Attention	Which heads look where, and which matter?	Lag-profile head taxonomy, periodicity heads (induction-head analog), head/MLP mean-ablation ΔMASE, first-step decoder cross-attention	Pattern support varies by architecture
L4	How does each model organize the data?	Activation clustering with approximate labels, AMI across models	Descriptive
Exemplars	What does the difference look like?	Per-family case studies: forecasts, per-series lens curves, window attention maps	Illustrative, not statistical
Confirm	Which dev findings are real?	One-shot re-test of dev hypotheses on a sealed private corpus	The gold standard

Three design points are load-bearing:

• Time alignment. Token states are pooled onto shared fixed-width time windows (default 32 steps) via overlap-weighted pooling matrices built from each adapter's declared token_time_spans. Everything cross-model operates on [series, window, dim] tensors, so patch-based and per-step tokenizations become comparable.
• The stitching baseline. Both models saw the same input, so "layer A predicts layer B" is partly trivial. A hand-crafted input-feature probe (raw window, FFT magnitudes, summary stats) predicts the same targets, and only the gain above it is reported as evidence of shared learned structure.
• Exploration vs confirmation. Everything computed on the dev corpus is exploratory — many comparisons are looked at, so significant results are hypotheses. The confirm stage tests them exactly once on a sealed private corpus and reports verdicts; that section of the report is the evidence, the rest is context.

Stage pipeline and gates

pipeline.stage_names() is the authority for both the exact stage list and the order they run in:

corpus → extract → budget → frontend → layer_screen → l0 → internals → lens →
l1 → l2 → l3 → attention → cluster → sae → concepts → exemplars → register →
confirm → report

One line per stage (the detailed question/method/limitation for each is in "What each stage tells you" below, generated from the same source the report itself reads):

Stage	One-line job
corpus	Render the benchmark's own trust ladder (audit, diversity) — never recomputes it
extract	Capture the residual stream to a zarr store; run the alignment gate
budget	Params, measured FLOPs, latency, VRAM, captured-FLOP coverage
frontend	Tokenizer behavior before any layer: quantization, scale equivariance, NaN handling
layer_screen	Cheap per-model proxy for which layers deserve expensive analysis
l0	MASE/sMAPE/pinball per family, paired bootstrap, Holm-corrected — behavioral
internals	Per-layer effective dimension, family-probe decodability, CKA-to-input
lens	Skip lens + tuned ridge lens; where the forecast crystallizes in depth
l1	Linear CKA + RSA, every layer pair, every model pair — geometric
l2	Ridge stitching gain over an input-feature baseline — linear-translatable
l3	Corruption sensitivity fingerprints + within-model activation patching — causal within-model
attention	Lag profiles, periodicity heads, head/MLP ablation
cluster	Activation clustering, AMI across models — descriptive
sae	TopK SAE per target: fidelity, dead rate, forecast preservation, ground-truth alignment
concepts	Ablation battery → concept clustering → cross-model transfer → deterministic descriptions
exemplars	Per-family case studies from the MASE-gap distribution — illustrative
register / confirm	Freeze dev hypotheses; test once on the sealed private corpus
report	One self-contained HTML

Gates decide what actually runs, and each writes its own JSON so the report can name it as the skip reason:

• Tier gate (tiers.json) — a stage needs a minimum ModelAdapter tier (0 black-box … 3 decomposable); the tier is derived from which methods a subclass overrides, never hand-declared.
• Run-shape gate (shapes.json) — solo / pair / panel, derived from len(cfg.models). A solo run drops every comparison-only stage (l1, l2, cluster, exemplars) with a stated reason rather than erroring.
• Routing gate (routing.json) — a model whose measured impulse response is not time-localized (NotTimeLocalized) routes to l0/budget/report only.

Stages self-skip when their artifacts already exist on disk, unless a per-stage config fingerprint (manifest.py) shows the config changed since those artifacts were written — a stale skip is refused, not silently honored, and --allow-stale is required to proceed anyway.

Run folder layout

Every run writes to <run.out_dir>/<run.name>/ (runs/smoke/ for the smoke config). One subdirectory per stage that ran, named after the stage:

runs/<name>/
├── activations.zarr/        # extract: the chunked zarr v2 activation store
├── corpus/card.json         # corpus: the trust-ladder summary
├── budget/model_budget.json
├── l0/{summary.json,metrics.parquet,noise_floor.json}
├── internals/, lens/, l1/, l2/, l3/, attention/, cluster/
├── sae/<model>/<layer>_ablation.json, concepts.json, transfer.json, concept_stage.json
├── exemplars/
├── hypotheses.json           # register
├── confirm/confirmation.json
├── fairness/card.json        # every measured asymmetry between the models in this run
├── report.html                # the one self-contained deliverable
└── report/{findings.json,coverage.json,results.csv,results.parquet,model_similarity.json,...}

report/coverage.json is the ground truth for what actually rendered, skipped, or failed in a given run — read it (or the report's own "Run coverage" section) before trusting that a section's absence means anything other than what its stated reason says.

What each stage tells you

Four fixed questions per stage (ROADMAP.md sec 21 J2), generated from tsfm_lens/stage_docs.py by render_stage_docs.py -- edit that file, not this section, and re-run the script to update it.

corpus

Question. Can the benchmark corpus itself be trusted -- is it free of copied reference series, free of near-duplicates, and diverse enough that a model can't do well here just by memorizing one shape?

How. Reads the sealed corpus's own manifest (ROADMAP.md sec 34 item B1's audit block, when present) and, optionally, an existing benchmark_validation report, and renders both as a fixed trust ladder of named claims -- never recomputes a leakage or diversity check itself, and never imports benchmark_validation.

Good vs. bad result. Good: every trust-ladder row reads measured with a real reference and a stated verdict. Bad: a row reads not_checked -- the corpus was built with --references none, so every candidate passed the leakage gate trivially, which is NOT the same as a corpus that was checked and found clean -- or not_recorded, meaning no audit ever ran at all.

What it cannot tell you. Two rows are always unresolvable and are rendered that way rather than omitted: whether the private split's distribution genuinely matches the dev split's (no mechanism in this repo compares them), and whether any model being compared was trained on data shaped like this corpus (no audit anywhere can inspect a checkpoint's own training data). A measured row is evidence the corpus is sound by the checks that exist -- it is not proof no other check would find a problem.

extract

Question. Has each model's internal computation actually been captured, and lined up correctly with real time, so every later stage has something trustworthy to work from?

How. Runs each model once over the benchmark series, hooks its internal layers to save their activations, and pools each model's own tokens onto a shared time-window axis so architectures that read one timestep per token and architectures that read dozens per token become directly comparable.

Good vs. bad result. Good: the alignment check's diagonal-hit fraction sits near 1.0 at every captured layer, and no non-finite (inf/NaN) activations are recorded. Bad: the alignment gate fails outright — the declared mapping between token position and true time is wrong for this checkpoint or library version — or a layer's stored activations contain non-finite values that would silently poison every downstream computation that reads it.

What it cannot tell you. This stage doesn't evaluate or compare anything — it only says whether each model's computation was captured and time-aligned correctly. A passing alignment check means the token-to-time mapping is trustworthy; it says nothing about whether the captured activations are interesting, whether the models are similar, or whether any later stage will find anything at all.

budget

Question. What does each model cost to run, and is its forecast quality bought with more compute?

How. Runs a handful of extra forward (and, optionally, full-forecast) passes through each model while measuring parameter counts, FLOPs, wall-clock latency, and peak GPU memory, then re-plots each model's L0 accuracy against those costs instead of looking at accuracy alone.

Good vs. bad result. Good: FLOPs are measured directly (not falling back to an unmeasured or 'suspicious_low' reading) and the model sits down-and-to-the-left on the compute-normalized chart — lower error for less compute. Bad: a flops_sanity: suspicious_low verdict (the measurement can't see a fused or custom kernel and is undercounting compute), or the L0 stage didn't run, so cost can't be normalized by quality at all.

What it cannot tell you. Cost is measured only at this run's own context length, batch, and horizon — it does not generalize to a different sequence length, and FLOPs are not the same thing as latency (a model with fewer FLOPs can still be slower on real hardware). It also only tells you the price of a forward pass, never whether that price bought anything mechanistically interesting — that's what every later stage is for.

frontend

Question. What does each model do to its input BEFORE any layer runs -- how coarsely does it quantize the series, does scaling the input scale the forecast back out cleanly, how does accuracy degrade as recent context goes missing, and what happens when a context value is NaN or the whole context is constant?

How. Reads each re-quantizing tokenizer's own bin geometry to score how coarse one quantization step is relative to a series' own amplitude; calls predict() on the input scaled up/down by large factors and compares the rescaled-back forecast to the original; calls predict() with the most recent context progressively withheld (a staleness scenario, not a front-trim) to trace a degradation curve; injects NaN at the front, middle, and back of the context to see whether predict() errors, silently produces a non-finite forecast, or genuinely handles it; and feeds constant contexts (0, 1, 1e3, 1 plus 1e-7 noise) to check that the forecast and every captured activation stay finite.

Good vs. bad result. Good: a re-quantizing tokenizer's bin width is a small fraction of the series' own amplitude with little saturating clipping, near-zero scale-equivariance residual, graceful (not cliff-shaped) accuracy degradation as recent context is withheld, and a NaN verdict of 'handled' or an explicit, clean error rather than a silently corrupted forecast. Bad: heavy clipping or a large quantization step relative to signal amplitude, a scale-equivariance residual that grows with the scale factor, a sharp cliff at a specific context length, or a 'propagates' NaN verdict -- a non-finite input silently becomes a non-finite forecast with no error to flag it -- or a constant context that yields NaN activations or forecasts (real corpora contain constant windows).

What it cannot tell you. Every diagnostic here is about the FRONT DOOR only -- what happens before or around a forward pass, never what happens inside one (that's every other stage's job). Quantization resolution is meaningful only for a re-quantizing tokenizer and is reported as 'not applicable', never a fabricated zero, for a continuous-embedding architecture. NaN handling is checked at only a handful of hand-placed positions and one missing-fraction, not exhaustively; and none of these probes says anything about forecast quality on ordinary, well-formed input -- L0 is what answers that.

layer_screen

Question. Which of this model's own layers are worth spending the expensive stages (Lens, L1, L2, L3, attention, SAE) on?

How. Scores every layer a model actually captured with a cheap, architecture-agnostic proxy for 'interesting residual-stream activity' — by default, how much a layer's own trajectory through activation space bends and does work — and flags the highest scorers within a compute budget.

Good vs. bad result. Good: the selected layer(s) coincide with, or precede, the layers where L1/L2/L3/Lens later find the strongest real signal — the cheap proxy pointed at something genuine. Bad: a selection sitting at the very first or very last layer with no real signal there, or a run where the report has to state the method fell back to a uniform-stride null because its own selection failed.

What it cannot tell you. This is a cheap proxy for interestingness, not interestingness itself — it won a single-corpus, single-checkpoint-pair bake-off, not a validated rule across architectures. It also only screens the layers capture_layer_stride actually captured, so at a stride greater than 1 it cannot see, and cannot recommend, half the model's blocks; it answers 'which captured layer looks most active,' not 'which is the best layer in the whole model.'

l0

Question. Which model actually forecasts more accurately, and on what kinds of data?

How. Runs every model's native forecasting procedure on every benchmark series, scores each series' error against a naive same-scale baseline, and statistically compares the paired per-series differences within each data family, correcting for testing many families -- and, when more than two models are configured, many model pairs -- at once.

Good vs. bad result. Good: a family where the paired difference's confidence interval clearly excludes zero after correction — a real, replicable accuracy edge on that kind of data. Bad: every family's interval spans zero (no reliable difference survives correction), or a gap that looks real in the bar chart but isn't backed by a significant test.

What it cannot tell you. This is purely behavioral: it can tell you which model wins and where, but nothing about why, or what mechanism inside either model produces the gap — that's what every stage after this one exists to investigate. It also measures point-forecast accuracy only; a model can score well here and still have badly miscalibrated uncertainty (a separate calibration diagnostic covers that, but it is not this headline number). And it is the ONLY stage that compares more than two models: everything after it compares the designated pair only, so with three models the extra pairs are unexamined there rather than weakly evidenced.

internals

Question. What does each model's own internal representation look like at each depth, on its own terms?

How. For each layer of each model separately, measures how many effective dimensions its activations actually use, how close its representation still is to simple statistics of the raw input, and how well a simple classifier can read the data's family straight out of it.

Good vs. bad result. Good: a clear expand-then-compress effective-dimensionality curve, a family-decodability curve that rises well above chance at some depth, and a representation that visibly moves away from raw-input statistics with depth. Bad: a flat effective-dimensionality curve or a family-probe accuracy that never clears chance — either means this per-model view has nothing to say about that model's depth behavior.

What it cannot tell you. Everything here is per-model and descriptive — it never directly compares models against each other (L1 exists for that). 'Decodable' does not mean 'used by the forecast': a layer can carry perfect family information the model itself never reads out. It's also limited to the window-pooled representation, so structure that lives only within a window and not across windows is invisible to it.

lens

Question. At what depth does each model's forecast actually take its final shape?

How. Feeds each layer's own activations into the model's normal output pathway, as if that layer were the last one, to see how good a forecast it already produces; separately, fits a small linear readout per layer to check whether the forecast is already recoverable there even if the model's own head can't decode it yet.

Good vs. bad result. Good: a curve that reaches close to final accuracy well before the last layer ('crystallizes early'), consistent with a similarly early rise in the tuned-lens readout. Bad: a flat, uninformative curve where the tuned lens also shows nothing until the very last layer — the forecast genuinely needs the model's full depth — or the two disagree sharply, itself informative about miscalibrated early layers.

What it cannot tell you. The direct (skip-lens) readout can be misleadingly pessimistic about early layers for reasons unrelated to information content — the model's final block was never trained to decode them — which is exactly why the tuned-lens readout exists alongside it. Even together, both only detect linearly recoverable structure; a forecast could be present in a layer in a form no linear readout can see.

l1

Question. Do the models represent this data similarly at all?

How. Compares every pair of layers, one from each model, by how similarly they organize the same set of series geometrically — a comparison that ignores each layer's own arbitrary rotation and scale — and checks the result against a shuffled-series control.

Good vs. bad result. Good: a peak similarity that sits clearly above the shuffled-series null, especially one that follows a sensible pattern by depth (early layers matching early layers). Bad: a similarity score that's high everywhere and barely rises above the null — a sign the number mostly reflects that the models see the same input, not that they've learned anything in common.

What it cannot tell you. This is correlational: a high score never establishes that the two models compute the same thing, only that their representations are geometrically similar — which two under-trained or too-simple networks can also produce for reasons that have nothing to do with shared learning. It says nothing about mechanism; L2 and L3 exist specifically because this stage can't answer that.

l2

Question. Can one model's layer actually be translated into the other's, beyond what's explained by both simply seeing the same input?

How. Fits a simple linear map from one model's layer activations to the other's, compares how well it predicts against a hand-built baseline that only ever sees the raw input (never the other model), and reports the gap between the two as the real evidence.

Good vs. bad result. Good: the best pair's gain over the input baseline has a confidence interval that clearly sits above zero — real evidence the source layer predicts the target layer better than raw input alone could. Bad: that interval includes zero — the apparent shared structure could just be an artifact of the models seeing the same series.

What it cannot tell you. Only the gain over the input baseline counts as evidence here — raw predictive accuracy alone is untrustworthy, since both models process the same input and would score well on that basis alone. This also only detects linear translatability (a genuine nonlinear correspondence between two layers can score zero gain), and it is still not a causal claim: a good stitch means the two layers' information is compatible, not that either model's forecast actually depends on it.

l3

Question. Where, causally, does each model actually carry a specific structural property of the data — and does damaging it there hurt the forecast?

How. Corrupts one structural property of the input at a time (trend, seasonality, added noise, and so on), measures how much each layer's activations shift because of it, then splices the clean, uncorrupted activations back into an otherwise-corrupted forward pass, one layer at a time, to see how much of the clean forecast that single splice restores.

Good vs. bad result. Good: a restoration curve that rises toward 1.0 at some depth for a given corruption — that layer causally carries the fix for that property — and two models whose sensitivity fingerprints peak at matching relative depths for the same corruption. Bad: a restoration curve that never clears a small fraction anywhere, or a corruption whose measured sensitivity is small only because it touches a tiny fraction of the input's timesteps in the first place, not because the model is robust to it.

What it cannot tell you. The activation-shift fingerprints are a magnitude-of-change measure by themselves, not a causal one — a layer can shift a lot without that shift ever affecting the final forecast, which is exactly why the patching curve exists alongside it. Even the patching curve is causal only within one model: this stage never transplants activations between models, so any cross-model comparison here is two separately-measured within-model curves side by side, never a joint causal test.

attention

Question. Which attention heads look where in time, and which of them (or which MLP blocks) actually matter to the forecast?

How. Averages each head's attention weight as a function of how far back in time it looks, flags heads whose attention concentrates near seasonal-period multiples, and separately re-runs the forecast with individual heads or MLP blocks zeroed out to see how much accuracy degrades.

Good vs. bad result. Good: a head with a clean, repeating attention stripe at a real seasonal lag (a periodicity head) whose removal via ablation measurably hurts accuracy on exactly that kind of data — pattern and causal importance agreeing. Bad: no head or MLP block's removal moves accuracy at all, or the architecture reports 'unsupported' because it exposes no attention weights to inspect.

What it cannot tell you. The lag-profile and periodicity-head numbers are unconditional averages over sampled series, so a head that's periodic only on some data families and flat on others can look unremarkable in the averaged view even though it's doing real, family-specific work. Pattern support (whether attention weights or ablation are even available) varies by architecture, so a missing or 'unsupported' result here is a capability gap, not evidence the model lacks that structure.

cluster

Question. How does each model organize the whole benchmark on its own terms, and do the models group the data the same way?

How. Projects each model's activations, at its own side of the strongest cross-model layer pair, down to a small number of dimensions, clusters them, labels each cluster by what kind of data dominates it, and compares how much each pair's groupings agree using a chance-corrected overlap score.

Good vs. bad result. Good: clean, well-separated clusters that line up with real, interpretable properties of the data, and a partition-agreement score meaningfully above chance between a pair of models. Bad: a single smeared blob with no real separation, or cluster labels that don't correspond to anything a human would recognize as a coherent group.

What it cannot tell you. The 2D map is a visualization only — apparent distances or gaps between clusters on the plot are not quantitative and should never be read as a diversity or separation measure. Cluster labels are approximate (majority family plus salient statistics), not ground truth, and everything here is descriptive: it says how each model happens to partition the data, not why, and not whether that partition is causally meaningful to either model's forecast.

sae

Question. Can each model's layer be decomposed into a small number of individually interpretable features, do those features actually matter to the forecast, and do the two models' features play the same role?

How. Trains a sparse dictionary that reconstructs a layer's activations from only a handful of active 'feature' directions at a time, then checks reconstruction quality, how many learned features ever fire at all, whether swapping in the reconstruction changes the forecast, and whether any feature's activation tracks a known ground-truth property of the data (like a trend order or a seasonal period). A causal channel battery then patches each alive feature's direction into a clean forward pass and measures a real, per-channel response (not just correlation with a label), scored against a random-direction null. Features are clustered into named 'roles' by their shared response and structural signature, and each model's roles are matched against the other model's by cosine similarity of their null-normalized response fingerprints, checked against an untrained-twin floor.

Good vs. bad result. Good: high reconstruction fidelity, a low dead-feature rate, forecast-preservation deltas that stay within the SAE's own seed-to-seed noise floor, and features whose activation clearly tracks a real ground-truth field well above what random label shuffling would produce by chance. For the causal layer: a channel response that clears the random-direction null, and a cross-model role match rate that clears its own untrained-twin floor (not just zero). Bad: the large majority of features never firing at all (a common failure mode), a forecast-preservation delta far outside the noise floor, a ground-truth alignment score that isn't meaningfully above its own permutation-null control, or a role match rate that looks large but sits below the untrained-twin floor — architecture match alone can produce that, not shared learned structure.

What it cannot tell you. A feature firing on a particular kind of series is illustrative correlation unless the causal channel battery has confirmed a real patched response for that feature — check the channel result, not just the ground-truth alignment score, before reading any feature as meaningful. A cross-model role match is a geometric correspondence in response space, not evidence the two models use that role the same way causally — read the match rate against its own untrained-twin floor before treating any match as evidence of a shared feature; a rate that does not clear that floor is not an established claim for that pair, and this is a per-run measurement, not a fixed property of the method. Only the small alive fraction of the dictionary (often under 10%) can ever show up as an example, and the headline alignment number is inflated by searching many candidate ground-truth fields per feature, so it must always be read next to its permutation-null control, never on its own.

concepts

Question. Which features does each model's dictionary actually use when it forecasts, do they group into named concepts, and does another model group the same series the same way?

How. For every trained dictionary, zeroes each candidate feature out of the model's own reconstruction on the series where that feature fires hardest, and measures nine properties of the forecast against a random-direction null. Features that move something are clustered by what they move into concepts (or the dictionary is declared non-modular when no clean grouping exists). Each concept's top series are then checked in every other model: does some feature there separate the same series better than random series of the same kind would, and does that feature's own top series come back the other way? Finally every concept is described in a sentence composed only from what was measured. Separately, every causal feature across every model is also pooled into one shared space and clustered directly against each other, so a group there can span several models even when a dictionary's own per-target clustering above found no clean grouping within it.

Good vs. bad result. Good: features whose removal moves the forecast well above the null, concepts with several members each, and transfers that clear the matched null in both directions. Bad: most targets declared non-modular (the dictionary's causal features do not group), or transfer rates that are high only because the matched null is weak for that kind of series.

What it cannot tell you. A transfer that clears its null says another model separates the same series, not that it uses the feature the same way causally. The rates are uncorrected for the many tests made, and nothing yet says how often a concept would transfer to a second dictionary trained on the SAME model, so a rate cannot be read as high or low on its own. Everything here is measured on the development corpus only.

exemplars

Question. What does an aggregate difference between models actually look like on one real series?

How. Picks a handful of concrete series per data family — specifically ones where the models' forecasts disagree the most — and shows each one's context, true continuation, both forecasts, per-layer lens curves, and pooled attention pattern side by side.

Good vs. bad result. Good: a case that makes an aggregate statistic from an earlier section tangible — you can see exactly where one model's forecast goes wrong on a concrete series. Bad: a case that reads as an ordinary, unremarkable series despite being flagged as the most divergent — worth treating skeptically rather than trusting the selection blindly.

What it cannot tell you. These series are deliberately chosen for maximal disagreement, not for being typical or representative — this stage cannot tell you how often such a disagreement happens, only that it can happen and roughly what it looks like when it does. It is illustrative evidence only, the same evidence class as a hand-picked example in a paper, not a statistical claim.

register

Question. Which of the (many) exploratory findings from the dev corpus are worth spending the one-shot private confirmation on?

How. Reads the dev-corpus findings this run has already produced and freezes a small, explicit list of testable hypotheses — plus a content hash of exactly which dev artifact each one came from — before the private (held-out) corpus is ever touched.

Good vs. bad result. Good: a registry whose hypotheses map cleanly onto real dev findings with clear pass/fail criteria. Bad: a registry that can't be built at all because a dev artifact it depends on doesn't exist — which correctly blocks confirmation rather than letting it run against nothing.

What it cannot tell you. This stage provides no evidence itself about which model is better or how they compare — it only decides, once, which questions get to be asked of the private data. It cannot rescue a finding that wasn't registered here; anything not pre-registered cannot later be confirmed, by design.

confirm

Question. Which of the exploratory dev-corpus findings are actually real, rather than an artifact of looking at many comparisons?

How. Re-runs exactly the pre-registered hypotheses, exactly once, on a sealed corpus that no exploratory analysis in this run has ever seen, and reports which ones hold up.

Good vs. bad result. Good: a hypothesis that replicates on the private corpus with the same direction and a confidence interval excluding zero after correction — the strongest evidence this pipeline can produce. Bad: a dev finding that fails to replicate (a real and expected outcome some of the time, since dev findings are exploratory by construction), or a hypothesis marked 'untestable' because this stage doesn't yet re-test that kind of claim on private data.

What it cannot tell you. This section is the actual evidence, but only for the specific hypotheses that were pre-registered — it says nothing about any exploratory finding that wasn't registered. It also cannot be re-run: if the private corpus is ever peeked at, this evidence is spent and a fresh held-out corpus is required. A 'not confirmed' verdict does not mean the dev finding was wrong, only that it didn't survive being tested exactly once, out of sample.

report

Question. Given everything the other stages found, is this a fair comparison, and what's actually solid enough to act on?

How. Collects every stage's artifacts into one document, states upfront which measured asymmetries between each pair of models (parameters, compute, coverage, forecast determinism) apply before any result is read, and marks every section as rendered, skipped, or failed rather than silently omitting anything.

Good vs. bad result. Good: every enabled stage's section actually rendered (no 'failed' rows in the coverage panel) and the fairness card names its asymmetries with real measured values rather than 'not yet measured'. Bad: any section marked 'failed' (a bug, not an expected degrade) or a fairness row that couldn't be measured, which should downgrade how much weight a depth-located claim in that section deserves.

What it cannot tell you. The report presents what the pipeline measured; it adds no new evidence of its own. A 'rendered' status only means a section produced output without error, not that a result inside it is strong — that's for the evidence-class ladder shown once at the top (geometric / translatable / causal-within-model / descriptive / illustrative / behavioral) to decide, section by section.

Glossary

The recurring vocabulary of this repo's report, one sentence each (ROADMAP.md sec 21 J3), generated from tsfm_lens/glossary.py by render_glossary.py -- edit that file, not this section, and re-run the script to update it. The report renders the same entries under its "How to read this report" preamble.

Activation patching. Writing a model's own cached clean activations back into its corrupted forward pass, one layer (and optionally one window) at a time, so that recovered forecast quality localizes where a property is causally carried; always within one model, never transplanted between models.

Where it appears: Perturbation & patching's restoration curves and layer × window heatmaps.

AMI (adjusted mutual information). How much two clusterings of the same series agree, corrected for the agreement expected by chance, used to ask whether two models carve the benchmark into similar groups without requiring their cluster labels to correspond.

Where it appears: Activation clusters.

Captured FLOP fraction (coverage). The share of a model's real per-forecast computation that the captured layers actually perform, measured rather than assumed — an encoder-only capture surface or a strided capture leaves the rest unobserved, and any depth-located claim about a model below 90% carries that qualifier automatically.

Where it appears: The fairness card; Cost and capacity; auto-appended to depth-located findings.

catch22 (feature space). The 22 (or 24, adding mean/std) general-purpose time-series features benchmark_validation computes per series and robustly scales (with winsorization against outlier swamping) before computing diversity, effective dimensionality, or near-collisions on them — the space every diversity number in this report is measured in, never the UMAP coordinates used only for the companion plot.

Where it appears: The corpus card's diversity and feature-space-map figures.

Concept. A group of SAE features from one model's dictionary whose removal moves the forecast in the same way (the same channels, the same direction), found by clustering each feature's ablation fingerprint and admitted only when every group has at least a minimum number of members, so a dictionary whose causal features do not group is reported as non-modular rather than forced into concepts.

Where it appears: The SAE section's concept cards; produced by the concepts stage.

Concept atlas. A grouping of causal SAE features pooled across every model and clustered by ablation-fingerprint similarity, so that every pair inside a group clears a minimum cosine, groups below the minimum size stay unassigned, and a group can span several models (unlike a plain 'concept', which stays within one model's dictionary).

Where it appears: The SAE section's concept atlas figure and table; produced by the concepts stage.

Convergent concept. An atlas concept whose member models share the same causal EFFECT on the forecast but fire on different, unrelated INPUTS per model — a measured finding about this run's own data, not a weaker or failed version of a fully shared concept.

Where it appears: Model comparison — sharing-class column and Q1's answer box.

Corruption sensitivity fingerprint. A layers × corruptions matrix of how much each layer's activations move when a specific structural property is destroyed in the input, forming a per-model signature whose shape across depth is what gets compared across models — never the raw magnitudes, which are not calibrated between corruptions.

Where it appears: Perturbation & patching's sensitivity heatmap and cross-model agreement.

Crystallization depth. The shallowest relative depth at which the skip-lens forecast is already within a set tolerance (10% by default) of the model's final forecast quality — where the answer is essentially settled, not where computation stops.

Where it appears: Forecast lens.

Dead feature. A dictionary atom in a trained sparse autoencoder that never activates on any evaluation row, so it contributes nothing and inflates any statistic computed over "all features"; the dead rate is reported because a mostly-dead dictionary can otherwise look like a healthy one.

Where it appears: Sparse features (SAE) — dictionary health, next to reconstruction fidelity.

Depth axis (index / block / compute). Which definition of "how deep" a figure uses: index counts only the layers this run captured, block (the default) counts every block in the whole stack including surfaces never captured, and compute uses cumulative measured FLOPs; an encoder-only capture surface therefore caps well below 1.0 on block rather than misleadingly reaching it.

Where it appears: The fairness card's "Depth axis" row, and every depth-plotted figure's own note.

Diagonal-hit fraction (alignment). The share of probe windows where injecting an impulse at window w perturbs aligned window w most, verifying empirically that the token-to-time mapping an adapter declares is true for the installed library version; below 1.0 at late layers is normal, near zero anywhere is a broken mapping.

Where it appears: The Alignment & Provenance panel; run by a gate at extraction time.

Effective dimensionality. The participation ratio of a representation's PCA spectrum — roughly, how many directions actually carry variance — used as a descriptive profile of where a model expands or compresses, not as a quality score.

Where it appears: Model internals' per-layer depth profile.

Epoch (corpus). A fresh, non-overlapping seed range used to mint a brand-new private split when an existing one is suspected of having leaked, so a peeked-at private corpus is regenerated rather than reused.

Where it appears: The corpus card's header line; private benchmark confirmation.

Evidence class. The rung a claim sits on — geometric, linearly-translatable, causal-within-model, descriptive, illustrative, or confirmatory — stated for every finding so that a confident-looking chart cannot be read as stronger evidence than its method supports.

Where it appears: "How to read this report", and the caveat under every finding.

Evidence ladder. The six rungs a cross-model concept claim climbs, from weakest to strongest: same forecast effect, same driving inputs, reproducible across an independent SAE seed, other models' own dictionaries select the same inputs, the same causal effect measured on the same inputs, and confirmed on a sealed private corpus — a concept's verdict names the highest rung it actually reaches, and a rung nothing in this run measures renders 'not measured', never a failed rung.

Where it appears: Model comparison — the verdict table; produced by report/derived.py::concept_verdicts.

Family and archetype. Two levels of benchmark grouping: the family is which generator produced a series (parametric, random_parametric, mixture, …) and the archetype is the structural recipe drawn within it (seasonal_dominant, intermittent_bursts, …); per-family statistics are the unit multiple-comparison correction is applied across.

Where it appears: Behavioral profile, Representational geometry's per-family CKA, Activation clusters.

Far horizon. Shape of the late part of the forecast -- its furthest-out steps -- a higher value changes the long-range forecast's shape by a larger amount.

Where it appears: Sparse features (SAE) -- the causal channel battery's evidence columns, and every channel name rendered from it.

Flatness. How close the forecast is to a constant line -- a higher value makes the forecast closer to a flat, constant line.

Where it appears: Sparse features (SAE) -- the causal channel battery's evidence columns, and every channel name rendered from it.

Frequency balance. Centre of mass of the forecast's frequency spectrum -- low means the forecast is dominated by slow, smooth movement, high means by fast wiggles -- a higher value shifts the forecast toward higher-frequency, choppier movement.

Where it appears: Sparse features (SAE) -- the causal channel battery's evidence columns, and every channel name rendered from it.

Ground-truth feature alignment. The correlation between a learned sparse feature's activation and a known generator parameter (a seasonal period, a trend order, an anomaly flag), which is this repo's substitute for the subjective max-activating-example judgment usual in SAE work; because each feature is matched to its best of many candidate fields, it is read against a label-permutation null.

Where it appears: Sparse features (SAE).

Holm correction. The multiple-comparison adjustment applied across families so that testing several groups at once does not manufacture a significant result; a claimed strength needs both a corrected p below alpha and a confidence interval excluding zero.

Where it appears: Behavioral profile; Private benchmark confirmation.

Input-feature baseline. A deliberately simple probe (raw window values, FFT magnitudes, summary statistics) predicting the same targets the stitching map does, whose whole job is to absorb the part of the agreement that follows from shared input rather than shared learned structure.

Where it appears: Stitching probes; the quantity every reported gain is measured against.

Leakage (instance-level vs. distributional). Two different failures a benchmark can have: instance-level leakage is the exact series having been seen in a model's training data, which the leakage gate audits directly against real reference series; distributional leakage is only that kind of data having been seen, which is carried by construction for every real_derived-tier sample and cannot be audited by this or any other check.

Where it appears: The corpus card's leakage-gate figure and trust ladder.

Level. Mean value of the forecast -- its overall height -- a higher value raises the whole forecast.

Where it appears: Sparse features (SAE) -- the causal channel battery's evidence columns, and every channel name rendered from it.

Linear CKA. A 0–1 similarity between two models' representations of the same series at a chosen layer pair, computed on the shared window axis and bootstrapped with the series as the resampling unit; it is correlational, and both models seeing the same input inflates it on its own.

Where it appears: Representational geometry (the layer-pair heatmap and its depth curve).

MASE. Mean absolute error divided by the error of a naive same-scale baseline on the same series, so that 1.0 means "no better than naive" and series of wildly different amplitudes can be averaged together; series whose scale denominator is too small to be reliable are excluded and counted, not silently kept.

Where it appears: Behavioral profile (the primary forecast-quality metric), and every ΔMASE elsewhere.

Mean ablation (head / MLP). Replacing one attention head's or MLP's output with its average over the batch and measuring the forecast damage, which ranks components by how load-bearing they are without needing to know what they compute.

Where it appears: Attention analysis' head and MLP ΔMASE rankings.

Minimal sufficient set. The smallest set of a model's own attention heads found by a greedy search whose patching restores most of a corruption's damage — sufficient because patching it works, but never proven the unique or provably smallest such set, since a greedy search only ever finds a small set, not the smallest.

Where it appears: The seasonality circuit.

Near horizon. Shape of the first part of the forecast -- the steps just after the context ends -- a higher value changes the near-term forecast's shape by a larger amount.

Where it appears: Sparse features (SAE) -- the causal channel battery's evidence columns, and every channel name rendered from it.

Near-collision fraction. In the catch22 feature space — never on UMAP coordinates — the fraction of series whose nearest neighbor sits closer than a fixed threshold; a distinct diversity signal from the shape-matching redundancy fraction, since the two are computed on different representations and never merged until the report.

Where it appears: The corpus card's diversity figure.

Necessity. Whether ablating a candidate head set on an otherwise-clean forecast damages it toward the fully-corrupted level — the complement of sufficiency, and required alongside it because either alone is a weaker claim than both together.

Where it appears: The seasonality circuit.

Noise floor. How much a model's own metric moves between two identical repeat runs (nonzero for a sampled decoder, zero for a deterministic one), measured so that a delta can be reported as a multiple of it rather than against an implied zero — a ΔMASE under about 2× its own floor is not interpretable.

Where it appears: The fairness card's determinism row; every ΔMASE in ablation, patching and SAE sections.

Path patching. Decomposing one head's total causal effect into a direct part (measured with every other head in the found set frozen at its clean value) plus a part routed through each other head individually, then checking whether the parts sum back to the total — a check that can fail even for a correctly-implemented decomposition, if the heads genuinely interact nonlinearly rather than contributing independent, additive paths.

Where it appears: The seasonality circuit.

Periodicity head. An attention head that puts more mass than a uniform-baseline share at multiples of a series' dominant seasonal lag — the time-series analogue of an induction head, identified by lag profile rather than by what it is named.

Where it appears: Attention analysis' head taxonomy.

Probe decodability. The held-out accuracy of a simple linear classifier reading a property (family, seasonality, trend) straight off a layer's activations, which establishes that the information is linearly present there and nothing more — never that the model uses it.

Where it appears: Model internals; the SAE section's ground-truth alignment.

Provenance-driven. Said of a concept's part when what best explains which series it fires on is a fact about how the benchmark was BUILT (which tier or generator produced a series) rather than a structural property of the time series itself, so 'fires on sequential_par series' is read as a property of the corpus, not of time series in general.

Where it appears: Model comparison — every input-profile table; produced by sae/concept_profiles.py.

Random-direction null. The same causal channel battery re-run on directions drawn at random from the same activation space instead of a trained SAE feature's decoder direction, so a channel response is only quotable once it is compared against how large a response chance alone produces, not against zero.

Where it appears: SAE causal channel testing (Component A).

Reach. Whether a patched-in intervention (a feature direction, a layer's activations) actually changes anything the forecast head reads, checked before any effect is trusted — confirmed by two controls: patching a layer into itself must move the forecast by exactly 0.0, and patching an earlier layer into a later read must move it by something nonzero; a clean, flat, entirely plausible-looking curve can still mean the intervention never reached anywhere the model reads (CLAUDE.md sec 11.42).

Where it appears: SAE causal channel testing (Component A).

Redundancy fraction. The share of all pairwise series comparisons — by cross-correlation or banded DTW shape matching, after z-normalization — that fall below a near-duplicate distance threshold; a high value can be a true finding rather than a defect, e.g. many pure-seasonal series sharing one period and differing only in noise.

Where it appears: The corpus card's diversity figure, read alongside effective dimensionality.

Registered hypothesis. A dev-corpus finding written down before the private corpus is opened; only registered hypotheses can be confirmed, and a finding discovered afterwards cannot be rescued by the confirmation stage no matter how strong it looks.

Where it appears: Private benchmark confirmation; the "[exploratory]" prefix marks everything that is not one.

Relative depth. A layer's position rescaled to 0–1 so models with different layer counts share one axis — a convention that makes the comparison drawable, not a claim that the same fraction is the same computational stage in two architectures.

Where it appears: Model internals, Forecast lens, Representational geometry, Perturbation — any curve plotted against depth.

Response fingerprint. A feature's or role's vector of null-normalized responses across the causal channel battery, used as the primary signal for matching one model's roles against another's by cosine similarity — a match rate computed this way is never rendered without its own untrained-twin floor beside it, because two architecturally similar but never-jointly-trained models can already look substantially 'matched' by chance.

Where it appears: Cross-model role correspondence table (Component C).

Role. A cluster of a single model's SAE features that share a similar structural signature and causal response profile, named from its dominant response channel and its best-matching ground-truth field — a description of that model's own feature space, not a claim that any other model has an equivalent role until cross-model matching (see response fingerprint) has actually checked.

Where it appears: Superseded by concept clustering (ROADMAP.md sec 30, 2026-09-11) -- see the concept cards and concept universality table instead; the roles table/role cards this pointer named are no longer rendered.

RSA (representational similarity analysis). The rank-correlation companion to CKA: instead of comparing representations directly it compares each model's matrix of series-to-series distances, so it survives any monotone rescaling CKA would not.

Where it appears: Representational geometry, reported alongside CKA.

Sealed corpus (public dev / private test). A corpus written with per-sample hashes and a manifest, split into a public dev half used for all exploration and a private test half drawn from a disjoint seed range that is opened exactly once; the private half is a consumable, and a peeked one is replaced rather than reused.

Where it appears: Private benchmark confirmation; the run header's dataset line.

Seasonal strength. Energy in the forecast concentrated at the series' own dominant repeating rhythm -- a higher value makes the forecast more strongly periodic.

Where it appears: Sparse features (SAE) -- the causal channel battery's evidence columns, and every channel name rendered from it.

Sharing class. Whether an atlas concept's models also agree on the INPUTS that drive it, not just its effect: 'shared' when every model in it is connected through agreeing cross-model input pairs, 'partially shared' when only some pairs agree, 'convergent' when none do, and 'single-model' when only one model holds the concept at all.

Where it appears: Model comparison — the sharing map and every concept card; produced by sae/concept_profiles.py.

Skip lens and tuned lens. Two ways to ask what forecast a middle layer already implies: the skip lens patches that layer's states in as the final block's output and lets the model's own head decode them, while the tuned lens fits a held-out ridge probe instead, correcting the skip lens's miscalibration at early layers.

Where it appears: Forecast lens.

sMAPE and pinball loss. The two companion forecast metrics: sMAPE is a symmetric percentage error (scale-free but unstable near zero), and pinball loss scores the predicted quantiles rather than the point forecast, which is the only one of the three sensitive to whether a model's uncertainty is honest.

Where it appears: Behavioral profile's per-family metric table.

Spread. How much the forecast varies within itself (its standard deviation) -- a higher value makes the forecast swing more from step to step.

Where it appears: Sparse features (SAE) -- the causal channel battery's evidence columns, and every channel name rendered from it.

Stitching gain. The extra held-out R² a ridge map from one model's layer to the other's achieves above a hand-crafted input-feature probe on the same targets — the gain, never the raw R², is the evidence, because the models read the same input and a raw R² is therefore partly trivial.

Where it appears: Stitching probes (both directions, A→B and B→A, reported separately).

Sufficiency. Whether patching a candidate head set's clean values into an otherwise-corrupted forecast restores it toward the clean level — the complement of necessity.

Where it appears: The seasonality circuit.

The series as the resampling unit. Every bootstrap and every train/test split in this repo resamples whole series, never windows, because windows within one series are strongly dependent and every model scores the same series — splitting on windows would make every interval far too narrow.

Where it appears: Every confidence interval and p-value in the report.

Tier (synthetic / real_derived). Whether a benchmark series touched real data at all: synthetic series are generated from scratch and carry exact ground truth with no distributional leakage, while real_derived series are built from real corpora and inherit their distribution even though every instance still passes the leakage audit.

Where it appears: The corpus this run reads; recorded per sample in the sealed manifest.

Token width. How many timesteps one of a model's tokens covers (about 32 for a patch tokenizer, 1 for a per-timestep quantizer), which sets the finest lag that model's attention can express and is multiplied into every plotted lag axis so a lag of 300 means the same 300 timesteps for every model.

Where it appears: Attention analysis' lag axes; the fairness card's finest-resolvable-lag row.

Transfer. Whether another model separates the same series a concept fires hardest on, scored as the best of that model's features against random series sets drawn with the same mix of series kinds, and called reciprocal only when that feature's own top series are in turn separated by the original concept.

Where it appears: The SAE section's concept universality table; produced by the concepts stage.

Trend. Slope of the forecast -- whether it rises or falls as it continues forward -- a higher value pushes the forecast to slope upward more steeply.

Where it appears: Sparse features (SAE) -- the causal channel battery's evidence columns, and every channel name rendered from it.

Untrained-twin floor. The same measurement run against a randomly-initialized copy of a model's own architecture, giving the level any two networks of that shape reach before either has learned anything; a similarity that fails to clear it is architecture, not learning.

Where it appears: Null-baseline comparisons for CKA and stitching gain.

Window. A pooled fixed-length interval of real time (32 timesteps by default, alignment.window) that every model's own tokens are mapped onto, so an architecture reading one timestep per token and one reading thirty-two become comparable at all; it is a unit of time, never a unit of any one model's tokenization.

Where it appears: Every cross-model section — the shared [series, window, dim] axis all of them operate on.

A worked example — reading a real report end to end

docs/worked_example.md walks one real run (google/timesfm-2.5-200m-pytorch vs. amazon/chronos-t5-base, 288 series) section by section, quoting that run's own artifacts: which number is the finding, why a particular confidence interval changes the conclusion, which caveat applies before any depth-located claim, and — the part nothing else in this repo teaches — which reported number to deliberately not act on, and why. Read it once with a report open; the stage docs above and the glossary tell you what things mean, and this tells you how to weigh them.

Statistical treatment

The resampling unit is always the series: windows within a series are strongly dependent, both models score the same series, and corruptions are applied to the same series. Concretely:

• L0 family comparisons are paired bootstrap tests of per-series MASE differences, Holm-corrected across families; a "strength" requires a corrected p below alpha and a CI excluding zero. Ratios are kept as effect sizes.
• L1 CKA at the peak pair uses a cluster bootstrap (resample series, keep their windows together); family-conditioned CKA gets per-family CIs.
• L2 reports a CI on the best stitching gain by resampling validation series with the fitted probes held fixed — evaluation uncertainty, deliberately not refit variability.
• L3 keeps per-series sensitivity deltas so fingerprint agreement gets a paired cluster bootstrap (the same series resample applied to both models); behavioral deltas carry CIs.
• Profile probe accuracies are CIs over held-out series against a majority-class chance line. Cluster AMI gets a series-bootstrap CI.
• Bootstrap p-values are approximate and floored at 1/n_boot; they are decision aids, not precision instruments. stats.n_boot_heavy caps the expensive statistics separately from the cheap ones.

Install

pip install -r requirements.txt          # core
pip install chronos-forecasting timesfm          # the models you compare (timesfm>=2.0; see below)
pip install umap-learn                   # optional, nicer cluster maps

Quickstart

Smoke run (no downloads, no GPU, ~1 minute) — validates every stage including confirmation and produces a real report from two mock architectures:

python run.py --config configs/smoke.yaml
open runs/smoke/report.html

Real comparison:

# 1. point data.path at a tsfm_benchmark PUBLIC (dev) directory
# 2. verify token-time alignment on your installed library versions:
python run.py --config configs/default.yaml --check-alignment timesfm
python run.py --config configs/default.yaml --check-alignment chronos
# 3. explore freely on dev:
python run.py --config configs/default.yaml
# 4. when analysis is FROZEN, enable confirm with the PRIVATE directory
#    and run the one-shot confirmation:
python run.py --config configs/default.yaml --stages confirm,report

Stage selection and reruns:

python run.py --config configs/default.yaml --stages extract,l1,report
python run.py --config configs/default.yaml --stages l2 --force l2

Stages skip themselves when their artifacts exist; the report renders whatever subset ran. Every stage has an enabled flag and its own knobs in the YAML (configs/default.yaml documents all of them inline).

CLI reference

run.py --help is authoritative; this groups the flags that actually get used day to day (see configs/README.md for --list-configs):

Flag	What it does
--config C	Path to a YAML config.
--list-configs	List every config in configs/ with its derived shape/adapters/purpose.
--stages a,b,c	Run only these stages (default: every enabled stage).
--force STAGES / --force all	Rerun selected stages even if artifacts exist. --force only reaches stages named in --stages.
--allow-stale	Proceed even though a stage's config fingerprint no longer matches its on-disk artifacts (default: refuse).
--allow-partial-report	Don't fail the run when one report section builder raises (default: still write report.html with the failure named, then raise).
--check-alignment MODEL	Run the empirical impulse alignment check for one model and exit.
--discover-layers MODEL [--contains S]	Print module names for one model (layer-regex debugging).
--discover-spans MODEL [--span-stride N]	Measure a model's token→time map with an impulse sweep and cross-check it against the declared spans.
--probe-adapter MODEL	The zero-code entry point: load one model and print which strategy resolved at each probed seam.
--check-adapter MODEL	The full one-command adapter checklist (conformance, alignment, span discovery, every optional capability) as typed pass/warn/fail/not_applicable rows.
--new-adapter NAME --checkpoint ID	Scaffold models/contrib/<name>_adapter.py from the template, plus a matching smoke config.
--doctor	Full preflight, including loading every model for conformance + alignment.
--no-preflight / --allow-preflight-fail	Skip, or run despite failing, the fast static preflight that otherwise runs automatically.
--verify-provenance RUN_DIR	Diff a finished run's recorded git SHA/library versions/device/config hash against the current environment.
--export-results RUN_DIR	Re-export report/results.csv/.parquet from an existing run's findings, without re-rendering HTML.
--prune RUN_DIR [--yes]	Delete a run's regenerable caches (SAE *.pt, *.zarr stores). Dry run unless --yes; afterwards only --stages report runs there.
--bundle RUN_DIR [--bundle-out P] [--bundle-max-file-mb N]	Zip report.html plus every small readable artifact, with a SHARE_MANIFEST.json.
--verbose / --no-verbose	Force report.verbose regardless of the config.

Sharing and disk usage

A run directory is mostly regenerable tensors. Measured on a real 7-model run (7.3 GB): SAE checkpoints (sae/<model>/*.pt, including the @r1/@r2 seed replicates) about 5.0 GB, activations.zarr about 2.5 GB, and every JSON artifact plus report.html about 110 MB. On the mock smoke run with SAE, concepts and confirm enabled (tests/test_share.py's fixture) the run is 3.85 MB, 1.2 MB of it caches; pruning leaves 2.56 MB, and the bundle zip is 0.53 MB.

python run.py --bundle runs/<name>            # share: <name>_share.zip next to the run
python run.py --prune runs/<name>             # dry run: what would be removed, by kind
python run.py --prune runs/<name> --yes       # reclaim disk

--bundle includes report.html (always) and every non-cache file with a data extension up to --bundle-max-file-mb (default 5); backup renders (report_*.html, *.pre_*) are left out. SHARE_MANIFEST.json inside the zip lists every included and excluded file with its reason (heavy cache, over size cap, backup render, extension).

--prune writes pruned.json and keeps activations_attrs.json (the store's layer names and stack layout, so depth figures stay on the same axis). Afterwards only --stages report can run in that directory: any other stage is refused with the prune date, because it would read deleted caches. The report re-renders the same sections with the same text; a section that needs a deleted checkpoint (the SAE feature exemplar panel) says cache pruned: ... (see pruned.json) instead of drawing. To recompute, run the config into a new run directory (change run.name). A .zarr symlink is unlinked, never followed. The end of a successful run logs the run size and these two commands.

Confirmation discipline

The private corpus is a consumable. Run the confirm stage once, after all exploration is frozen; it refuses to overwrite existing confirmation artifacts, requires seal verification by default (confirm.require_seal: true), and logs loudly when it runs. If the private set is ever peeked at, regenerate a fresh private epoch with tsfm_benchmark rather than reusing it. Findings that were not pre-registered as dev hypotheses cannot be rescued by the confirm stage — that is by design.

GPU notes

Extraction and forecasting run batched under autocast (bf16 by default); activations are stored float16 in a chunked zarr store; CKA and ridge solves are closed-form on-device in fp32. Models are loaded lazily and released between stages (run.keep_models_loaded: true to keep both resident). The main VRAM knobs are models[*].batch_size, capture_layer_stride, and the max_rows caps in l1/l2/internals. Bootstraps add analysis-time CPU/GPU cost only, never extraction cost; tune stats.n_boot / stats.n_boot_heavy if analysis stages feel slow.

The alignment caveat (read this once)

Cross-architecture comparison is only as valid as the token-to-time mapping. Adapters declare spans; libraries change. The --check-alignment command runs an empirical impulse test: an input impulse in window w must perturb aligned window w most, and the command prints the per-layer diagonal-hit fraction. Values well below 1.0 at late layers are normal (attention mixes positions); near-zero everywhere means the declared spans are wrong for your installed version — fix the adapter before trusting any cross-model number. alignment.sanity_check: true also runs a spot check during extraction.

Tested adapter targets: chronos-forecasting>=1.2 with chronos-t5-* checkpoints, chronos-forecasting>=1.4 with chronos-bolt-* checkpoints (patch geometry is read from the checkpoint config; forecasts are deterministic, so its L3 curves carry no sampling noise), and the timesfm PyPI package's 2.5-only API (>=2.0, which dropped the old TimesFmHparams/TimesFmCheckpoint/TimesFm class entirely, with no Python>=3.12-compatible release that keeps it) with google/timesfm-2.5-200m-pytorch. TimesFM's attention runs through a fused SDPA kernel by default with no exposed weights, so attention_patterns temporarily swaps in the library's own unfused dot-product math to recover them (same output, same query/key/value, restored right after). Its feed-forward block has no wrapping MLP submodule (mlp_info finds nothing), so MLP mean-ablation is unavailable for this checkpoint family; head-level ablation still works via the attention output projection. An older timesfm install that still exposes the Hparams API would need the class-based loader instead of the current adapter.

Extending

New model — bring your own checkpoint. Nothing downstream changes in principle; several real adapters spanning encoder-decoder, decoder-only, encoder-only and cross-series-attention architectures have proven this out, and the rare shared-code gap a new architecture surfaces gets fixed once in the shared module, not per-adapter.

Start at docs/ADDING_A_MODEL.md — the ModelAdapter contract, the decision tree (try the zero-code probe path before writing anything; scaffold a contrib adapter with run.py --new-adapter only if it refuses), the commands to run at each step, what a refusal from one of them means, the licensing convention, and a worked case study (the Sundial normalization bug) of the single most common class of adapter mistake. tsfm_lens/ADAPTERS.md (generated, never hand-edited) is its companion: every currently-registered adapter's tier, capabilities and default checkpoint, plus the narrative explaining what each verification step measures. Between the two, nothing about adding a model should require reading adapter source code to answer.

Layer screening. Before the expensive per-layer analyses (Profile, L1, SAE), the layer_screen stage (analysis/layer_screen.py, method work_bend by default) scores each model's own layers as a cheap residual-trajectory proxy for "worth a closer look," and sae.targets: auto consumes its selection. It runs automatically right after extract; run_layer_screen_bakeoff.py (study driver, dev branch) is the standalone, null-controlled comparison that chose work_bend over the alternatives.

SAE phase. sae.enabled: true trains a real TopKSAE (sae/models.py/train.py) straight from the store's activations, with a real eval harness (sae/eval.py: reconstruction fidelity, dead-feature rate, forecast preservation) and a ground-truth feature-alignment score (sae/ground_truth.py). sae/crosscoder.py is a feasibility-tested (not yet a pipeline stage) joint dictionary across two models' activations, and sae/matching.py correlates two independently-trained dictionaries' activation profiles to find cross-model feature partners without one. extraction.hooks.token_patch is the shared intervention primitive feature ablation and L3 patching both reuse.

Design decisions and caveats

• CKA is correlational and reported as such; the report's framing keeps L1 claims geometric, L2 claims linear-translatability, L3 claims causal.
• Stitching direction matters (A→B and B→A both reported); R² is variance-weighted on a held-out series split, never a window split. The L2 gain CI holds fitted probes fixed, so it quantifies how stable the number is on new series, not how stable refitting would be.
• Patching is within-model. Cross-model activation transplants are not meaningful without a learned mapping; what is compared is each model's restoration-by-depth curve on a shared relative-depth axis.
• Chronos (sampled) forecasts add noise to its L3 curves that TimesFM and Chronos-Bolt (both deterministic) do not have.
• Fingerprint comparison interpolates over relative depth, which assumes depth fraction is the right axis to compare a 50-layer decoder against a 12-block encoder; it is a reasonable convention, not a fact.
• The AMI bootstrap resamples label pairs, not the clustering itself; it reflects assignment stability, not KMeans initialization variance.
• Cluster labels are approximate by construction (majority family + salient signal statistics); they are hypotheses for reading the maps.
• A smoke-mode confirm is mechanically real but distributionally identical to dev (same generators, shifted seed); genuine confirmation value comes from sealed private corpora.
• Metrics (MASE, sMAPE, pinball) are implemented in ~20 lines rather than importing a large evaluation framework; formulas are in analysis/l0_behavioral.py where they can be audited. Statistical machinery is similarly ~80 audited lines in analysis/stats.py built on numpy, because the bootstrap designs (cluster, paired) are the substance and generic CI libraries obscure them.

Layout

tsfm_lens/
  config.py            typed YAML config, per-stage enables and defaults
  data.py              sealed-corpus loader (+ jsonl fallback, smoke generator)
  models/              ModelAdapter contract; TimesFM/Chronos/Chronos-Bolt/
                       Chronos-2/Sundial/mock adapters, conformance + capability-
                       matrix generators
  extraction/          hooks, time alignment (+ impulse check), zarr store
  analysis/            stats (bootstrap/Holm), l0 behavioral, layer_screen
                       (cheap per-model layer scoring, feeds sae.targets),
                       internals profile, lens (skip + tuned forecast lens,
                       + spectral_lens's frequency-domain crystallization
                       depth), l1 geometry, l2 stitching, l3
                       perturbation+patching (per window), attention (lag
                       profiles, head/MLP ablation, cross-attn), clustering,
                       exemplars (case studies), confirm (private benchmark)
  sae/                 TopKSAE baseline (models/train/eval/ground_truth),
                       cross-model matching, crosscoder (feasibility-tested,
                       not yet a pipeline stage)
  report/              single-file interactive HTML report
  pipeline.py          stage DAG, artifact skipping, dependency resolution
run.py                 CLI
render_stage_docs.py, render_glossary.py, render_adapter_docs.py
                       doc generators (`--check` to confirm they aren't stale)
run_sae_ablation.py, run_sae_describe.py, run_sae_roles.py, run_probe_sweep.py
                       standalone scripts kept on `main` because a passing
                       test imports or invokes each one directly; every other
                       standalone `run_*.py` study driver (including
                       run_layer_screen_bakeoff.py, run_crosscoder_stage0.py
                       and run_crosscoder_ladder.py, once their own study-only
                       tests moved with them) that reuses an already-extracted
                       run's config/data/store for probes that aren't (yet, or
                       ever) pipeline stages lives on the `dev` branch only
configs/               default.yaml (real pair), smoke.yaml (mocks)
tests/test_smoke.py    full pipeline end-to-end + confirmation verdict test