Glassbox

Prove what your model actually did, and find out why your AI system changed.

Trace the circuit behind a model's decision, measure how faithful that explanation is, and draft the EU AI Act Annex IV technical documentation from one function call, on open-weight models. A newer investigation layer compares runs of an AI system before and after a change and tests which change actually mattered.

Website · Live Demo · Paper · PyPI · User Guide · What is proven

Most explainability tools describe a model from the outside and can't tell you whether the explanation is true. Glassbox reads the inside. It ranks every attention head by its causal effect on a decision, keeps the minimal circuit that drives it, and then measures how faithful that circuit is by re-running the model with parts ablated.

The finding behind the tool: a model's confidence is essentially uncorrelated with how faithful its explanation is (r = 0.009). An auditor who relies on a confidence score is relying on something unrelated to faithfulness. Faithfulness has to be measured, not assumed.

Two parts, two maturity levels:

Every claim in this README is tracked, with its evidence and status, in ASSURANCE.md.

Install

pip install glassbox-mech-interp

Extras: pip install "glassbox-mech-interp[compliance]" (Annex IV engine) · [jupyter] (notebook widget) · [dev] (tests + lint). No install? Try the live demo on Hugging Face.

The investigation layer (glassbox.v7) ships in the same package from 4.6.0. To re-run the experiments or reproduce an evidence package, install from source, since the experiments and recipes are not in the wheel:

git clone https://github.com/designer-coderajay/glassbox-mech.git
cd glassbox-mech
pip install -e ".[dev]"

60-second quickstart: explain one decision

from transformer_lens import HookedTransformer
from glassbox import GlassboxV2

model = HookedTransformer.from_pretrained("gpt2")   # any open-weight transformer
gb    = GlassboxV2(model)

result = gb.analyze(
    prompt    = "When Mary and John went to the store, John gave a drink to",
    correct   = " Mary",
    incorrect = " John",
)

print(result["circuit"])         # [(9, 9), (9, 6), ...]  <- (layer, head)
print(result["faithfulness"])    # {'sufficiency': 1.0, 'comprehensiveness': 0.543, 'f1': 0.704, ...}

Measured on GPT-2 IOI with the current build: suff 1.00 · comp 0.543 · F1 0.704 · Grade B, in about 1.8 s on an Apple M1 Pro.

What you get back

• The circuit: which attention heads causally drive the decision, ranked.
• Faithfulness metrics and a grade (A–D): sufficiency, comprehensiveness and their F1, computed by ablation, so they are reproducible rather than asserted.
• A plain-English explanation and a self-graded evidence tier.
• An EU AI Act Annex IV draft: all 9 sections laid out. Fields the analysis can measure (circuit, faithfulness, model details) are filled; fields only the provider can supply, including the §8 declaration of conformity, are marked [PROVIDER TO COMPLETE].

from glassbox import AnnexIVReport
report = AnnexIVReport(model_name="gpt2", system_purpose="...", provider_name="Acme Corp",
                       provider_address="...")
report.add_analysis(result)
report.save_json("annex-iv.json")

Honest by design

When a model can't do a task, Glassbox says so instead of inventing a clean story. Point it at raw GPT-2 for a credit decision and it finds no faithful circuit: F1 0.00–0.08, Grade C, CHECKS FAILED. That refusal is intended; the tool won't certify what it can't explain. ("CHECKS FAILED" is Glassbox's own quality gate, not a legal ruling. Before 4.7.0 it was labelled "NON-COMPLIANT".)

Measured results

Full methodology and raw data: BENCHMARKS.md. Reproduce with scripts/benchmark.py.

How it compares

Model coverage

Validated end-to-end on nine architecture families (ten model series; Yi uses the Llama architecture) from 82M to 12B: GPT-2, Pythia/GPT-NeoX, GPT-Neo, OPT, Llama-3, Mistral, Gemma-2, Qwen2/2.5, Yi and Phi-3, with grouped-query attention and RMSNorm handled correctly.

• Beyond about 13B: gradient-based attribution needs a multi-GPU cluster. This is implemented but not yet validated live.
• Closed APIs (GPT-4, Claude, Gemini): only a weaker black-box tier is available, because faithfulness can't be measured without the weights.

See docs/VALIDATION_LOG.md.

EU AI Act: Annex IV

Providers of high-risk AI systems must draw up Annex IV technical documentation (Article 11 of Regulation (EU) 2024/1689). Glassbox drafts the 9-section structure from one analysis: measured evidence fills what it can, and the rest is marked for the provider. Report packs also map to the NIST AI RMF and ISO/IEC 42001.

Application dates for high-risk obligations were amended by Regulation (EU) 2026/1744 (the Digital Omnibus on AI). Check the current legal text for the dates that apply to your system; this README does not restate them.

Glassbox produces evidence and documentation, not a conformity declaration. It is not legal advice and does not by itself make anyone compliant. The EU AI Act does not require mechanistic interpretability. See Legal Notices.

Investigating why an AI system changed

Research stage. In the PyPI package from 4.6.0. Licensed under BSL 1.1.

When a RAG assistant or an agent starts behaving differently after a week of changes (new index, edited prompt, new framework version), the investigation layer:

1. Records each run as a local trace: nested spans, tool calls and hashed content. Raw text is not stored by default.
2. Compares many runs before and after the change. A noise-aware group diff separates real differences from run-to-run sampling noise.
3. Tests each change alone and each change reverted alone, then reports which changes moved the outcome and which did not.
4. Packages the result as an evidence bundle with a sha256 manifest that anyone can verify.

Try the built-in demo, a small RAG incident with three simultaneous changes:

python -m glassbox.v7.investigate demo --out demo_out --scorer gpt2

It writes a plain-language REPORT.md and an evidence package. The live demo has two worked examples: this RAG incident on GPT-2, and a simulated tool-using agent whose true effects are known and shown next to Glassbox's answer.

On your own system, a "change" is any configuration difference you can switch on and off and re-run: model version, prompt, retrieval setting, framework or tool version. You supply a function that runs your system under a given configuration and records a trace, plus one that reads the outcome; glassbox.v7.intervene.run_interventions does the rest. Only the change types listed in the experiments below have been tested.

What has been tested so far

Each experiment was pre-registered: the protocol was committed before the code and the run, and the results were committed unchanged. All were designed and run by the author, with the cause known in advance; none has yet been reproduced independently.

Not shown yet:

• that it finds unknown causes in real production systems;
• that it works across frameworks in general (so far: one second stack, retrieval only);
• that it saves engineers time. There are no external users so far.

A first divergence shows where runs differ, not why. Causal statements need the controlled interventions above, case by case.

How it works, start to finish

The whole system in one picture

Open docs/anatomy/glassbox-bench-light.html in a browser to scrub through it (details in docs/anatomy/README.md). The three dial readings are the IOI benchmark on GPT-2 small from Measured results; the per-head scores and which heads are lit are drawn for the picture, not taken from a run.

Two paths, two maturity levels. Path 1 (explain one decision) is released on PyPI. Path 2 (find out why an AI system changed) is the investigation layer, research stage; see above.

Path 1: explain one decision (GlassboxV2.analyze)

_{Find the circuit. Score every head in 3 passes, keep the few that carry the decision.}

_{Test it by taking it out. Faithfulness is computed by re-running the model with parts ablated, not asserted.}

Path 2: find out why an AI system changed (glassbox.v7)

_{Which change mattered? Switch each change on alone and revert each alone. Where runs first differ is not why.}

Run the built-in demo, a small RAG incident with three simultaneous changes:

python -m glassbox.v7.investigate demo --out demo_out --scorer gpt2

Which entry point should I use?

Documentation

Limitations

• White-box only for circuits: circuit analysis needs weights, activations and gradients. Open models are fully supported; closed APIs get the black-box tier only.
• Decisions, not free-form chat: circuit analysis needs a contrast between two outcomes to attribute against.
• Offline and sampled: Glassbox audits representative decisions for documentation and monitoring. It is not a real-time monitor on every request.
• Faithful ≠ fair: it measures whether the explanation matches the computation, not whether the decision is correct or unbiased.
• Investigation layer:
  • so far validated only on small open-weight models and author-designed incidents;
  • small effects (about 0.16 and below at 30 runs per configuration) can be missed;
  • redaction of trace content is best-effort;
  • a hash manifest proves integrity, not truth.

Developer tooling: diagram and illustration skills

Two optional Agent Skills are set up under .claude/skills/ for people who work on this repo with Claude Code. .claude/ is git-ignored here, so they live in your checkout and are not shipped with the package.

Re-install either with git clone <source> .claude/skills/<name>.

The bench figure in docs/anatomy/ is built with anatomy; the three English illustrations in docs/illustrations/ follow the Xiaohei style and are hand-drawn vectors with their generator included, not AI-generated art.

Contributing

Contributions are welcome. See CONTRIBUTING.md and our Code of Conduct, and run the suite with pytest --cov=glassbox -v.

License

Glassbox is dual-licensed. NOTICE holds the authoritative file-by-file map.

BSL 1.1 summary:

• Free for non-production use, and for production use inside your own organisation (for example, investigating or documenting your own AI systems).
• Needs a commercial license to offer a compliance, AI-governance, AI-investigation, debugging or observability service or product to third parties.
• Converts to Apache-2.0 four years after each version is first published.
• Copies of cross_model.py and the investigation layer obtained before 2026-10-11 remain under the terms they were published with.

The circuit-discovery and faithfulness core stays MIT and is never feature-gated. For commercial licensing, contact mahale.ajay01@gmail.com. Academic and non-profit research use may qualify for a no-cost license.

Citation

@article{mahale2026causally,
  title         = {Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language Explanations},
  author        = {Mahale, Ajay Pravin},
  year          = {2026},
  eprint        = {2603.09988},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi           = {10.48550/arXiv.2603.09988}
}