📊 Irreducible Path Entropy in Neural Networks
A Quantitative Information-Theoretic Framework for Entropy Propagation Across Computational Decision Trajectories
📌 Overview
This repository contains the full research output for the paper:
"Irreducible Path Entropy in Neural Networks" Samir Baladi — EntropyLab Independent Research Series, May 2026
The paper introduces Irreducible Path Entropy (H_path) — a formally defined,
layer-integrated information-theoretic metric that quantifies how much uncertainty
accumulates, transforms, and becomes unrecoverable along the inference trajectory
of a neural network.
The framework is grounded exclusively in information theory and systems-level analysis, without semantic, cognitive, or anthropomorphic assumptions.
📋 OSF Preregistration
| Field | Value |
|---|---|
| Registration Type | OSF Preregistration |
| Registry | OSF Registries |
| Associated Project | https://osf.io/yaevt |
| Date Registered | May 20, 2026 · 6:17 AM UTC |
| License | CC-By Attribution 4.0 International |
| Internet Archive | osf-registrations-7wp9h-v1 |
| Registration DOI | 10.17605/OSF.IO/7WP9H |
🧭 Motivation
Modern neural networks achieve high performance while remaining structurally opaque. As noted by Hinton (2023), the learning algorithm is designed — but the precise inference dynamics remain inaccessible even to the architects who built them.
Existing interpretability tools address specific aspects of this opacity. This work addresses a gap: no unified, layer-integrated metric existed for characterising entropy accumulation along the full computational decision path.
H_path fills this gap.
🔬 Core Constructs
Local Path Entropy
H_path(l) = − Σ_k p_{l,k} · log p_{l,k}
Shannon entropy of the conditional activation distribution at layer l.
Cumulative Path Entropy
H_path^(L) = Σ_{l=1}^{L} H(P_l)
Total informational uncertainty accumulated across all L layers.
Irreducible Path Entropy
H_irr^(L) = H_path^(L) − H_red^(L)
The component of path entropy that cannot be recovered from external observations.
Observability Index
Ω(N) = 1 − H_irr^(L) / H_path^(L) ∈ [0, 1]
Ω = 1→ fully observable networkΩ = 0→ completely irreducible inference dynamics
Reducibility Condition
A layer l is reducible if there exists a measurement operator M_l such that:
I(h_l ; M_l(y)) ≥ H_path(l) − δ*
where δ* is the reducibility tolerance threshold.
Entropic Leakage
Δ(L) = H_path^(L) − I(x ; h_L)
Uncertainty introduced across computation not explained by retained input information.
📐 Scope and Interpretive Closure
This framework is restricted to formal quantitative analysis of entropy propagation in artificial neural networks.
Not within scope:
- General theories of intelligence, cognition, or consciousness
- Claims regarding intentionality, agency, or phenomenology
- Semantic or anthropomorphic interpretation of results
Within scope:
- Reproducible computational analysis
- Information-theoretic formalisation
- Systems-level characterisation of inference behaviour
- Experimentally observable entropy dynamics
📂 Repository Structure
irreducible-path-entropy/
│
├── 📄 README.md # This file
├── 📄 LICENSE # MIT License
├── 📄 CHANGELOG.md # Version history
├── 📄 AUTHORS.md # Author and contributor metadata
│
├── 📁 paper/
│ ├── Irreducible_Path_Entropy_Baladi_2026.pdf # Publication-ready paper
│ └── preprint_metadata.json # Zenodo/OSF submission metadata
│
├── 📁 formalism/
│ ├── definitions.md # All formal definitions (1–5)
│ ├── reducibility_conditions.md # Reducibility threshold derivations
│ ├── observability_index.md # Ω construction and properties
│ └── entropic_leakage.md # Δ(L) derivation and interpretation
│
├── 📁 figures/
│ ├── fig1_path_entropy_accumulation.png # Layer-wise H_path vs H_red
│ ├── fig2_reducibility_phase_diagram.png # Phase diagram (ρ vs I)
│ └── fig3_observability_architectures.png # Ω across architecture types
│
├── 📁 numerical/
│ ├── entropy_estimator.py # k-NN entropy estimation module
│ ├── mutual_information.py # MI estimator for H_red
│ ├── observability_compute.py # Ω computation pipeline
│ ├── architecture_comparison.py # MLP / CNN / Transformer benchmarks
│ └── requirements.txt # Python dependencies
│
├── 📁 experiments/
│ ├── protocol.md # Full reproducibility protocol
│ ├── config_feedforward.yaml # MLP experiment configuration
│ ├── config_cnn.yaml # CNN experiment configuration
│ └── config_transformer.yaml # Transformer experiment configuration
│
└── 📁 references/
└── bibliography.bib # BibTeX reference file
⚙️ Reproducibility Protocol
All results are reproducible under the following conditions:
- Fixed weights — no stochastic inference-time modifications
- Consistent discretisation — activation binning scheme fixed across layers
- Fixed estimator parameters — bandwidth / neighbourhood
kheld constant - Fixed dataset —
D = {x_i}held constant across comparative measurements - Fixed random seed —
seed=42for deterministic behaviour
Estimation Pipeline
Step 1 → Record activations {h_l(x_i)} at each layer l
Step 2 → Apply k-NN entropy estimator → H_path(l)
Step 3 → Estimate I(h_l ; y) → H_red^(L)
Step 4 → Compute Ω = 1 − H_irr / H_path
🏗️ Architecture Findings (Illustrative)
| Architecture | Depth | H_path (nats) |
Ω Index |
Regime |
|---|---|---|---|---|
| MLP (2L) | 2 | 0.31 | 0.91 | Reducible |
| MLP (6L) | 6 | 0.68 | 0.74 | Reducible |
| MLP (12L) | 12 | 1.14 | 0.61 | Reducible |
| CNN (8L) | 8 | 0.87 | 0.68 | Reducible |
| Transformer (12L) | 12 | 1.42 | 0.52 | Borderline |
| Transformer (24L) | 24 | 2.05 | 0.39 | Irreducible |
Values are illustrative. Empirical calibration required for specific architectures.
🧪 Test Results
$ pytest tests/
============================= test session starts =============================
collected 19 items
tests/test_entropy_estimator.py ......... [47%]
tests/test_mutual_information.py ..... [73%]
tests/test_observability.py ...... [100%]
============================= 19 passed in 0.435s =============================
👤 Author
Samir Baladi Independent Interdisciplinary Researcher Ronin Institute / Rite of Renaissance
- 📧 gitdeeper@gmail.com
- 🆔 ORCID: 0009-0003-8903-0029
- 🐙 GitHub: gitdeeper12
- 🦊 GitLab: gitdeeper12
- 🏕 Codeberg: gitdeeper12
📚 Key References
| # | Reference |
|---|---|
| 1 | Sundararajan et al. (2017). Axiomatic attribution for deep networks. ICML. |
| 2 | Alain & Bengio (2016). Understanding intermediate layers via linear probes. arXiv:1610.01644. |
| 3 | Elhage et al. (2021). A mathematical framework for transformer circuits. Anthropic. |
| 4 | Tishby & Schwartz-Ziv (2017). Opening the black box via information. arXiv:1703.00810. |
| 5 | Kozachenko & Leonenko (1987). Sample estimate of entropy of a random vector. PIT. |
| 6 | Cover & Thomas (2006). Elements of Information Theory (2nd ed.). Wiley. |
| 7 | Hinton, G. (2023). Interview. 60 Minutes, CBS News. |
| 8 | Baladi, S. (2026). ENTRO-OMEGA: Unified Adaptive Stabiliser. DOI: 10.5281/zenodo.19562999. |
🔗 Links
| Resource | Link |
|---|---|
| 📄 Zenodo Preprint | doi.org/10.5281/zenodo.20222840 |
| 📝 OSF Registration | doi.org/10.17605/OSF.IO/7WP9H |
| 📦 PyPI Package | pypi.org/project/entropath |
| 🐙 GitHub Repository | github.com/gitdeeper12/ENTRO-PATH |
| 🦊 GitLab Mirror | gitlab.com/gitdeeper12/ENTRO-PATH |
| 🪣 Bitbucket Mirror | bitbucket.org/gitdeeper-12/ENTRO-PATH |
| 🏕 Codeberg Mirror | codeberg.org/gitdeeper12/ENTRO-PATH |
| 🏛️ ENTRO-OMEGA (E-LAB-10) | doi.org/10.5281/zenodo.19562999 |
| 🔬 OSF Project | osf.io/yaevt |
| 📚 Internet Archive | archive.org/details/osf-registrations-7wp9h-v1 |
| 🆔 ORCID Profile | orcid.org/0009-0003-8903-0029 |
📜 License
This project is released under the MIT License. See LICENSE for full terms.
The OSF registration is released under CC-By Attribution 4.0 International.
EntropyLab Independent Research Series · May 2026 Information Theory · Neural Network Interpretability · Entropy Dynamics
Registration DOI: 10.17605/OSF.IO/7WP9H · Preprint DOI: 10.5281/zenodo.20222840
Metadata
Release files for entropath 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| entropath-1.0.0.tar.gz | 167.7 kB | Without provenance | Details for entropath-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| entropath-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for entropath-1.0.0-py3-none-any.whl |
Total release size: 182.4 kB
Release files / entropath-1.0.0.tar.gz
| Download URL | entropath-1.0.0.tar.gz |
|---|---|
| Size | 167.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cd1c774187d2105fce0608a60968714129436f716968e718ff2db1fc2c17f471
|
|
BLAKE2b-256 checksum How to use checksums |
9f2e0eba11f5d8f7bba4ad415bc951621270f7e6113a0d1816d72cfa759d08ca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
ENTRO-PATH-Uploader/1.0
|
Release files / entropath-1.0.0-py3-none-any.whl
| Download URL | entropath-1.0.0-py3-none-any.whl |
|---|---|
| Size | 14.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2e547c04e4ba454cb48e3984e5fc1a3636f6b6ce012b3729dca80ab322f12faf
|
|
BLAKE2b-256 checksum How to use checksums |
282062b4ff1daa223e4ecff5011cfe6b1b6b044f8db1406925e7d755d975dccd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
ENTRO-PATH-Uploader/1.0
|
Security for entropath 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.