📊 Irreducible Path Entropy in Neural Networks

A Quantitative Information-Theoretic Framework for Entropy Propagation Across Computational Decision Trajectories

---

---

📌 Overview

This repository contains the full research output for the paper:

"Irreducible Path Entropy in Neural Networks" Samir Baladi — EntropyLab Independent Research Series, May 2026

The paper introduces Irreducible Path Entropy (H_path) — a formally defined, layer-integrated information-theoretic metric that quantifies how much uncertainty accumulates, transforms, and becomes unrecoverable along the inference trajectory of a neural network.

The framework is grounded exclusively in information theory and systems-level analysis, without semantic, cognitive, or anthropomorphic assumptions.

---

📋 OSF Preregistration

Field	Value
Registration Type	OSF Preregistration
Registry	OSF Registries
Associated Project	https://osf.io/yaevt
Date Registered	May 20, 2026 · 6:17 AM UTC
License	CC-By Attribution 4.0 International
Internet Archive	osf-registrations-7wp9h-v1
Registration DOI	10.17605/OSF.IO/7WP9H

---

🧭 Motivation

Modern neural networks achieve high performance while remaining structurally opaque. As noted by Hinton (2023), the learning algorithm is designed — but the precise inference dynamics remain inaccessible even to the architects who built them.

Existing interpretability tools address specific aspects of this opacity. This work addresses a gap: no unified, layer-integrated metric existed for characterising entropy accumulation along the full computational decision path.

H_path fills this gap.

---

🔬 Core Constructs

Local Path Entropy

H_path(l) = − Σ_k p_{l,k} · log p_{l,k}

Shannon entropy of the conditional activation distribution at layer l.

Cumulative Path Entropy

H_path^(L) = Σ_{l=1}^{L} H(P_l)

Total informational uncertainty accumulated across all L layers.

Irreducible Path Entropy

H_irr^(L) = H_path^(L) − H_red^(L)

The component of path entropy that cannot be recovered from external observations.

Observability Index

Ω(N) = 1 − H_irr^(L) / H_path^(L)  ∈ [0, 1]

• Ω = 1 → fully observable network
• Ω = 0 → completely irreducible inference dynamics

Reducibility Condition

A layer l is reducible if there exists a measurement operator M_l such that:

I(h_l ; M_l(y)) ≥ H_path(l) − δ*

where δ* is the reducibility tolerance threshold.

Entropic Leakage

Δ(L) = H_path^(L) − I(x ; h_L)

Uncertainty introduced across computation not explained by retained input information.

---

📐 Scope and Interpretive Closure

This framework is restricted to formal quantitative analysis of entropy propagation in artificial neural networks.

Not within scope:

• General theories of intelligence, cognition, or consciousness
• Claims regarding intentionality, agency, or phenomenology
• Semantic or anthropomorphic interpretation of results

Within scope:

• Reproducible computational analysis
• Information-theoretic formalisation
• Systems-level characterisation of inference behaviour
• Experimentally observable entropy dynamics

---

📂 Repository Structure

irreducible-path-entropy/
│
├── 📄 README.md                          # This file
├── 📄 LICENSE                            # MIT License
├── 📄 CHANGELOG.md                       # Version history
├── 📄 AUTHORS.md                         # Author and contributor metadata
│
├── 📁 paper/
│   ├── Irreducible_Path_Entropy_Baladi_2026.pdf   # Publication-ready paper
│   └── preprint_metadata.json                     # Zenodo/OSF submission metadata
│
├── 📁 formalism/
│   ├── definitions.md                    # All formal definitions (1–5)
│   ├── reducibility_conditions.md        # Reducibility threshold derivations
│   ├── observability_index.md            # Ω construction and properties
│   └── entropic_leakage.md               # Δ(L) derivation and interpretation
│
├── 📁 figures/
│   ├── fig1_path_entropy_accumulation.png   # Layer-wise H_path vs H_red
│   ├── fig2_reducibility_phase_diagram.png  # Phase diagram (ρ vs I)
│   └── fig3_observability_architectures.png # Ω across architecture types
│
├── 📁 numerical/
│   ├── entropy_estimator.py              # k-NN entropy estimation module
│   ├── mutual_information.py             # MI estimator for H_red
│   ├── observability_compute.py          # Ω computation pipeline
│   ├── architecture_comparison.py        # MLP / CNN / Transformer benchmarks
│   └── requirements.txt                  # Python dependencies
│
├── 📁 experiments/
│   ├── protocol.md                       # Full reproducibility protocol
│   ├── config_feedforward.yaml           # MLP experiment configuration
│   ├── config_cnn.yaml                   # CNN experiment configuration
│   └── config_transformer.yaml           # Transformer experiment configuration
│
└── 📁 references/
└── bibliography.bib                  # BibTeX reference file

---

⚙️ Reproducibility Protocol

All results are reproducible under the following conditions:

1. Fixed weights — no stochastic inference-time modifications
2. Consistent discretisation — activation binning scheme fixed across layers
3. Fixed estimator parameters — bandwidth / neighbourhood k held constant
4. Fixed dataset — D = {x_i} held constant across comparative measurements
5. Fixed random seed — seed=42 for deterministic behaviour

Estimation Pipeline

Step 1  →  Record activations {h_l(x_i)} at each layer l
Step 2  →  Apply k-NN entropy estimator → H_path(l)
Step 3  →  Estimate I(h_l ; y) → H_red^(L)
Step 4  →  Compute Ω = 1 − H_irr / H_path

---

🏗️ Architecture Findings (Illustrative)

Architecture	Depth	H_path (nats)	Ω Index	Regime
MLP (2L)	2	0.31	0.91	Reducible
MLP (6L)	6	0.68	0.74	Reducible
MLP (12L)	12	1.14	0.61	Reducible
CNN (8L)	8	0.87	0.68	Reducible
Transformer (12L)	12	1.42	0.52	Borderline
Transformer (24L)	24	2.05	0.39	Irreducible

Values are illustrative. Empirical calibration required for specific architectures.

---

🧪 Test Results

$ pytest tests/
============================= test session starts =============================
collected 19 items

tests/test_entropy_estimator.py ......... [47%]
tests/test_mutual_information.py ..... [73%]
tests/test_observability.py ...... [100%]

============================= 19 passed in 0.435s =============================

---

👤 Author

Samir Baladi Independent Interdisciplinary Researcher Ronin Institute / Rite of Renaissance

• 📧 gitdeeper@gmail.com
• 🆔 ORCID: 0009-0003-8903-0029
• 🐙 GitHub: gitdeeper12
• 🦊 GitLab: gitdeeper12
• 🏕 Codeberg: gitdeeper12

---

📚 Key References

#	Reference
1	Sundararajan et al. (2017). Axiomatic attribution for deep networks. ICML.
2	Alain & Bengio (2016). Understanding intermediate layers via linear probes. arXiv:1610.01644.
3	Elhage et al. (2021). A mathematical framework for transformer circuits. Anthropic.
4	Tishby & Schwartz-Ziv (2017). Opening the black box via information. arXiv:1703.00810.
5	Kozachenko & Leonenko (1987). Sample estimate of entropy of a random vector. PIT.
6	Cover & Thomas (2006). Elements of Information Theory (2nd ed.). Wiley.
7	Hinton, G. (2023). Interview. 60 Minutes, CBS News.
8	Baladi, S. (2026). ENTRO-OMEGA: Unified Adaptive Stabiliser. DOI: 10.5281/zenodo.19562999.

---

🔗 Links

Resource	Link
📄 Zenodo Preprint	doi.org/10.5281/zenodo.20222840
📝 OSF Registration	doi.org/10.17605/OSF.IO/7WP9H
📦 PyPI Package	pypi.org/project/entropath
🐙 GitHub Repository	github.com/gitdeeper12/ENTRO-PATH
🦊 GitLab Mirror	gitlab.com/gitdeeper12/ENTRO-PATH
🪣 Bitbucket Mirror	bitbucket.org/gitdeeper-12/ENTRO-PATH
🏕 Codeberg Mirror	codeberg.org/gitdeeper12/ENTRO-PATH
🏛️ ENTRO-OMEGA (E-LAB-10)	doi.org/10.5281/zenodo.19562999
🔬 OSF Project	osf.io/yaevt
📚 Internet Archive	archive.org/details/osf-registrations-7wp9h-v1
🆔 ORCID Profile	orcid.org/0009-0003-8903-0029

---

📜 License

This project is released under the MIT License. See LICENSE for full terms.

The OSF registration is released under CC-By Attribution 4.0 International.

---