sigma-pi 0.1.1

A lightweight tool to calculate Predictive Integrity (PI) for PyTorch models.

ΣPI: Observe the Cognitive ability of Your AI Model

ΣPI is a lightweight, universal SDK to calculate Predictive Integrity (PI), a metric from the Integrated Predictive Workspace Theory (IPWT) of consciousness. It provides a powerful, real-time proxy for your model's "cognitive state" during training.

Stop just looking at loss. Start observing how your model learns.

What is Predictive Integrity (PI)?

PI is a score (0 to 1) reflecting a model's internal world model integrity, derived from prediction error (Epsilon), model uncertainty (Tau), and global gradient norm (Surprise). High PI indicates healthy learning; a drop can signal issues like overfitting before loss metrics do.

Why Use ΣPI?

• Early Warning for Training Instability: Detects subtle shifts in model "cognition" before loss metrics diverge.
• Insight into OOD Impact: Quantifies the "surprise" your model experiences when encountering out-of-distribution data.
• Understanding Model Overfitting: Reveals when your model's internal world becomes too rigid or too chaotic.
• Quantifying Cognitive Load: Provides a novel metric for the "effort" your model expends to integrate new information.

Model Zoo & Experiments

Our test suite is now centered around a lightweight (~1M parameter) Vision Transformer architecture to facilitate rapid experimentation on cognitive learning principles. We compare three main variants on CIFAR-10, using SVHN as an Out-of-Distribution (OOD) validation set.

The goal is to observe how different learning strategies perform under resource constraints, providing a clearer view of the benefits of mechanisms like Gated Backpropagation (GBP).

"Don't just train your model. Understand its mind."

Baseline ViT	4x1 MoE-ViT	16x4 MoE-ViT	16x4 GBP-MoE-ViT with 3σ Learning
~0.81M	~1.21M	~1.23M	~1.23M

Old result (CNN, ResNet...) in here.

Installation

pip install sigma-pi

How to Use

The sigma-pi package provides the core SigmaPI monitor. To replicate the experiments and use the full testing framework, you must first clone the repository.

git clone https://github.com/dmf-archive/SigmaPI.git
cd SigmaPI

Note: This package does not automatically install PyTorch. Please manually install the appropriate version for your system (CPU or CUDA) before proceeding.

After setting up PyTorch, install the testing framework dependencies:

pip install -e .[dev]

The testing framework is modular and configuration-driven.

1. Configure Your Experiment

Create or modify a configuration file in test/configs/. For example, test/configs/base_vit.py:

# test/configs/base_vit.py

# Model parameters
model_config = {
    'model_type': 'base',
    'embed_dim': 128,
    'depth': 6,
    # ... other model params
}

# Training parameters
train_config = {
    'epochs': 20,
    'batch_size': 256,
    # ... other training params
}

2. Run the Experiment

Launch the experiment from the root directory using the test/run_experiment.py script:

python test/run_experiment.py --config test/configs/base_vit.py

To run the other variants, simply point to their respective config files:

# Run MoE-ViT experiment
python test/run_experiment.py --config test/configs/moe_vit.py

# Run GBP-MoE-ViT experiment
python test/run_experiment.py --config test/configs/gbp_moe_vit.py

3. Integrating ΣPI

# (Inside the training/validation loop)

# Calculate loss
loss_epsilon = loss_fn(logits, target)

# Compute gradients
model.zero_grad()
loss_epsilon.backward()

# Calculate PI metrics
pi_metrics = pi_monitor.calculate(
    model=model,
    loss_epsilon=loss_epsilon,
    logits=logits
)

print(f"PI: {pi_metrics['pi_score']:.4f}, Surprise: {pi_metrics['surprise']:.4f}")

The returned pi_metrics dictionary contains:

• pi_score: The overall predictive integrity (0-1)
• surprise: Gradient norm indicating model adaptation
• normalized_error: Error scaled by model uncertainty
• cognitive_cost: Combined cost of error and surprise
• Additional component metrics for detailed analysis

Further Reading

PI is a concept derived from the Integrated Predictive Workspace Theory (IPWT), a computational theory of consciousness. To understand the deep theory behind this tool, please refer to https://github.com/dmf-archive/IPWT

Citation

If you wish to cite this work, please use the following BibTeX entry:

@misc{sigma_pi,
  author       = {Rui, L.},
  title        = {{ΣPI: Observe the Cognitive ability of Your AI Model}},
  year         = {2025},
  publisher    = {GitHub},
  url          = {https://github.com/dmf-archive/SigmaPI}
}

License

This project is licensed under the MIT License.