mobius-activations 0.4.0

A 3D activation function for PyTorch and TensorFlow with geometric properties.

Mobius Activations & Optimizer

A Python package for geometric deep learning in PyTorch and TensorFlow, featuring a 3D activation function and a geometrically-aware optimizer.

This package provides two primary components for deep learning models that need to handle geometric or rotational data properties. The core idea is to apply transformations inspired by the non-orientable topology of a Mobius strip to high-dimensional feature vectors.

Note: This is an experimental package created for research and educational purposes. The concepts are unconventional and should be considered proofs of concept. Use in production environments is not recommended without extensive testing.

Components

1. MobiusActivation: A torch.nn.Module and tf.keras.layers.Layer that applies a learnable, rotational transformation to 3D vector spaces.
2. MobiusOptimizer: A torch.optim.Optimizer and tf.keras.optimizers.Optimizer that navigates the loss landscape using an orbital component in addition to standard gradient descent.

Installation

The package can be installed with pip. Optional extras are available for the required deep learning framework and visualization dependencies.

# For PyTorch + Visualization tools
pip install mobius-activations[torch,viz]

# For TensorFlow + Visualization tools
pip install mobius-activations[tensorflow,viz]

1. The MobiusActivation Layer

The MobiusActivation layer contains one or more internal mobius_blocks. Each block applies a "Mobius twist"—a magnitude-preserving rotation—to a 3D feature vector.

Mathematical Formulation

ReMU: A Single, Pure Transformation

For a single ReMU block with input vector z, the transformation a = F(z) is calculated as follows:

1. The rotation angle θ is determined by the input's magnitude ||z|| and a learnable parameter k (twist tightness): θ = k * ||z||

2. The input vector z is rotated by θ around a given axis. For a Z-axis rotation, the transformation is a = R_z(θ)z, where R_z(θ) is the rotation matrix:

       [ cos(θ)  -sin(θ)  0 ]
   R_z(θ) = [ sin(θ)   cos(θ)  0 ]
       [   0       0      1 ]
   

S-ReMU: An Interference of Multiple Transformations

An S-ReMU calculates the final activation a_final as the weighted sum of multiple independent ReMU transformations (a_i), each with its own axis, twist tightness k_i, and learnable weight w_i:

a_final = Σ (w_i * a_i)

Operational Modes & Configuration

• projection Mode (Default): The layer contains a single mobius_block and requires an input of exactly 3 features.
• grouped Mode (Recommended): The layer contains multiple mobius_blocks and operates on high-dimensional inputs. It automatically processes features in parallel groups of 3. The in_features must be divisible by 3.
• Learnable Mode (Recommended): Set learnable=True. The layer will create and train its own k and w parameters.

Example (PyTorch, Learnable & Grouped S-ReMU)

import torch.nn as nn
from mobius_activations.torch import MobiusActivation

# Input has 12 features. This creates a layer with 4 parallel, 
# learnable S-ReMU blocks (each with 3 realities for x, y, z axes).
model = nn.Sequential(
    nn.Linear(50, 12),
    nn.BatchNorm1d(12),
    MobiusActivation(in_features=12, mode='grouped', learnable=True, axes=['x', 'y', 'z']),
    nn.Linear(12, 1)
)

Advanced Usage: Accessing Internal Parameters

For testing, visualization, or advanced analysis, you may need to access the learnable parameters (k_params, w_params) of a MobiusActivation layer. These parameters are encapsulated within the internal mobius_blocks.

Access is a multi-step process:

1. Get the main MobiusActivation layer from your model.
2. Access its .mobius_blocks attribute, which is a list (ModuleList in PyTorch).
3. Select the specific internal block you want to inspect by its index (e.g., [0]).
4. Access the parameters (.k_params, .w_params) from that specific block.

# PyTorch Example: Accessing parameters from a trained grouped layer

# Assume 'model' is the trained model from the example above
# 1. Get the main layer
mobius_layer = model[2]

# 2. Access the list of blocks
internal_blocks = mobius_layer.mobius_blocks
print(f"The layer has {len(internal_blocks)} internal blocks.")

# 3. Select the first internal block
first_block = internal_blocks[0]

# 4. Access the learnable parameters of that block
k_params_of_first_block = first_block.k_params
w_params_of_first_block = first_block.w_params

print(f"First k parameter of the first block: {k_params_of_first_block[0].item()}")

2. The MobiusOptimizer

This optimizer adds an orbital component to the standard momentum-based update, causing the optimization path to spiral. This is effective at escaping saddle points and finding wider minima.

Mathematical Formulation

A standard Momentum optimizer updates weights W at step t using the gradient g_t and momentum m_t:

1. m_t = β * m_{t-1} + (1-β) * g_t
2. W_{t+1} = W_t - η * m_t (where η is the learning rate)

The MobiusOptimizer modifies this update rule:

1. Descent Step: ΔW_descent = -η * m_t
2. Orbital Step: ΔW_orbital = -τ * (g_t ⟂ m_t)
   • τ (tau) is the new twist_rate hyperparameter.
   • The ⟂ operator finds a vector perpendicular to the gradient and momentum. In the optimizer's 3D groups, this is the cross-product.
3. Final Update: W_{t+1} = W_t + ΔW_descent + ΔW_orbital

PyTorch Usage

from mobius_activations.torch import MobiusOptimizer

# model = YourPytorchModel()
optimizer = MobiusOptimizer(model.parameters(), lr=1e-3, twist_rate=0.1)

TensorFlow / Keras Usage

from mobius_activations.tensorflow import MobiusOptimizer

# model = YourKerasModel()
optimizer = MobiusOptimizer(learning_rate=1e-3, twist_rate=0.1)
# model.compile(optimizer=optimizer, loss='binary_crossentropy')

3. The Visualization Suite

The package includes utilities to help interpret the geometric behavior of the components.

• visualize_optimizer_path: Shows the unique spiraling descent of the MobiusOptimizer.
• visualize_transformation_flow: Shows how a MobiusActivation layer warps the entire feature space.
• visualize_neuron_state: Provides analysis of the transformation at a single point.

from mobius_activations.visualize import visualize_optimizer_path

# Example of visualizing the optimizer's path
visualize_optimizer_path(start_point=[-4, 3.5], learning_rate=0.1, twist_rate=0.15)

4. Potential Applications

These components are best suited for problems with underlying geometric or periodic properties.

• Computer Vision: Learning rotational invariance for object recognition.
• Signal Processing: Modeling phase shifts and harmonic interference.
• Natural Language Processing: Modeling semantic relationships as rotations in embedding space.
• Physics & Robotics: Modeling systems with real-world rotational dynamics.

License

This project is licensed under the MIT License.