ghonn-models-pytorch 0.2.0

Python library with polynomial neural networks

GHONN Models Pytorch brings advanced neural architectures to your PyTorch projects: Higher Order Neural Units (HONU), Higher Order Neural Networks (HONN), Gated Higher Order Neural Units (GHONU), Gated Higher Order Neural Networks (GHONN), and the Self-Attention Gated Higher Order Neural Network (SA-GHONN).

✨ Polynomial neurons at the core: These models excel at capturing complex, nonlinear relationships—especially when working with polynomial signals. Their adaptable design makes them a strong choice for a wide range of machine learning tasks.

🔗 Gated variants for extra power: The gated architectures use a dual HONU neuron setup—one as a dynamic gate, the other as the main predictor—enabling richer and more expressive modeling.

🧠 Full forecasting network: SA-GHONN normalizes the input (optionally with RevIN), maps each timestep through a HONU/GHONU embedding, adds positional encoding, and passes the sequence through a Transformer encoder with self-attention. Its forecast head is configurable: flat uses a linear projection of the full encoded sequence, pool mean-pools the sequence and applies an MLP, and query uses learned forecast queries with cross-attention over the self-attended sequence before a final linear projection. Thus, self-attention is always used, while cross-attention is specific to the query head.

🛠️ Modular and flexible: Build your own architectures with ease. Layers can be stacked directly or connected via linear mappings, giving you full control over your network’s structure.

👉 Curious how it works in practice? Check out the example notebooks and usage guides included in this repository.

📖 Project Documentation 📖

Visit Read The Docs Project Page or read the following README to know more about Gated Higher Order Neural Network Models Pytorch (GHONN for short) library.

✨ Features

• Polynomial neurons: Capture complex, nonlinear relationships using higher-order neural units.
• Gated architectures: Leverage dual-neuron setups for richer modeling capacity.
• SA-GHONN: Use a complete self-attention forecasting network built around HONU/GHONU components.
• Conv-GHONN: Apply GHONU polynomial filters to causal temporal windows for sequence feature extraction.
• Modular design: Easily stack and combine layers for custom architectures.
• Efficient computation: Optimized for high-order polynomial calculations, even on CPUs.
• Seamless PyTorch integration: All components are standard PyTorch modules.
• Supports regression & classification: Flexible for a wide range of ML tasks.
• Ready-to-use examples: Example notebooks and guides included.

Neuron Types ⚡

• HONU: The fundamental building block for higher-order modeling. For example, a 2nd order HONU is defined as:

  

  where:

  • $\tilde{y}(k)$ is the neuron output for input sample $k$
  • $w_{i,j}$ are the learnable weights
  • $x_i, x_j$ are input features
  • $\mathbf{w}$ is the weight vector
  • $\mathrm{col}^{r=2}(\mathbf{x})$ is the column vector of all 2nd order combinations of input features
  • $r$ is the polynomial order

  This structure ensures polynomial relationships between input datapoints and high computation performance.

• gHONU: Combines two HONUs—one as a predictor (typically linear activation), the other as a dynamic gate (e.g., tanh)—multiplying their outputs for enhanced ability to capture complex patterns.

Network Layers 🧩

• HONN: Single-layer networks of HONU neurons. Supports both raw outputs for stacking and linear heads for custom output dimensions.
• gHONN: Single-layer networks of gHONU neurons, with the same flexible output options as HONN.

Full Models and Components 🧠

• SA-GHONN: Self-attention Gated Higher-Order Neural Network forecaster. The pipeline is RevIN (optional) -> HONU/GHONU embedding (GhonuBank or temporal ConvGhonn) -> positional encoding -> Transformer encoder self-attention -> forecast head. The flat head uses a linear projection, the pool head uses mean pooling followed by an MLP, and the query head uses learned forecast queries to cross-attend to the self-attended sequence before a linear projection.
• HonuBank and GhonuBank: Grouped, vectorized banks of HONU and GHONU neurons.
• ConvGhonn (Conv-GHONN): Applies GHONU polynomial filters to causal temporal windows and can be used directly or as the SA-GHONN embedding when temporal_lookback is greater than zero.
• RevIN: Reversible instance normalization for non-stationary time series.

Why Choose GHONN Models? 🚀

• Efficient high-order computation: Optimized for fast polynomial calculations, even at high orders and on CPUs.
• Flexible & modular: Easily stack, combine, or adapt layers and neurons for custom architectures.
• PyTorch-native: All components are standard PyTorch modules for seamless integration.
• Versatile: Supports both regression and classification tasks.
• Quick start: Example notebooks and guides included to help you get going fast.

🧪 Examples & Usage

You can find helpful, step-by-step Jupyter notebooks in the examples folder, which offer practical demonstrations and implementation suggestions.

You may also find the code snippets below useful as a starting point.

HONU initialization

import ghonn_models_pytorch as gmp

kwargs = {
    "weight_divisor": 100,  # Divides weights to help with numerical stability
    "bias": True            # Whether to use a bias term in the model
}

# Create a Higher Order Neural Unit (HONU) with 3 inputs and degree 2
honu_neuron = gmp.HONU(
    in_features=3,          # Number of input features
    order=2,                # Degree of the polynomial
    activation="identity",  # Activation function
    **kwargs
)

HONN initialization

import ghonn_models_pytorch as gmp

kwargs = {
    "weight_divisor": 100,
    "bias": True
}

# Create single HONU based layer - HONN with 4 neurons of different orders and activation functions.
honn_layer = gmp.HONN(
    in_features=3,                          # Number of input features
    out_features=2,                         # Number of output features
    layer_size=4,                           # Number of neurons in the layer
    polynomial_orders=(2, 3),               # Degree of the polynomials; cycles if shorter than layer_size
    activations=("identity", "sigmoid"),    # Activation functions for the neurons in the layer. If shorter work like a rolling buffer
    output_type="linear",                   # Output type of the layer. Can be "linear" or "sum" or "raw"
    **kwargs
)

Neuron, Layer or Model training as usual

for i in range(0, data.size(0), batch_size):
    # Get the batch
    batch = data[i:i+batch_size]
    # Forward pass
    output = honn_layer(batch)
    # Compute loss
    loss = criterion(output, target)
    # Backward pass
    loss.backward()
    # Update weights
    optimizer.step()

💡 Tips & Tricks

Default SA-GHONN initialization

import torch
import ghonn_models_pytorch as gmp

# Forecast 12 steps from a 24-step, 3-feature input sequence.
sa_ghonn = gmp.SAGHONN(
    input_length=24,
    in_features=3,
    out_features=12,
)

sequence = torch.randn(8, 24, 3)
forecast = sa_ghonn(sequence)
# forecast.shape == (8, 12, 1)

SA-GHONN uses a GHONU bank as its default sequence embedding. It then adds positional encoding, processes the sequence with a Transformer encoder, and applies the selected forecasting head. Set temporal_lookback to use a temporal ConvGhonn embedding, or set head_type="query" to use learned forecast queries with cross-attention. Use predicted_feature_indices and aux_all_features when forecasting multiple input features.

• In the case of GHONU based units it is often benefitial to have different initial learning rate between the two neurons.
• more TBD

🛠️ Installation

PyPI version:

pip install ghonn-models-pytorch

The latest version from GitHub:

pip install git+https://github.com/carnosi/ghonn_models_pytorch

Compatibility notice

The verion 0.2+ API uses vectorized PyTorch modules and does not guarantee backward compatibility with earlier releases. HONU accepts inputs shaped (..., in_features) and returns (..., out_features). A GHONU with gate_order=0 omits its gate and behaves as a HONU. Gates can be frozen with standard PyTorch controls:

ghonu.gate.requires_grad_(False)

📚 References

This repository is inspired by the foundational research presented in the following papers. While the original studies utilized legacy implementations, this PyTorch-based version offers a more user-friendly and computationally efficient alternative, maintaining the same core objectives and functionality.

HONU:

[1] P. M. Benes and I. Bukovsky, “Railway Wheelset Active Control and Stability via Higher Order Neural Units,” IEEE/ASME Transactions on Mechatronics, vol. 28, no. 5, pp. 2964–2975, Oct. 2023, doi: 10.1109/TMECH.2023.3258909.

[2] I. Bukovsky, G. Dohnal, P. M. Benes, K. Ichiji, and N. Homma, “Letter on Convergence of In-Parameter-Linear Nonlinear Neural Architectures With Gradient Learnings,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 8, pp. 5189–5192, Aug. 2023, doi: 10.1109/TNNLS.2021.3123533.

[3] I. Bukovsky, “Deterministic behavior of temperature field in turboprop engine via shallow neural networks,” Neural Comput & Applic, vol. 33, no. 19, pp. 13145–13161, Oct. 2021, doi: 10.1007/s00521-021-06013-7.

[4] P. M. Benes, I. Bukovsky, M. Vesely, J. Voracek, K. Ichiji, and N. Homma, “Framework for Discrete-Time Model Reference Adaptive Control of Weakly Nonlinear Systems with HONUs,” in Computational Intelligence, C. Sabourin, J. J. Merelo, K. Madani, and K. Warwick, Eds., Cham: Springer International Publishing, 2019, pp. 239–262. doi: 10.1007/978-3-030-16469-0_13.

GHONU:

[1] O. Budik, I. Bukovsky, and N. Homma, “Potentials of Gated Higher Order Neural Units for Signal Decomposition and Process Monitoring,” Procedia Computer Science, vol. 253, pp. 2278–2287, Jan. 2025, doi: 10.1016/j.procs.2025.01.288.

Our other project

AISLEX: A Python package for Approximate Individual Sample Learning Entropy (LE) anomaly detection. Easily integrate LE-based novelty detection into your neural network workflows, with both Python and JAX implementations.

📝 How To Cite

If ghonn_models_pytorch has been useful in your research or work, please consider citing our article:

Work in progress. Use GHONU (10.1016/j.procs.2025.01.288) for now please.

BibText:

Work in progress. Use GHONU (10.1016/j.procs.2025.01.288) for now please.

📄 License

This project is licensed under the terms of the MIT License.