aptx-neuron 0.0.9

A PyTorch implementation of the APTx Neuron.

APTx Neuron

This repository provides a PyTorch implementation of the APTx Neuron, as introduced in the paper "APTx Neuron: A Unified Trainable Neuron Architecture Integrating Activation and Computation".

Paper Title: APTx Neuron: A Unified Trainable Neuron Architecture Integrating Activation and Computation

Author: Ravin Kumar

Sources:

• Arxiv.org
• Research Gate

Github Repositories:

• APTx Neuron (Pytorch + PyPI Package): APTx Neuron
• APTx Activation Function (Pytorch + PyPI Package): APTx Activation Function
• Experimentation Results with MNIST (APTx Neuron): MNIST Experimentation Code

Cite Paper as:

Kumar, Ravin. "APTx Neuron: A Unified Trainable Neuron Architecture Integrating Activation and Computation." arXiv preprint arXiv:2507.14270 (2025).

Or,

@article{kumar2025aptx,
  title={APTx Neuron: A Unified Trainable Neuron Architecture Integrating Activation and Computation},
  author={Kumar, Ravin},
  journal={arXiv preprint arXiv:2507.14270},
  year={2025}
}

---

APTx Neuron

Abstract: We propose the APTx Neuron, a novel, unified neural computation unit that integrates non-linear activation and linear transformation into a single trainable expression. The APTx Neuron is derived from the APTx activation function, thereby eliminating the need for separate activation layers and making the architecture both optimization-efficient and elegant. The proposed neuron follows the functional form y = Σ_{i=1..n} ((α_i + tanh(β_i * x_i)) * γ_i * x_i) + δ, where all parameters α_i, β_i, γ_i, and δ are trainable. We validate our APTx Neuron-based architecture on the MNIST dataset, achieving up to 96.69% test accuracy within 11 epochs using approximately 332K trainable parameters. The results highlight the superior expressiveness and training efficiency of the APTx Neuron compared to traditional neurons, pointing toward a new paradigm in unified neuron design and the architectures built upon it. Source code is available at https://github.com/mr-ravin/aptx_neuron.

The APTx Neuron is a novel computational unit that unifies linear transformation and non-linear activation into a single, expressive formulation. Inspired by the parametric APTx activation function, this neuron architecture removes the strict separation between computation and activation, allowing both to be learned as a cohesive entity. It is designed to enhance representational flexibility while reducing architectural redundancy.

Mathematical Formulation

Traditionally, a neuron computes the output as:

y = φ( Σ_{i=1..n} (w_i * x_i) + b )

where:

• x_i are the inputs,
• w_i are the weights,
• b is the bias,
• and φ is an activation function such as ReLU, Swish, or Mish.

The APTx Neuron merges these components into a unified trainable expression as:

y = Σ_{i=1..n} ((α_i + tanh(β_i * x_i)) * γ_i * x_i) + δ

where:

• x_i is the i-th input feature,
• α_i, β_i, and γ_i are trainable parameters for each input,
• δ is a trainable scalar bias.

This equation allows the neuron to modulate each input through a learned, per-dimension non-linearity and scaling operation. The term (α_i + tanh(β_i * x_i)) introduces adaptive gating, and γ_i * x_i provides multiplicative control.

---

📥 Installation

pip install aptx_neuron

or,

pip install git+https://github.com/mr-ravin/aptx_neuron.git

---

Usage

1. APTx Neuron-based Layer with all α_i, β_i, γ_i, and δ are trainable:

The setting is_alpha_trainable = True keeps α_i trainable. Each APTx neuron will have (3n + 1) trainable parameters, where n is input dimension. Note: The default value of is_alpha_trainable is True.

import aptx_neuron
import torch

input_dim = 8  # assuming input vector to be of dimension 8.
output_dim = 2 # assuming output dimension equals 2.

aptx_neuron_layer = aptx_neuron.aptx_layer(input_dim=input_dim, output_dim=output_dim, is_alpha_trainable=True)

batch_size = 3 # assuming batch size is 3
input_tensor = torch.ones((batch_size, input_dim)) # dimension: [3, 8]
output_tensor = aptx_neuron_layer(input_tensor)    # dimension: [3, 2]

2. APTx Neuron-based Layer with α_i = 1 (not trainable); While β_i, γ_i, and δ are trainable:

The setting is_alpha_trainable = False makes α_i fixed (non-trainable). Each APTx neuron will then have (2n + 1) trainable parameters, thus reducing memory and training time per epoch. Here, n is input dimension.

import aptx_neuron
import torch

input_dim = 8  # assuming input vector to be of dimension 8.
output_dim = 2 # assuming output dimension equals 2.

aptx_neuron_layer = aptx_neuron.aptx_layer(input_dim=input_dim, output_dim=output_dim, is_alpha_trainable=False)  # α_i is fixed (not trainable)

batch_size = 3 # assuming batch size is 3
input_tensor = torch.ones((batch_size, input_dim)) # dimension: [3, 8]
output_tensor = aptx_neuron_layer(input_tensor)    # dimension: [3, 2]

3. Single APTx Neuron with trainable α_i, β_i, γ_i, and δ:

The setting is_alpha_trainable = True keeps α_i trainable. Each APTx neuron will have (3n + 1) trainable parameters, where n is input dimension. Note: The default value of is_alpha_trainable is True.

import aptx_neuron
import torch

input_dim = 8  # assuming input vector to be of dimension 8.

aptx_neuron_unit = aptx_neuron.aptx_neuron(input_dim=input_dim, is_alpha_trainable=True)

batch_size = 3 # assuming batch size is 3
input_tensor = torch.ones((batch_size, input_dim)) # dimension: [3, 8]
output_tensor = aptx_neuron_unit(input_tensor)     # dimension: [3, 1]

4. Single APTx Neuron with fixed α_i = 1 and trainable β_i, γ_i, and δ:

The setting is_alpha_trainable = False makes α_i fixed (non-trainable). Each APTx neuron will then have (2n + 1) trainable parameters, thus reducing memory and training time per epoch. Here, n is input dimension.

import aptx_neuron
import torch

input_dim = 8  # assuming input vector to be of dimension 8.

aptx_neuron_unit = aptx_neuron.aptx_neuron(input_dim=input_dim, is_alpha_trainable=False)  # α_i is fixed (not trainable)

batch_size = 3 # assuming batch size is 3
input_tensor = torch.ones((batch_size, input_dim)) # dimension: [3, 8]
output_tensor = aptx_neuron_unit(input_tensor)    # dimension: [3, 1]

For a layer with output_dim = m, the total number of trainable parameters is m(3n + 1) when α_i is trainable, or m(2n + 1) when α_i = 1 is fixed. Here, n is the input dimension (i.e., input_dim) and m is the number of neurons in the layer (i.e., output_dim).

Important:

If δ (i.e., bias) is not required, it can be disabled by setting use_delta=False. By default, use_delta is set to True. We can disable δ in both aptx neuron unit and aptx layer as: aptx_neuron(input_dim=input_dim, use_delta=False) and aptx_layer(input_dim=input_dim, output_dim=output_dim, use_delta=False) respectively.

---

Working Demonstration

• Example: APTx Neuron Unit

import torch
import torch.nn as nn
from aptx_neuron import aptx_neuron

# Create single neuron
model = aptx_neuron(input_dim=4)

# Input (batch_size=3, input_dim=4)
x = torch.tensor([
    [1., 2., 3., 4.],
    [5., 6., 7., 8.],
    [9., 10., 11., 12.]
])

# Target output (batch_size=3, 1)
target = torch.tensor([
    [1.0],
    [2.0],
    [3.0]
])

# Forward pass
y = model(x) # shape: [3, 1]

# Compute loss
loss_fn = nn.MSELoss()
loss = loss_fn(y, target)

# Backward pass
loss.backward()

print("Output:")
print(y)

print("\nLoss:", loss.item())

if model.alpha.grad is not None:
    print("\nGradient alpha:")
    print(model.alpha.grad)
else:
    print("\nAlpha is not trainable (no gradient).")

print("\nGradient beta:")
print(model.beta.grad)

print("\nGradient gamma:")
print(model.gamma.grad)

print("\nGradient delta:")
print(model.delta.grad)

• Example: APTx Neuron Layer

import torch
import torch.nn as nn
from aptx_neuron import aptx_layer

# Create APTx layer
model = aptx_layer(input_dim=4, output_dim=2)

# Input (batch_size=3, input_dim=4)
x = torch.tensor([
    [1., 2., 3., 4.],
    [5., 6., 7., 8.],
    [9., 10., 11., 12.]
])

# Target output (batch_size=3, output_dim=2)
target = torch.tensor([
    [1., 2.],
    [3., 4.],
    [5., 6.]
])

# Forward pass
y = model(x) # shape: [3, 2]

# Loss
loss_fn = nn.MSELoss()
loss = loss_fn(y, target)

# Backward
loss.backward()

print("Output:")
print(y)

print("\nLoss:", loss.item())

if model.alpha.grad is not None:
    print("\nGradient alpha shape:", model.alpha.grad.shape)
else:
    print("\nAlpha is not trainable (no gradient).")

print("\nGradient beta shape:", model.beta.grad.shape)
print("\nGradient gamma shape:", model.gamma.grad.shape)
print("\nGradient delta shape:", model.delta.grad.shape)

---

Conclusion

This work introduced the APTx Neuron, a unified, fully trainable neural unit that integrates linear transformation and non-linear activation into a single expression, extending the APTx activation function. By learning per-input parameters α_i, β_i, and γ_i for each input x_i, and a shared bias term δ within a neuron, the APTx Neuron removes the need for separate activation layers and enables fine-grained input transformation. The APTx Neuron generalizes traditional neurons and activations, offering greater representational power. Our experiments show that a fully connected APTx Neuron-based feedforward neural network achieves 96.69% test accuracy on the MNIST dataset within 11 epochs using approximately 332K trainable parameters, demonstrating rapid convergence and high optimization efficiency. This design lays the groundwork for extending APTx Neurons to CNNs and transformers, paving the way for more compact and adaptive deep learning architectures.

---

📜 Copyright License

Copyright (c) 2025 Ravin Kumar
Website: https://mr-ravin.github.io

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 
files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, 
modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 
Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the 
Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.