alrelu-torch 0.1.0

ALReLU activation package for PyTorch with fixed and trainable alpha variants.

alrelu-torch

PyPI-installable PyTorch package for the ALReLU activation with 2 variants:

1. ALReLU (default): fixed alpha=0.01
2. TrainableALReLU: trainable alpha parameter

Formula:

ALReLU(x, alpha) = max(abs(alpha * x), x)

Installation

After publishing:

pip install alrelu-torch

From source:

pip install .

Usage

1) Functional API

import torch
from alrelu_torch import alrelu

x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
y = alrelu(x)  # alpha=0.01 by default

2) Fixed module

import torch.nn as nn
from alrelu_torch import ALReLU

model = nn.Sequential(
    nn.Linear(32, 64),
    ALReLU(alpha=0.01),
    nn.Linear(64, 10),
)

3) Trainable alpha module

import torch.nn as nn
from alrelu_torch import TrainableALReLU

model = nn.Sequential(
    nn.Linear(32, 64),
    TrainableALReLU(alpha_init=0.01, non_negative=True),
    nn.Linear(64, 10),
)

Development

Install dev tools:

pip install -e .[dev]

Run tests:

pytest

Build package:

python -m build