nonlinear 0.0.11

A hand-curated collections of activations functions for deep learning research.

Non-Linear

A hand-curated collections of activations functions for deep learning research.

Channel-Based Activation Functions

Activation Function	Formula
FPAF	$f(x) = a \cdot \mu(x)$ if $x \geq 0$, else $b \cdot \nu(x)$
DPReLU	$f(x) = ax$ if $x \geq 0$, else $bx$
DualLine	$f(x) = ax + m$ if $x \geq 0$, else $bx + m$
FReLU	$f(x) = \text{ReLU}(x) + b$
LeLeLU	$f(x) = ax$ if $x \geq 0$, else $0.01ax$
PREU	$f(x) = ax$ if $x \geq 0$, else $ax \exp(bx)$
ShiLU	$f(x) = a \cdot \text{ReLU}(x) + b$
StarReLU	$f(x) = a \cdot \text{ReLU}(x)^2 + b$
EPReLU	$f(x) = kx$ if $x \geq 0$, else $\frac{x}{a}$ (where $k$ updates dynamically)
PairedReLU	$f(x) = \left[\text{ReLU}(ax - b), \text{ReLU}(cx - d)\right]$ (concatenated along channel dimension)
RMAF	$f(x) = \frac{abx}{(0.25(1 + \exp(-x)) + 0.75)^c}$
PTELU	$f(x) = x$ if $x \geq 0$, else $\vert a \vert \tanh(\vert b \vert x)$
TaLU	$f(x) = x$ if $x \geq \vert b \vert$; $\tanh(x)$ if $x > \vert a \vert$; else $\tanh(\vert a \vert)$
TanhLU	$f(x) = a \tanh(cx) + bx$