rani 1.0.1

RANI: Rational Adaptive Normalizing Initialization — a novel self-normalizing activation function with learnable parameters

RANI — Rational Adaptive Normalizing Initialization

A novel self-normalizing activation function with learnable per-layer parameters.

Author: Nikki Rani

---

What is RANI?

RANI is a new activation function that combines three properties that no existing activation function has simultaneously:

Property	RANI	SELU	GELU	Swish	PReLU
Self-normalizing (no BatchNorm needed)	✓	✓	✗	✗	✗
Learnable per-layer parameters	✓	✗	✗	✗	✓
Zero transcendentals on positive branch	✓	✓	✗	✗	✓

---

Formula

RANI(x) = λ · x · (1 + α·|x|) / (1 + |x|)    for x ≥ 0   ← rational, no exp()
RANI(x) = λ · α · (eˣ − 1)                     for x < 0   ← exponential

Default parameters — derived from self-normalizing fixed-point equations:

Parameter	Symbol	Default value	Role
Asymmetry	α	1.6733	Controls negative saturation depth and positive slope
Scale	λ	1.0507	Controls overall function amplitude

Both parameters are learned per layer during training via backpropagation.

---

Install

pip install rani

---

Quick Start — PyTorch

import torch.nn as nn
from rani import RANI, init_rani_network

# Build a network with RANI activations
model = nn.Sequential(
    nn.Linear(784, 512),
    RANI(),                    # alpha=1.6733, lam=1.0507 — both learnable
    nn.Linear(512, 256),
    RANI(),
    nn.Linear(256, 128),
    RANI(),
    nn.Linear(128, 10),
)

# Apply the mathematically correct weight initialization for RANI
# (LeCun normal: std = 1/sqrt(fan_in) — derived from the fixed-point equations)
init_rani_network(model)

# Train normally with any optimizer
import torch
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

Functional version with fixed parameters:

from rani import rani
import torch

x = torch.randn(32, 256)
y = rani(x)                    # alpha=1.6733, lam=1.0507 fixed

Check what the network learned after training:

for name, module in model.named_modules():
    if isinstance(module, RANI):
        print(f"{name}: alpha={module.alpha.item():.4f}, lam={module.lam.item():.4f}")
# Each layer learns its own optimal alpha and lambda values

---

Why does it work without BatchNorm?

RANI is mathematically self-normalizing. If the input to a layer has mean 0 and variance 1, the output also has mean 0 and variance 1 — regardless of how many layers deep you go.

This is proved via the fixed-point equations:

E[ RANI(Z) ]   = 0    ← mean is preserved as zero
E[ RANI(Z)² ]  = 1    ← variance is preserved as one

where Z ~ N(0, 1). The initialization values α = 1.6733, λ = 1.0507 are the exact numerical solution to this system.

After initialization, α and λ are free to be learned — each layer adapts to its own optimal values. This is the key advantage over SELU, where these values are permanently fixed.

---

FLOP Count — Why the Positive Branch Is Cheap

The positive branch uses only:

Operation	Count
Multiply	3
Add	2
Divide	1
Transcendentals (exp, erf, sigmoid)	0

Compare with GELU (needs erf), Swish (needs exp via sigmoid), and Mish (needs 2 × exp + tanh). RANI's positive branch is faster on both CPU and GPU for this reason.

---

Running the Tests

cd rani-activation
pytest tests/test_torch.py -v

Expected output: 14 passed

---

Mathematical Foundation

The complete derivation covers 6 parts: formal definition, regularity proofs, self-normalizing fixed-point equations, stability analysis via the Banach contraction theorem, and FLOP count comparison.

Key results:

Result	Description
Theorem 3.1	RANI is continuous on all of ℝ
Theorem 3.3	RANI is Lipschitz with L = λ·max(1, α)
Theorem 3.4	RANI saturates at −λα as x → −∞
Result 4.1	α* = 1.6733, λ* = 1.0507 (self-normalizing init)
Theorem 5.1	Fixed point (0, 1) is stable attractor, ρ(J) < 1

Full mathematical details and paper citation coming soon.

---

Package Structure

rani/
├── __init__.py         ← package entry point
├── torch_rani.py       ← PyTorch implementation
└── tf_rani.py          ← TensorFlow implementation (pending Python 3.14 support)

---

Citation

Citation details will be added after paper publication.

---

License

Apache License 2.0 — see LICENSE for details.

Author

Nikki Rani GitHub | PyPI