steklov-activations 0.1.0

Compact-support piecewise-polynomial activation functions with tunable sparsity

Steklov Activations

Compact-support piecewise-polynomial activation functions with tunable sparsity.

Paper: Steklov Activations: Piecewise-Polynomial Gates with Compact Support and Tunable Sparsity

What is this?

SteklovSiLU is a drop-in replacement for GELU/SiLU/ReLU that produces exact zeros outside a finite support region. One parameter α controls the support width:

• α = 2.0 → approximates GELU (sup error < 0.0091)
• α = 0.8 → 28–64% of activations are exactly zero per token
• α = 0.1 → 87% zeros, 98.4% natural NVIDIA 2:4 compliance
• α = 0.005 → 90% zeros, 99.2% compliance, quality matches SiLU

The exact zeros enable structural pruning (permanently remove dead neurons) and hardware-accelerated N:M sparse inference.

Install

pip install steklov-activations

# With Triton kernel support
pip install steklov-activations[triton]

Quick Start

Drop-in activation replacement

from steklov_activations import SteklovSiLU

# GELU-like (α=2.0, sup error < 0.0091)
act = SteklovSiLU(alpha=2.0)

# High sparsity
act = SteklovSiLU(alpha=0.1)

# Learnable α (network finds its own sparsity level)
act = SteklovSiLU(alpha=2.0, learnable=True)

Swap activations in an existing architecture

from transformers import AutoConfig, AutoModelForCausalLM
from steklov_activations import swap_activation

# Initialize model with random weights
config = AutoConfig.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_config(config)

# Replace all GELU activations with SteklovSiLU
swap_activation(model, alpha=0.8)
# Replaced 12 activation(s) with SteklovSiLU(alpha=0.8, order=3)

# Train from scratch (post-hoc swap on pretrained models
# does NOT produce meaningful sparsity)

Profile inactivity

from steklov_activations import InactivityProfiler

profiler = InactivityProfiler(model)

# Run calibration data through the model
for batch in calibration_loader:
    with torch.no_grad():
        model(**batch)

# Get report
report = profiler.report(threshold=0.999)
print(report.summary())
# Inactivity Report (50,000 tokens, threshold=99.9%)
#   Dynamic sparsity (per-token zeros): 64.4%
#   Structural sparsity (permanently dead): 9.0%
#   ...

profiler.remove_hooks()

Prune dead neurons

from steklov_activations import prune_inactive_neurons

# Remove permanently dead neurons from weight matrices
pruned = prune_inactive_neurons(model, report)
# Total: pruned 2,800 neurons across 12 layers

# Model is now physically smaller and faster
# Save the pruned model
model.save_pretrained("my-model-pruned")

Key Results (from paper)

Setting	Result
Vision (CIFAR-10/100, 5 seeds)	Steklov α=0.8 wins all 4 benchmarks
GPT-2 Small (124M, 5 seeds)	22.39 PPL vs GELU 22.97 (p=0.30)
GPT-2 Medium (354M, 3-4 seeds)	Matches GELU within seed variability
LLaMA 105M (3 seeds)	All variants beat SiLU
LLaMA 105M α=0.005 (1 seed)	90% zeros, PPL better than dense SiLU
Pruning 7% of 354M neurons	+0.05 PPL (magnitude pruning: model collapse)
N:M compliance at α≤0.1	98–99% natural 2:4; SiLU: 0%
Inference after pruning	3–5% faster than unpruned GELU

Important Notes

• Train from scratch. Post-hoc activation swap on pretrained models does not produce meaningful sparsity. The network must train with compact support to develop the inactivity patterns.
• Single α is sufficient. A single global α shared across all layers works as well as per-layer α.
• Architecture matters. GPT-2-style models produce permanently dead neurons (good for pruning). LLaMA-style models produce per-token zeros with no permanent death (good for N:M hardware sparsity).

Citation

@article{masalskikh2026steklov,
  author  = {Masalskikh, A.},
  title   = {Steklov Activations: Piecewise-Polynomial Gates with Compact Support and Tunable Sparsity},
  journal = {Zenodo},
  year    = {2026},
  doi     = {10.5281/zenodo.19454642},
  url     = {https://doi.org/10.5281/zenodo.19454642}
}

License

Apache 2.0