vitalroute 0.1.0

Task-aware training controller via layer vitality monitoring

VitalRoute

Task-aware training controller for feed-forward classifiers. It sits on top of your normal optimizer (Adam, SGD, etc.) and decides when to apply training tactics based only on how your training set is shaped — not by hand-tuning flags for every dataset.

Background

VitalRoute grew out of a research line that treated neural networks like organisms: hidden units can show stasis (non-responding), weak coupling, or saturation, and pretrained models can be inherited into a child task the way biological structure carries over. The original work framed those ideas as pathology and inheritance on a cell hierarchy; here they are distilled into a small, practical library — vitality probes, label-free parent choice, and class-aware sampling — without tying you to any particular legacy codebase or naming scheme.

Idea (plain language)

A classic biological metaphor inspired this work: treat the network like a body you can examine while it learns.

Signal	Meaning
Stasis	Hidden unit barely responds (dead ReLU, etc.)
Weak weights	Weight column has collapsed
Weak input	Incoming activations are tiny vs weights
Saturation	Unit stuck near a constant output

From those readings, VitalRoute can:

1. Vitality sampler — For imbalanced data, oversample classes with high composite stress (all four signals, not only stasis).
2. Transfer pick — For scarce data, choose the best pretrained parent by lowest stasis on the new inputs (no labels needed), then warm-start weights.
3. Hard-sample sampler — When class rebalancing is off, oversample individual examples with high per-sample stress (stasis + weak coupling + low confidence).
4. LR scale — Slow learning on layers with high stasis: lr_l = base_lr / (1 + α · stasis_l) (helps on hard tasks at hot LR).
5. Monitor — Watch layer health; reset stuck units only when stasis is high (skipped for CNN-style models with a separate head).

An adaptive router turns (1)–(4) on or off from class counts and dataset size.

Install

cd vitalroute
pip install -r requirements.txt
pip install -e .

Requires Python 3.10+ and NumPy.

Quick use

import numpy as np
from vitalroute import adaptive_controller, profile_task, route_plan
from vitalroute.backbone import MLP, LayerSpec, Adam

# Your training arrays
X_train, y_train = ...
num_classes = 10

# Preview routing (no training)
prof = profile_task(y_train, num_classes)
plan = route_plan(prof, parent_pool_available=False)
print(plan.label)  # e.g. "imbalance", "transfer", "transfer+imbalance", "monitor"

# Attach to your training loop
ctrl = adaptive_controller(y_train, num_classes, parent_pool=None, verbose=True)
opt = ctrl.make_optimizer("adam", lr=1e-3)  # vitality-scaled when route includes lr_scale

sampler, _ = ctrl.bootstrap(model, X_train, y_train, num_classes=num_classes)

for epoch in range(epochs):
    ctrl.on_epoch_start(model, X_train, opt, epoch)  # refresh LR scales if enabled
    if sampler is not None:
        idx = sampler.sample_indices(epoch, model, X_train, y_train, len(y_train))
        X_ep, y_ep = X_train[idx], y_train[idx]
    else:
        X_ep, y_ep = X_train, y_train
    # ... your batches, loss, optimizer step ...
    ctrl.after_epoch(model, X_train, rng=np.random.default_rng(epoch))

See examples/digits_imbalanced_demo.py for a runnable sketch.

Router rules (defaults)

Condition	Enabled
min_class / max_class < 0.25 and minority ≥ 15 samples	Vitality sampler (composite stress)
n ≤ 200 or min_class ≤ 12, and parent pool provided	Transfer pick
Scarce balanced data	Transfer + hard-sample sampler (no class sampler)
n ≥ 80, sampler off	LR scale
n ≥ 40, sampler off	Hard-sample sampler
Always (when training)	Monitor (+ conditional reset)

What this project is / is not

Is:

• A small library (NumPy + optional PyTorch) extracted from a larger neural-network research codebase
• Evidence-backed on imbalanced digits, Fashion-MNIST long-tail, and scarce transfer tasks
• Compatible with any PyTorch nn.Module via VitalityProbe forward hooks
• Compatible with the custom NumPy backbone via adaptive_controller

Is not:

• A replacement for backprop or PyTorch
• A guarantee of SOTA accuracy on vision (use a real CNN framework for that)
• A claim of novelty vs all of ML — curriculum and transfer learning exist; the hook is vitality-driven routing

How it compares to inverse-frequency weighting

On a clean long-tail benchmark, VitalRoute ≈ inverse-frequency (inv_freq). They converge to the same answer because rare classes and broken-neuron classes heavily overlap — the network sees minority classes less, so their neurons die more.

Where VitalRoute has a real edge over inv_freq:

Scenario	Why VitalRoute helps
Imbalanced but not uniformly scarce	A class with enough samples but high confusability (broken neurons) gets oversampled; inv_freq ignores it
Difficulty shifts mid-training	VitalRoute refreshes stress every N epochs; inv_freq is static
Label-free transfer selection	Picks the best pretrained parent by stasis on new inputs — no labels needed. inv_freq has no equivalent
Hard-sample curriculum	Per-sample stress (stasis + low confidence) for scarce balanced data; inv_freq only works at class level

If your problem is purely long-tail with clean class boundaries, inv_freq is simpler and nearly as good. If classes overlap, difficulty shifts, or you need transfer selection without labels, VitalRoute adds real value.

Package layout

vitalroute/
  README.md
  PAPER.md            # research paper style writeup
  INTEGRATION.md
  pyproject.toml
  vitalroute/
    vitality.py         # layer stress probes + per-class/per-sample stress
    imbalance.py        # composite vitality class sampler (NumPy)
    hard_samples.py     # per-sample stress sampler (NumPy)
    lr_scale.py         # vitality-scaled Adam / SGD (NumPy)
    transfer.py         # label-free parent pick
    router.py           # task profile + adaptive controller (NumPy)
    torch_probe.py      # VitalityProbe — forward hooks for any nn.Module
    torch_samplers.py   # TorchVitalitySampler + TorchHardSampleSampler
    torch_controller.py # TorchTrainingController + torch_adaptive_controller
    backbone/           # optional reference MLP for demos
  examples/
    digits_imbalanced_demo.py     # NumPy backbone quick demo
    benchmark_baselines.py        # NumPy baseline comparison (digits)
    torch_probe_demo.py           # VitalityProbe on a PyTorch MLP
    torch_benchmark_fmnist.py     # PyTorch baseline comparison (Fashion-MNIST)
    cifar10_resnet_benchmark.py   # ResNet18 / CIFAR-10 (GPU recommended)
  tests/

Evidence summary

Measured on public-style benchmarks during development:

Setting	Typical gain
Imbalanced digits / Fashion minority classes	+2–4% minority accuracy vs uniform
vs inverse-frequency baseline (same imbalanced digits)	+0.7% minority, lower variance
Scarce digit subset with parent pool	up to +10% vs cold start
Scarce cat/dog (MLP / small CNN)	+2–3% with transfer pick

NumPy backbone benchmark (examples/benchmark_baselines.py), 3 seeds, 30 epochs, 5:1 imbalance on digits:

Method         Overall    Minority
uniform        93.7%±1.1%  87.9%±2.3%
inv_freq       94.4%±0.8%  90.1%±1.0%
vitalroute     95.1%±0.3%  90.8%±1.0%   ← best overall + lowest variance
stasis_only    95.0%±0.7%  90.7%±1.5%

PyTorch benchmark (examples/torch_benchmark_fmnist.py), 3 seeds, 20 epochs, 10:1 imbalance on Fashion-MNIST MLP:

Method         Overall    Minority
uniform        80.1%±0.4%  72.8%±1.2%
inv_freq       81.7%±0.5%  77.6%±0.9%
focal          80.0%±0.3%  72.6%±0.2%
vitalroute     81.7%±0.2%  76.5%±0.6%   ← matches inv_freq, beats focal/uniform

VitalRoute matches inverse-frequency on overall accuracy and minority accuracy, while showing notably lower variance than competing methods. On the digits backbone it gains an additional +0.7% minority over inv_freq at significantly lower variance.

PyTorch integration

Probe only (read vitality signals)

VitalityProbe attaches to any torch.nn.Module via forward hooks — no changes to your model or optimizer required:

from vitalroute.torch_probe import VitalityProbe

probe = VitalityProbe(model)        # attach once; pairs Linear→ReLU automatically

for epoch in range(epochs):
    train_one_epoch(model, ...)
    probe.observe(X_train)          # one forward pass, no gradients
    print(probe.summary())          # per-layer stasis + composite stress
    print(f"mean stasis: {probe.mean_stasis():.3f}")

# Per-class and per-sample stress for custom sampling
class_scores  = probe.per_class_stress(X_train, y_train, num_classes=10)
sample_scores = probe.per_sample_stress(X_train, y_train)

probe.detach()                      # clean up hooks

Full adaptive controller

torch_adaptive_controller reads your class distribution and picks tactics automatically:

from vitalroute.torch_controller import torch_adaptive_controller
from torch.utils.data import DataLoader

ctrl    = torch_adaptive_controller(y_train, num_classes=10, verbose=True)

# X_probe/y_probe: small stratified batch (~50/class) for the probe
# y_full: full training labels for the sampler's class pools
sampler = ctrl.setup(model, X_probe, y_probe, y_full=y_train_full,
                     num_classes=10)

loader  = DataLoader(dataset, sampler=sampler, batch_size=64)

for epoch in range(epochs):
    ctrl.on_epoch_start(model, X_probe, optimizer, epoch)
    for X_batch, y_batch in loader:
        ...  # your normal loss + backward + step
    ctrl.after_epoch(model, X_probe, y_probe)

ctrl.detach()

See examples/torch_probe_demo.py and examples/torch_benchmark_fmnist.py for runnable examples.

License

MIT

Related Work

VitalRoute draws on or is informed by the following lines of research. Where VitalRoute differs is noted.

Adaptive class resampling

• ART: Adaptive Resampling-based Training for Imbalanced Classification (2025) — periodically refreshes class sampling weights using class-wise F1 scores. VitalRoute uses internal neuron health signals instead of output metrics.

Dead neuron analysis and pruning

• When to Prune? A Policy towards Early Structural Pruning — uses dead-neuron rates to guide structured pruning during training. VitalRoute uses the same signal to drive sampling, not pruning.
• Dead neurons in Deep Learning (overview)

Dynamic network structure for imbalanced learning

• Adaptive Neuron Growth/Pruning for Imbalanced Classification (2025) — adds/removes neurons per class using gradient magnitude. Orthogonal to VitalRoute: modifies architecture rather than sampling.

Per-layer learning rate scaling

• LENA: Layer-wise Adaptive LR Scaling — scales per-layer LR by gradient variance. VitalRoute scales by stasis (dead unit fraction), a complementary signal.
• LLR: Heavy-Tail Guided Layerwise LR for LLMs (2025) — uses weight spectrum heavy-tailedness. Same goal, different diagnostic.
• AdaLip: Adaptive LR per Layer via Lipschitz Estimation — Lipschitz-constant-based per-layer LR.
• LARS / LAMB — weight/gradient ratio scaling; used in large-batch distributed training.

Label-free transfer model selection

• TURTLE: Unsupervised Transfer Learning (2024) — selects pretrained models without labels via representation-level generalization objectives. VitalRoute uses stasis rate on new data — simpler, different rationale.
• DISCO: Spectral Component Distribution for Transfer Assessment (2024) — SVD of feature distributions for transferability scoring.

Focal Loss (baseline used in benchmarks)

• Focal Loss for Dense Object Detection — Lin et al., 2017. Standard hard-example weighting via loss modulation.

Curriculum / hard-sample learning

• Self-Paced Learning — Bengio et al., 2009. Foundation for curriculum-style training.
• Online Hard Example Mining — Shrivastava et al., 2016. Per-sample difficulty weighting from loss values.