Zeta Scaling Toolkit
Experimental PyTorch utilities for attention, learning-rate schedules, coordinates, and linear initialization. Numerical behavior is covered by the linked tests; none of these components is validated as a training improvement on large-scale tasks.
Components
NonCollapsingAttentioncomputessoftmax(tau * cos(q, k)) @ v, wheretauis learnable, starts attau0(default 10), and is bounded to[tau_min, tau_max](defaults 1 and 50) using a logit-sigmoid parameterization. This parameterization ensures non-zero gradients throughout the bounded range, allowing the temperature parameter to recover if perturbed beyond the bounds. Its normalization and softmax calculations use at least float32, and the result is cast tov.dtype. Boolean masks useTrueto retain a position; floating masks are added to the logits. Fully masked query rows return zeros. Shapes, dtypes, devices, and optional finite-value checks are validated; an empty key sequence raisesValueError, while empty query sequences and batches are supported. See attention property tests, edge-case tests, and regression tests.AsymptoticLRScheduleris the retained public class name for a schedule that linearly warms up, follows cosine decay tofloor_ratio * base_lr, then stays at that floor.floor_ratiomust be in[0, 1],total_steps >= 1, and0 <= warmup_steps < total_steps. Without warmup it is mathematically equivalent toCosineAnnealingLRwith the same floor inside the configured horizon, up to floating-point rounding; unlike stock cosine it stays clamped to the floor beyond that horizon. See scheduler property and resume tests.get_isotropic_coordsreturns pixel-centred(x, y)coordinates normalized bymax(height, width).get_scale_invariant_vit_coordsreturns(radius, angle)or, withsmooth_angle=True,(radius, cos(angle), sin(angle)). See the coordinate tests and compatibility tests.make_corrected_linearapplies orthogonal initialization scaled by a gain supported bytorch.nn.init.calculate_gain; its default is ReLU gain. GELU and SiLU are not supported as fixed gains. Usecalibrate_gainwith representative sample data for sequential GELU/SiLU stacks. See initialization properties and calibration regressions.
Deprecated A_attn, T, and A_coords arguments remain accepted, warn when
non-default values are supplied, and have no effect. Legacy attention
checkpoints containing q_scale and k_scale load with a warning and retain
the new module's initialized temperature; see
compatibility tests.
Installation
pip install -e .
For development and tests:
pip install -e ".[dev]"
Quick start
import torch
from zeta_scaling_toolkit import (
AsymptoticLRScheduler,
NonCollapsingAttention,
get_isotropic_coords,
make_corrected_linear,
)
attention = NonCollapsingAttention(d_k=64)
q, k, v = (torch.randn(2, 128, 64) for _ in range(3))
output = attention(q, k, v)
linear = make_corrected_linear(1024, 4096, nonlinearity="relu")
optimizer = torch.optim.Adam(linear.parameters(), lr=1e-3)
scheduler = AsymptoticLRScheduler(optimizer, total_steps=300, warmup_steps=10)
coords = get_isotropic_coords(height=224, width=448)
Testing and quality checks
pytest runs the full deterministic property and regression suite. Coverage can
be checked with:
pytest --cov=zeta_scaling_toolkit --cov-report=term-missing --cov-fail-under=95
ruff check zeta_scaling_toolkit tests
mypy --strict zeta_scaling_toolkit
Run the optional mutation check with python tools/mutation_check.py. It
temporarily injects targeted source changes and runs the full test suite for
each one, restoring the source afterward. KILLED means a test detected the
injected bug; SURVIVED means the suite missed it. This expensive check is not
part of the default CI matrix.
CI runs the CPU test matrix and a nightly Hypothesis profile. CUDA coverage is skipped when no CUDA device is available. See TESTING.md. Reviewers can see the passing full GitHub Actions matrix at commit 719bfb7.
Real-task evaluation
The matched-seed char-level language-model benchmark and its results are documented in benchmarks/RESULTS.md. The 100-step table is early-optimisation only. At 500 steps, only the scheduler-only variant improved loss on all ten matched seeds; attention-only, initialization-only, and the combined variant had higher loss. At 1,000 steps over 20 seeds, the scheduler-only variant improved loss on all 20 matched seeds; attention-only was worse, while initialization-only and all-components paired intervals included zero. These results apply to the measured model and dataset only.
What this can and cannot do
- Opt-in log-length temperature scaling compensates for softmax dilution as unmasked key count grows. The synthetic measurement and approximation are recorded in benchmarks/RESULTS.md and tested in test_properties.py. This does not establish a language-model training benefit.
- RoPE and ALiBi do not require a learned absolute-position table. Their relative-position and numerical properties are covered in test_properties.py; extrapolation quality is task-dependent and must be measured.
- Context is not infinite. The optional streaming mask exposes a fixed window
plus sink tokens and cannot recall information outside that window. Its
visibility set is tested in test_properties.py.
The mask alone is dense.
StreamingKVCachesupplies a bounded inference-only KV cache when used one token at a time; it detaches cached values and retains only sinks plus a recent window. Cache retention and agreement with the dense mask are tested in test_properties.py. - Attention uses SDPA when one scale applies to every row and chunks query rows for row-specific scales or fully masked rows. Dense masks and ALiBi biases still use quadratic memory when constructed for a full sequence. None of these mechanisms provide unbounded recall.
- RoPE angles are computed in float64 from int64 positions before casting back to the input type. Large positions should not be converted directly to float16 or bfloat16; the precision behavior is tested in test_properties.py.
- Language-model measurements use one 32-wide model and one small dataset; the synthetic vision task used 500 steps and is deliberately simple. Validated on a 1,000-step character-level LM and a separate 500-step synthetic vision task, but not on large-scale foundation models. These results do not generalize to large-scale training; see benchmarks/RESULTS.md.
Known limitations
- The epsilon in normalization makes scale invariance approximate near zero;
its relative effect becomes small for vector norms around
1e-2and larger. - Attention temperature is bounded to
[tau_min, tau_max]via a logit-sigmoid parameterization; at extreme raw_tau values, sigmoid saturation may produce zero gradients, but the parameter can recover for finite perturbations. - Fixed-temperature configurations (where tau_min == tau_max) are not supported; tau bounds must have tau_min < tau_max.
- GELU/SiLU require data-dependent
calibrate_gain; fixedcalculate_gaindoes not support them. - An all-masked attention row returns zeros, not a probability distribution.
- The
(cos(angle), sin(angle))representation removes the branch cut away from the origin; angle remains undefined at the origin. - For rectangular orthogonal matrices with fewer output than input features,
norm shrinkage of roughly
sqrt(out_features / in_features)is structurally expected. See initialization properties. calibrate_gainequalises the whole-batch norm, not each sample's norm. For GELU/SiLU in deep stacks without normalisation layers the batch norm can be preserved while the median sample's signal collapses (the fixed-seed 30-layer SiLU characterization test checks for a median per-row ratio below 0.5). Seetest_known_limitation_silu_calibration_*.calibrate_gainemits aRuntimeWarningwhen its target ratio is not reached within 2% (for example, bounded activations such as tanh cannot reach 1.0).calibrate_gainis now transactional: if validation fails on any layer, all parameters revert to their original values.- Nothing here has been validated on large-scale training runs.
License and support
Python >= 3.9 and PyTorch >= 2.0. MIT License; see LICENSE. Experimental results are not a substitute for evaluating these components on the intended model, data, and training budget.
Metadata
Release files for zeta-scaling-toolkit 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| zeta_scaling_toolkit-0.5.0.tar.gz | 44.4 kB | With provenance | Details for zeta_scaling_toolkit-0.5.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| zeta_scaling_toolkit-0.5.0-py3-none-any.whl | Python 3 | none | any | Details for zeta_scaling_toolkit-0.5.0-py3-none-any.whl |
Total release size: 63.3 kB
Release files / zeta_scaling_toolkit-0.5.0.tar.gz
| Download URL | zeta_scaling_toolkit-0.5.0.tar.gz |
|---|---|
| Size | 44.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fa50983829d74497a8d2d981ac7ea82f2e48c699a2bfe818aaa8f6cfa63393e3
|
|
BLAKE2b-256 checksum How to use checksums |
4351406f22390734e453b835448023cc91cf3ae53cd4b59630b17f019cb5c889
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / zeta_scaling_toolkit-0.5.0-py3-none-any.whl
| Download URL | zeta_scaling_toolkit-0.5.0-py3-none-any.whl |
|---|---|
| Size | 18.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9db3a62d20e447fe894ce0ffc51ce005035a499850e5c01b467f01250d14bb65
|
|
BLAKE2b-256 checksum How to use checksums |
29652fe7ae420a7049a112612e4bacc8181211044e2a74abf293e5b230c2cc16
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logSecurity for zeta-scaling-toolkit 0.5.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for zeta_scaling_toolkit-0.5.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.