Skip to main content

Zeta Scaling Toolkit

Experimental PyTorch utilities for attention, learning-rate schedules, coordinates, and linear initialization. Numerical behavior is covered by the linked tests; none of these components is validated as a training improvement on large-scale tasks.

Components

  • NonCollapsingAttention computes softmax(tau * cos(q, k)) @ v, where tau is learnable, starts at tau0 (default 10), and is bounded to [tau_min, tau_max] (defaults 1 and 50) using a logit-sigmoid parameterization. This parameterization ensures non-zero gradients throughout the bounded range, allowing the temperature parameter to recover if perturbed beyond the bounds. Its normalization and softmax calculations use at least float32, and the result is cast to v.dtype. Boolean masks use True to retain a position; floating masks are added to the logits. Fully masked query rows return zeros. Shapes, dtypes, devices, and optional finite-value checks are validated; an empty key sequence raises ValueError, while empty query sequences and batches are supported. See attention property tests, edge-case tests, and regression tests.
  • AsymptoticLRScheduler is the retained public class name for a schedule that linearly warms up, follows cosine decay to floor_ratio * base_lr, then stays at that floor. floor_ratio must be in [0, 1], total_steps >= 1, and 0 <= warmup_steps < total_steps. Without warmup it is mathematically equivalent to CosineAnnealingLR with the same floor inside the configured horizon, up to floating-point rounding; unlike stock cosine it stays clamped to the floor beyond that horizon. See scheduler property and resume tests.
  • get_isotropic_coords returns pixel-centred (x, y) coordinates normalized by max(height, width). get_scale_invariant_vit_coords returns (radius, angle) or, with smooth_angle=True, (radius, cos(angle), sin(angle)). See the coordinate tests and compatibility tests.
  • make_corrected_linear applies orthogonal initialization scaled by a gain supported by torch.nn.init.calculate_gain; its default is ReLU gain. GELU and SiLU are not supported as fixed gains. Use calibrate_gain with representative sample data for sequential GELU/SiLU stacks. See initialization properties and calibration regressions.

Deprecated A_attn, T, and A_coords arguments remain accepted, warn when non-default values are supplied, and have no effect. Legacy attention checkpoints containing q_scale and k_scale load with a warning and retain the new module's initialized temperature; see compatibility tests.

Installation

pip install -e .

For development and tests:

pip install -e ".[dev]"

Quick start

import torch
from zeta_scaling_toolkit import (
    AsymptoticLRScheduler,
    NonCollapsingAttention,
    get_isotropic_coords,
    make_corrected_linear,
)

attention = NonCollapsingAttention(d_k=64)
q, k, v = (torch.randn(2, 128, 64) for _ in range(3))
output = attention(q, k, v)

linear = make_corrected_linear(1024, 4096, nonlinearity="relu")
optimizer = torch.optim.Adam(linear.parameters(), lr=1e-3)
scheduler = AsymptoticLRScheduler(optimizer, total_steps=300, warmup_steps=10)

coords = get_isotropic_coords(height=224, width=448)

Testing and quality checks

pytest runs the full deterministic property and regression suite. Coverage can be checked with:

pytest --cov=zeta_scaling_toolkit --cov-report=term-missing --cov-fail-under=95
ruff check zeta_scaling_toolkit tests
mypy --strict zeta_scaling_toolkit

Run the optional mutation check with python tools/mutation_check.py. It temporarily injects targeted source changes and runs the full test suite for each one, restoring the source afterward. KILLED means a test detected the injected bug; SURVIVED means the suite missed it. This expensive check is not part of the default CI matrix.

CI runs the CPU test matrix and a nightly Hypothesis profile. CUDA coverage is skipped when no CUDA device is available. See TESTING.md. Reviewers can see the passing full GitHub Actions matrix at commit 719bfb7.

Real-task evaluation

The matched-seed char-level language-model benchmark and its results are documented in benchmarks/RESULTS.md. The 100-step table is early-optimisation only. At 500 steps, only the scheduler-only variant improved loss on all ten matched seeds; attention-only, initialization-only, and the combined variant had higher loss. At 1,000 steps over 20 seeds, the scheduler-only variant improved loss on all 20 matched seeds; attention-only was worse, while initialization-only and all-components paired intervals included zero. These results apply to the measured model and dataset only.

What this can and cannot do

  • Opt-in log-length temperature scaling compensates for softmax dilution as unmasked key count grows. The synthetic measurement and approximation are recorded in benchmarks/RESULTS.md and tested in test_properties.py. This does not establish a language-model training benefit.
  • RoPE and ALiBi do not require a learned absolute-position table. Their relative-position and numerical properties are covered in test_properties.py; extrapolation quality is task-dependent and must be measured.
  • Context is not infinite. The optional streaming mask exposes a fixed window plus sink tokens and cannot recall information outside that window. Its visibility set is tested in test_properties.py. The mask alone is dense. StreamingKVCache supplies a bounded inference-only KV cache when used one token at a time; it detaches cached values and retains only sinks plus a recent window. Cache retention and agreement with the dense mask are tested in test_properties.py.
  • Attention uses SDPA when one scale applies to every row and chunks query rows for row-specific scales or fully masked rows. Dense masks and ALiBi biases still use quadratic memory when constructed for a full sequence. None of these mechanisms provide unbounded recall.
  • RoPE angles are computed in float64 from int64 positions before casting back to the input type. Large positions should not be converted directly to float16 or bfloat16; the precision behavior is tested in test_properties.py.
  • Language-model measurements use one 32-wide model and one small dataset; the synthetic vision task used 500 steps and is deliberately simple. Validated on a 1,000-step character-level LM and a separate 500-step synthetic vision task, but not on large-scale foundation models. These results do not generalize to large-scale training; see benchmarks/RESULTS.md.

Known limitations

  • The epsilon in normalization makes scale invariance approximate near zero; its relative effect becomes small for vector norms around 1e-2 and larger.
  • Attention temperature is bounded to [tau_min, tau_max] via a logit-sigmoid parameterization; at extreme raw_tau values, sigmoid saturation may produce zero gradients, but the parameter can recover for finite perturbations.
  • Fixed-temperature configurations (where tau_min == tau_max) are not supported; tau bounds must have tau_min < tau_max.
  • GELU/SiLU require data-dependent calibrate_gain; fixed calculate_gain does not support them.
  • An all-masked attention row returns zeros, not a probability distribution.
  • The (cos(angle), sin(angle)) representation removes the branch cut away from the origin; angle remains undefined at the origin.
  • For rectangular orthogonal matrices with fewer output than input features, norm shrinkage of roughly sqrt(out_features / in_features) is structurally expected. See initialization properties.
  • calibrate_gain equalises the whole-batch norm, not each sample's norm. For GELU/SiLU in deep stacks without normalisation layers the batch norm can be preserved while the median sample's signal collapses (the fixed-seed 30-layer SiLU characterization test checks for a median per-row ratio below 0.5). See test_known_limitation_silu_calibration_*.
  • calibrate_gain emits a RuntimeWarning when its target ratio is not reached within 2% (for example, bounded activations such as tanh cannot reach 1.0).
  • calibrate_gain is now transactional: if validation fails on any layer, all parameters revert to their original values.
  • Nothing here has been validated on large-scale training runs.

License and support

Python >= 3.9 and PyTorch >= 2.0. MIT License; see LICENSE. Experimental results are not a substitute for evaluating these components on the intended model, data, and training budget.

Metadata

Release files for zeta-scaling-toolkit 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for zeta-scaling-toolkit 0.5.0
File Size Uploaded Provenance Actions
zeta_scaling_toolkit-0.5.0.tar.gz 44.4 kB With provenance Details for zeta_scaling_toolkit-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for zeta-scaling-toolkit 0.5.0
File Interpreter ABI Platform Actions
zeta_scaling_toolkit-0.5.0-py3-none-any.whl Python 3 none any Details for zeta_scaling_toolkit-0.5.0-py3-none-any.whl

Total release size: 63.3 kB

Release files / zeta_scaling_toolkit-0.5.0.tar.gz

Download URL zeta_scaling_toolkit-0.5.0.tar.gz
Size 44.4 kB
Tags Source
SHA-256 checksum
How to use checksums
fa50983829d74497a8d2d981ac7ea82f2e48c699a2bfe818aaa8f6cfa63393e3
BLAKE2b-256 checksum
How to use checksums
4351406f22390734e453b835448023cc91cf3ae53cd4b59630b17f019cb5c889
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / zeta_scaling_toolkit-0.5.0-py3-none-any.whl

Download URL zeta_scaling_toolkit-0.5.0-py3-none-any.whl
Size 18.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9db3a62d20e447fe894ce0ffc51ce005035a499850e5c01b467f01250d14bb65
BLAKE2b-256 checksum
How to use checksums
29652fe7ae420a7049a112612e4bacc8181211044e2a74abf293e5b230c2cc16
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Security for zeta-scaling-toolkit 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for zeta_scaling_toolkit-0.5.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.5.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page