physis-lm 0.1.2

Reference PyTorch implementation of the Physis-LM preprint (byte-native, non-autoregressive, hierarchical-hourglass language model). Made with AI assistance.

physis-lm

Reference PyTorch implementation of the preprint "Physis-LM: A Parallel Non-Autoregressive Language Model with Hierarchical Latent Compression and Toward Guaranteed Output Consistency" (Omur Bera Isik, 2026) — a byte-native, non-autoregressive language model built around a Dynamic-Depth Hierarchical Hourglass (DDHH), a Physis Long-Range Bridge (PLRB), Length Parameters (LP), and a Soft Consensus Module (SConM).

Made with AI assistance. Version 0.1.2.

What's new in 0.1.2

• The model now demonstrably generalizes on a task that requires routing a transformation through the bottleneck — after fixing a bug that made it look like it couldn't. AdaptiveSlotPool (Section 10.1) was the only attention in the model that did not RMSNorm its input before the K/Q projections; combined with the shared small init, its pre-softmax scores were std ~1e-7, softmax was uniform for all M slot queries, and the deep bottleneck latent Z collapsed to M copies of one vector (measured: cross-slot cosine 1.0000). A copy task hid this (the output->input cross-attention bypasses Z); a Caesar-shift task exposed it — training sat at chance. Fix (bug S2): RMSNorm before the K/V projections + a 1/sqrt(C) init on WQs/WKs. After it, a 3.7M and a 14.7M model both learn shift+3 to 100% held-out byte AND sequence accuracy on a seed-disjoint test set (26^16 prompt space, so this is rule-learning, not memorization). See docs/SCALE_FINDINGS.md study S8.
• On real text, it does NOT yet produce meaningful language — and the reason is compute, not the architecture. Trained on 1.1M bytes of Shakespeare (next-chunk prediction, an 11M model on CPU), the model's loss falls to ~4.8 bits/byte within ~150 steps and then plateaus: it learns the marginal character distribution (its samples are a single repeated character) and little context structure. This reproduced across three chunk lengths and two batch sizes. A byte model needs many epochs over even this small corpus to learn language, and at ~0.5-2s/step on a 4GB CPU box the model sees well under one epoch; plain cross-entropy (without the designed auxiliary losses) and the parallel-chunk objective compound it. Whether the architecture models language at real scale is untested here — gated by hardware, not answered. Study S9.
• Section 3b — Hugging Face wrapper (physis_lm.hf). PhysisLMConfig + PhysisLMForByteModeling round-trip through save_pretrained/from_pretrained and register with AutoModel. It is non-autoregressive, so generate() RAISES and points at generate_cpd rather than silently running HF's AR decoder. Optional import; transformers is not a hard dependency.
• Section 3a — JAX parity backend (physis_lm.jax_backend). The primitive layers and a full PhysisBlock, ported as pure functions over torch-copied weights and parity-tested to ~1e-10 in x64 (12 tests). Scope is stated plainly: the full end-to-end hourglass/PLRB/SConM forward is not ported (note J1).
• Appendix D — power-iteration spectral norm (spectral_norm_method= "power_iteration"). The fused CUDA kernels remain out of scope (no GPU), but the numerical method they would accelerate is implemented and selectable; converges to the exact SVD sigma_max. Its autograd surrogate makes gradcheck fail by design — documented, not hidden (note D1).
• Honest scale ceiling. On the 4GB CPU sandbox, models above ~15M are killed by transient out-of-memory spikes mid-training (PLRB's content-dependent tensor shapes fragment the allocator) even at ~1GB steady RSS; malloc_trim + bs=4 made 14.7M stable, and beyond that needs more RAM or a GPU. This is an environment limit, not an architecture result (study S8b).

What's new in 0.1.1

Four of the reference's documented findings now have fixes. Each was re-reproduced against 0.1.0 before being touched, is covered by a new test, and — except F5, which only changes an initialization and is a strict improvement — is off by default, so every 0.1.0 result still reproduces exactly.

• F5 fix (ON by default): the attention-pool downsampler is now norm-preserving (W_V init std 1/sqrt(C) + a learnable gain), so the residual stream no longer collapses below RMSNorm's eps floor at init. Strict improvement; pool_norm_preserving=False restores the old init.
• R2 fix (opt-in prototype): plrb_soft_routing=True gives PLRB a differentiable soft-routing path so W_route actually trains — the gradient route the paper claims but the literal argmax doesn't provide.
• F3 fix (opt-in prototype): generate_cpd(streaming=True) now runs, via periodic PLRB re-hashing (Remark 15.1). Exact (bit-identical to naive CPD) at rehash_period=1. It does not reach the paper's O(L_out) cost — the DDHH still recomputes each chunk, because its coarse levels genuinely can't be cached (finding F3 stands) — and that is stated, not hidden.
• F4 option (opt-in): fixed_depth_padding=True pins ell/n0 to a constant graph. A true logit-level padding-length invariance was investigated and shown structurally impossible for this architecture; see "Batching and train/inference consistency" below.
• Thinker-mode (opt-in prototype): thinker_steps > 1 (and optional thinker_scratchpad_slots) iterates the deep bottleneck over the latent as a parallelism-preserving deliberation loop (no autoregressive fallback). The mechanism is live and trainable; it is not a demonstrated reasoning gain — the extra steps must be trained to do useful work. See "Thinker-mode" below.
• License is now Apache-2.0 (was MIT).

Full detail and honest limits: docs/IMPLEMENTERS_NOTES.md (notes F3, F4, F5, R2) and docs/SCALE_FINDINGS.md (studies S3, S6, S7).

Read this first (honesty statement)

Physis-LM is currently in the demo stage; we cannot guarantee that everything works. If you train the model and get poor results, please be aware that this could be one of the contributing factors.

The paper's own title page states that no version of this architecture has ever been assembled, trained, or evaluated end-to-end, and that every quality claim in its Section 20 is an untested prediction. This package changes the first half of that sentence — the architecture is now assembled, correct at the paper's own toy scale, and heavily tested — and none of the second half:

• There are no pretrained weights. Anywhere. physis-lm train really trains (and toy-scale tests verify learning happens), but training a competitive model needs the paper's 50-500B-byte budgets and GPUs.
• Everything implemented is tested against the paper's own numbered claims; everything not implemented raises a clear error or is listed below.
• Three paper claims are demonstrably false as written (findings F3, F4, F5) and one is self-contradictory (R2); 0.1.1 adds an honest fix or mitigation for each (see "What's new" above) while the underlying findings are still documented in full, not papered over. See docs/IMPLEMENTERS_NOTES.md and docs/SCALE_FINDINGS.md.

Install

pip install physis-lm            # from a built wheel/PyPI
# or, from a source checkout:
pip install -e ".[test]" && pytest

Dependencies: numpy, torch>=2.1 (CPU is fine — this package is developed and tested entirely on one CPU core). Python >= 3.10.

Quickstart — library

import torch
from physis_lm import PhysisCoreConfig, PhysisLM

cfg = PhysisCoreConfig(C=32, M=8, H=4, W=8, Nmax=32, nref=2048,
                       NB=2, K_sconm=2, droute=8, R=2, bucket_target=8)
model = PhysisLM(cfg)

ids = torch.tensor([list(b"hello physis, this is a byte-native model")])
out = model(ids)                       # single parallel forward pass
print(out.logits.shape)                # (1, Nmax, 256) — all positions at once
print(out.sigma_score)                 # LP completion score (Section 12.6)

text, info = model.generate_cpd(ids, Lmax=128)   # Chunked Parallel Decoding
print(bytes(text.tolist()))

Training (the full Section 18.4 three-stage schedule, homoscedastic loss weighting, curriculum, checkpointing):

from physis_lm import data as D
from physis_lm.torch_backend import train as TR

ds = D.dataset_from_paths(["my_corpus/"], context_len=128,
                          Nmax=cfg.Nmax, pad_byte=cfg.pad_byte)
provider = D.batch_provider_from_dataset(ds, batch_size=4, pad_byte=cfg.pad_byte)
history = TR.train(model, provider, TR.TrainSettings(total_steps=1000,
                                                     ckpt_dir="ckpt"))

The model pads internally to exactly M * r**ell (the CV 15.3 hard requirement), so you never need to understand DDHH depth arithmetic: any input length up to cfg.max_supported_raw_length() - Nmax just works, and anything else raises an error telling you to increase nref.

Quickstart — CLI

physis-lm selftest                     # end-to-end pipeline check (~5 s)
physis-lm info --size base             # configs + parameter-count oracles
physis-lm train --data my_files/ --steps 500 --ckpt-dir ckpt
physis-lm generate --ckpt ckpt/final.pt --prompt "hello" --max-bytes 200
physis-lm scale-study                  # regenerate docs/SCALE_FINDINGS.md

train accepts .txt, .jsonl (--text-field), .csv/.tsv, .html (--html-mode strip|raw), and raw binary files; malformed inputs fail with messages naming the file, line, and problem.

Status table

Per-module status; each module's docstring carries the same note, and docs/IMPLEMENTERS_NOTES.md holds the full list of paper ambiguities/contradictions and the decision taken for each (tags like "note B7" below resolve there).

Module	Paper	Status	Tested by	Limitations / notes
core geometry, padding, output slice	7.2, Rem. 7.1, CV 15.3, Rem. 15.4	implemented	test_core (incl. the literal CV 15.3 sweep, hypothesis sweeps)	ell >= 1 clamp (G1)
core complexity + parameter oracles	7.7, C.1/C.6, Prop 9.1, 17.5, Sec. 1, 12.2, 18.7, App. A	implemented	test_core reproduces every printed table/figure	paper's 130,808 misprint recorded (P1)
core LSH oracle + config	9.3, Prop 9.2, CV 9.3, App. A.1	implemented	test_core (collision/recall sims; config rejections)	Omega = f(seed, round, B) (R1)
RMSNorm / RoPE / SwiGLU / spectral / local attention	6.2-6.6, 28.1-28.4, CV 8.3, Prop 8.2	implemented	test_torch_layers (banded==O(n^2) oracle, CV 8.3 both halves, circulant Jacobian, gradchecks)	banded impl is O(nW) memory; no CUDA kernels (App. D out of scope, no GPU)
PhysisBlock, pooling, upsample, bottleneck, LP	6.5, 7.3-7.4, 10, 12, CV 8.5, CV 12.6	implemented	test_torch_blocks (CV 8.5 both cases, all three LP propositions vs autograd)	B3, B8-B10 readings; 0.1.2 fixes AdaptiveSlotPool bottleneck-collapse bug S2 (RMSNorm + 1/sqrt(C) init)
PLRB	8-9, 19.5, C.6	implemented (fixed hashing; 0.1.1 opt-in soft routing)	test_torch_plrb (segmented == mask oracle, sentinel, zero W_route grad by default, soft-routing gives W_route a real gradient)	default trains everything except W_route as literally specified (R2); 0.1.1 plrb_soft_routing fixes this (prototype); learnable-Omega variant a disclosed stub (R3); dense hashing = C.6 trap at large n (study S5)
SConM + streaming readout	14, CV 14.2, Lemma 14.8, D.2	implemented	test_torch_sconm (LOO exact single/multi-head, LSE recovery, vertex bound, scopes)	no contraction claim — conditional on unmeasured L_g (S1; studies S1/S2); 0.1.2 adds opt-in power-iteration spectral norm (note D1)
Full model, integration, LSR, CPD	App. B, CV 15.2/15.3, Rem. 15.4, 7.8.2, 15	implemented	test_torch_model (the paper's own sixteen-join toy-scale integration test, ablations, LSR bit-exactness, CPD incl. streaming exact-at-period-1)	0.1.1 streaming CPD runs via periodic PLRB re-hash, exact at rehash_period=1, but does not reach O(L_out) — DDHH still recomputes (finding F3); CPD stops gracefully at the context limit
Losses, schedules, training loop, PTCC	16.3, 18, 19, CV 18.2/18.4/18.10	implemented	test_training (a real toy run that learns + checkpoint round-trip), test_data_ptcc_pcc	L1, T1-T3 decisions; no mixed precision / Flash Attention / grad checkpointing (throughput devices, no GPU here)
PCC	24, CV 24.2	implemented	test_data_ptcc_pcc (losslessness incl. adversarial streams, tier accounting)	ratios are corpus claims — only CV 24.2's qualitative pattern asserted (C1)
Data layer + CLI	task Sec. 5	implemented	test_data_ptcc_pcc, test_cli (end-to-end train->generate on real files)	formats defined precisely in data.py's docstring
Scale studies	task Sec. 2	implemented	test_scale_studies	untrained instantiations only; see report header
Thinker-mode (deliberation loop)	task Sec. 5	prototype (opt-in)	test_torch_model (default == single-pass, live + trainable, parallelism preserved)	mechanism only; genuine reasoning gains require training the steps (note K1)
JAX parity backend (layers + PhysisBlock)	task Sec. 3a	implemented (layer/block level)	test_jax_parity (12 tests, torch-vs-JAX at copied weights, ~1e-10 in x64)	full end-to-end hourglass/PLRB/SConM forward NOT ported — tested foundation only (note J1)
Hugging Face wrapper	task Sec. 3b	implemented	test_hf (config round-trip, save/from_pretrained identical logits, AutoModel, non-AR contract)	optional import; generate() raises -> generate_cpd; no pretrained weights
Appendix D spectral-norm kernel	App. D	power-iteration path implemented; fused CUDA kernels out of scope	test_torch_sconm (converges to SVD sigma_max; gradient flow)	no GPU for fused kernels; gradcheck fails by design (surrogate gradient, note D1)
Any-to-Physis weight transplant	task Sec. 4	deferred to a later release	—	intentionally not shipped in this version; design in docs/SECTIONS_3_4_DESIGN.md

Test suite: 1497 tests, all passing (pytest -q), dominated by cheap parametrized pure-math sweeps plus hypothesis-generated cases; every Computational Verification remark in the paper that can run on CPU is re-executed against this codebase's actual code.

The findings and their 0.1.1 fixes

• F3 — Streaming DDHH's premise ("levels above j* do not change") is false: spectral mixing is a global circular convolution at every level (quantified in study S7). 0.1.1: generate_cpd(streaming=True) now runs — bit-identical to naive CPD at rehash_period=1, and periodic PLRB re-hashing (Remark 15.1) for larger periods — but the DDHH still recomputes each chunk, so it does not reach the paper's O(L_out) streaming cost. The finding stands; the fix is honest about what it does and doesn't buy.
• F4 — the literal Section 19.5 claim that padding to a longer batch length leaves behavior identical is structurally impossible (padding changes ell, and extending the input shifts the output placeholders). The achievable invariant — batch-content independence at fixed shape — holds to 1e-10 and is tested (study S6). 0.1.1: fixed_depth_padding=True pins ell/n0 to a constant graph; a true logit-level length invariance was investigated and shown impossible here.
• F5 (found by this implementation) — at initialization the residual stream collapsed geometrically through the attention-pool downsamplers (~x0.02-0.09 per level), pushing deep levels under RMSNorm's eps floor (study S3). 0.1.1 (ON by default): the norm-preserving downsampler holds every level above the floor across the whole grid measured; S3 is now a before/after table.
• R2 — Section 9.7 is self-contradictory: it says the routing argmax "is not differentiated through" yet "gradients flow normally through W_route". The literal argmax gives W_route exactly zero gradient (tested), so the default fixed-hashing path trains everything except W_route. 0.1.1: plrb_soft_routing=True supplies a differentiable soft-routing path so W_route trains (prototype; off by default).

Batching and train/inference consistency

Because n0 = M * r**ell depends on the padded input length, this architecture's output for a fixed sequence is not invariant to how much padding you add: different padded lengths mean a different number of DDHH levels and a different spectral DFT length (finding F4). Two practical consequences:

1. The guarantee you can rely on is batch-content independence at a fixed batch shape: a sequence's logits do not depend on which other sequences share its batch, as long as the batch's padded length is the same. This holds to floating-point precision and is tested.
2. To keep training and inference consistent, either (a) use the same nref and the same batch-shaping discipline in both — so a given input lands at the same ell in training and at inference — or (b) set PhysisCoreConfig(fixed_depth_padding=True), which pins every input to ell_max / n0 = M*r**ell_max. Option (b) makes the computation graph constant across input lengths (removing the batch-shape-dependent DFT-length drift) at the cost of always running at maximum depth; it does not make two different-length inputs produce identical logits (that is impossible here), but it removes padding-length as a source of train/inference mismatch.

Thinker-mode (prototype)

PhysisCoreConfig(thinker_steps=T, thinker_scratchpad_slots=S) turns on an opt-in deliberation loop (task Section 5). Instead of a single deep-bottleneck pass over the compressed latent Z, the bottleneck is applied T times, adding a learnable per-step code each iteration, with S optional learnable "scratchpad" latent slots as working memory (dropped before readout so the output shape is unchanged). Every latent slot is refined simultaneously at each step, and the loop iterates over deliberation steps, never over output tokens — so the architecture's parallelism is preserved and there is no autoregressive fallback. thinker_steps=1 with no scratchpad (the default) is byte-for-byte the single-pass model.

What this is and isn't: the mechanism is implemented, live (non-zero step codes change the output), and trainable (gradients reach the step codes and scratchpad) — all tested. It is not a demonstrated reasoning improvement. The per-step codes are zero at initialization, so an untrained model's first step equals the ordinary pass and further steps just re-apply the bottleneck; making the extra steps do genuinely new work is a training problem (train with thinker_steps > 1, most plausibly with a ponder/ACT-style objective). This release ships the mechanism honestly labeled as a prototype, not a claim that iterating an untrained bottleneck reasons better. See docs/IMPLEMENTERS_NOTES.md note K1.

Repository layout

physis_lm/
  core.py            geometry, cost/parameter oracles, LSH math, config (no torch)
  data.py            byte-level corpus loading + batching (stage-1 padding only)
  pcc.py             Physis Context Compression (Section 24; no torch)
  cli.py             the `physis-lm` command
  scale_studies.py   Section-2 measurements -> markdown report
  torch_backend/
    layers.py blocks.py plrb.py sconm.py model.py losses.py train.py ptcc.py
tests/               1470 tests; conftest.py holds the paper's toy scale
docs/                PLAN.md (pre-implementation plan), IMPLEMENTERS_NOTES.md,
                     SCALE_FINDINGS.md

Citing

If you use this implementation, cite the Physis-LM preprint (Omur Bera Isik, 2026). This codebase: physis-lm 0.1.2, made with AI assistance, Apache-2.0 license.