H-Net Dynamic Chunking
Implementation of the dynamic chunking mechanism in H-net by Hwang et al. of Carnegie Mellon
Install
$ pip install h-net-dynamic-chunking
Usage
import torch
from h_net_dynamic_chunking import DynamicSequenceChunker
downsampler = DynamicSequenceChunker(512)
tokens = torch.randn(3, 1024, 512).requires_grad_()
downsampled, upsample_fn, *_ = downsampler(tokens)
assert upsample_fn(downsampled).shape == tokens.shape
3 layers hierarchy
import torch
from h_net_dynamic_chunking import DynamicSequenceChunker
downsampler1 = DynamicSequenceChunker(512)
downsampler2 = DynamicSequenceChunker(512)
downsampler3 = DynamicSequenceChunker(512)
tokens = torch.randn(3, 1024, 512).requires_grad_()
downsampled1, upsample_fn1, aux_loss1 = downsampler1(tokens)
# hierarchical network 1 ...
downsampled2, upsample_fn2, aux_loss2 = downsampler2(downsampled1)
# hierarchical network 2 ...
downsampled3, upsample_fn3, aux_loss3 = downsampler3(downsampled2)
# inner most network
# reconstituting
assert upsample_fn1(upsample_fn2(upsample_fn3(downsampled3))).shape == tokens.shape
HNet wrapper
import torch
from torch import nn
from h_net_dynamic_chunking.h_net import HNet
# 3 hierarchies, from 512 -> 1024 -> 2048 inner
net = HNet(
nn.Identity(),
HNet(
nn.Identity(),
HNet(
nn.Identity(),
nn.Identity(),
nn.Identity(),
dim = 2048
),
nn.Identity(),
dim = 1024,
dim_inner = 2048
),
nn.Identity(),
dim = 512,
dim_inner = 1024,
)
tokens = torch.randn(1, 1024, 512)
ret = net(tokens)
out, aux_loss = ret.output, ret.loss # (1, 1024, 512), ()
Example
Enwik8 with 2 hierarchies
$ pip install '.[examples]'
Then
$ python train.py
Citations
@misc{hwang2025dynamicchunkingendtoendhierarchical,
title = {Dynamic Chunking for End-to-End Hierarchical Sequence Modeling},
author = {Sukjun Hwang and Brandon Wang and Albert Gu},
year = {2025},
eprint = {2507.07955},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2507.07955},
}
@misc{balestriero2025lejepa,
title = {LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics},
author = {Randall Balestriero and Yann LeCun},
year = {2025},
eprint = {2511.08544},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2511.08544},
}
1 maintainer
Metadata
1 maintainer
Release files for h-net-dynamic-chunking 0.5.19
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| h_net_dynamic_chunking-0.5.19.tar.gz | 17.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| h_net_dynamic_chunking-0.5.19-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 37.4 kB
Release files / h_net_dynamic_chunking-0.5.19.tar.gz
| Download URL | h_net_dynamic_chunking-0.5.19.tar.gz |
|---|---|
| Size | 17.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
de4994031e50848e247fa535134f983f6cbd80796312f6c1b6e3d390ca3ae106
|
|
BLAKE2b-256 checksum How to use checksums |
abfbb65e3b513e66257fa3bef420361e71da5166f8100dc2dfe9501571ea814f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.17
|
Release files / h_net_dynamic_chunking-0.5.19-py3-none-any.whl
| Download URL | h_net_dynamic_chunking-0.5.19-py3-none-any.whl |
|---|---|
| Size | 19.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e91d225f5199e624ec6eccf0984269d7d51738c2214827a027d38786b73cf7de
|
|
BLAKE2b-256 checksum How to use checksums |
d6a857b524c28c001eab55f495b9a7d22e64330e37972fe0d7217e21b6313de4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.17
|
