BLT Trainer
A pure functional PyTorch implementation of a 96-layer Byte Latent Transformer (BLT) with Muon + ScheduleFreeAdamW optimization and weighted multi-corpus datamix.
Features
- Byte Latent Architecture: Vectorized entropy/surprise-based dynamic byte patching and cross-attention pooling.
- Downloaded Datamix: Datasets are downloaded and locally cached to disk using Apache Arrow memory-mapping to completely prevent Hugging Face API rate-limiting during training.
- Weighted Multi-Corpus Sampling:
- DCLM Baseline (
mlfoundations/dclm-baseline-1.0) - FLAN Collection (
Muennighoff/flan) - Wikipedia QA (
microsoft/wiki_qa) - OpenWebMath Reasoning (
open-web-math/open-web-math) - Ubuntu Dialogue QA (
sedthh/ubuntu_dialogue_qa)
- DCLM Baseline (
- Advanced Logging: Real-time throughput reporting with TPS (Tokens Per Second), Tokens Processed, step latency, and ETA.
- Dual Optimizer Partitioning: Muon optimizer for 2D hidden projection matrices and Schedule-Free AdamW for 1D embeddings and norm weights.
- Cut Cross-Entropy: Chunked sequence loss computation to eliminate peak VRAM spikes from materializing large vocabulary logits.
Installation
pip install -e .
Usage
Run Training
blt-train --batch-size 8 --block-size 512 --steps 10000 --device cuda
CLI Options
| Flag | Type | Default | Description |
|---|---|---|---|
--steps |
int | 10000 |
Total training steps |
--batch-size |
int | 8 |
Batch size per step |
--block-size |
int | 512 |
Byte context length |
--streaming |
flag | False |
Stream from Hugging Face instead of downloading locally (default: False, downloaded) |
--cache-dir |
str | None |
Custom directory to cache downloaded datasets |
--log-interval |
int | 10 |
Frequency of progress logs in steps |
--device |
str | cuda (if available) else cpu |
Execution device |
--compile |
flag | False |
Enable torch.compile kernel fusion |
--lr-muon |
float | 0.02 |
Learning rate for Muon optimizer |
--lr-adam |
float | 1e-3 |
Learning rate for Schedule-Free AdamW |
Logging Output
Logs include step progress, loss, Tokens Per Second (TPS), Tokens Processed, step latency, and estimated time of arrival (ETA):
Step 00010/10000 ( 0.1%) | Loss: 6.4215 | TPS: 14,250.3 | Tokens: 40,960 (40.96K) | Step Time: 35.9ms | ETA: 00:05:55 | VRAM: 3.45GB
Step 00020/10000 ( 0.2%) | Loss: 6.1023 | TPS: 14,310.0 | Tokens: 81,920 (81.92K) | Step Time: 35.7ms | ETA: 00:05:52 | VRAM: 3.45GB
1 maintainer
Metadata
1 maintainer
Release files for blt-trainer 0.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| blt_trainer-0.1.3.tar.gz | 12.6 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| blt_trainer-0.1.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 26.7 kB
Release files / blt_trainer-0.1.3.tar.gz
| Download URL | blt_trainer-0.1.3.tar.gz |
|---|---|
| Size | 12.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8517d43bdcedfb7943a9bb81365de0aeb15610b9ec437cf0910eaa769c20193f
|
|
BLAKE2b-256 checksum How to use checksums |
89e15a8f6bdd9f81d640fad0cb5492c4678fc17590a0c4c881c896b6d64c5e25
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.4
|
Release files / blt_trainer-0.1.3-py3-none-any.whl
| Download URL | blt_trainer-0.1.3-py3-none-any.whl |
|---|---|
| Size | 14.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6c30ad642528074936bded29988dfb78f075b3b6bff00d2aefee6dd8781ffc96
|
|
BLAKE2b-256 checksum How to use checksums |
5ee29fe0566aa3f818f46aad7c88e488272e3aee590690f7dcae68c5763f4929
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.4
|
Security for blt-trainer 0.1.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
