Anthracite 2.0.2

Anthracite is a PyTorch-first training and inference framework for building text-generation and embedding models from scratch or fine-tuning existing Anthracite checkpoints.

Version 2.0.2 focuses on predictable distributed startup, external-tokenizer compatibility, efficient decoder inference, safer checkpoints, and a cleaner production workflow.

What 2.0.2 changes

1. Multi-GPU external tokenizer fix

The old failure mode was especially confusing because the real tokenizer backend error occurred inside a spawned worker and PyTorch reported it as ProcessRaisedException plus a SIGTERM for another worker.

2.0.2 changes the startup order:

1. The parent process loads the supplied tokenizer first.
2. The parent materializes a self-contained tokenizer bundle in .anthracite-cache/tokenizer.
3. Anthracite round-trip loads that exact bundle before GPU workers start.
4. Workers receive the shared tokenizer path instead of reconstructing the original live tokenizer object.
5. A failed tokenizer backend is therefore a normal preflight error, not a secondary multiprocessing traceback.

Supported external tokenizer forms include Hugging Face tokenizer directories, tokenizer.json bundles, Hugging Face repo identifiers, low-level tokenizers objects, and live custom objects with encode/decode methods. For automatic multi-GPU worker startup, the supplied tokenizer must also be persistable through save_pretrained() or save(path).

For slow tokenizer families, Anthracite tries the Transformers slow tokenizer path before the fast conversion path. The distribution also includes sentencepiece and tiktoken so common slow-tokenizer backends are available after installation.

2. Anthracite-2.0 decoder architecture

The default text architecture is anthracite-2.0 and keeps the framework's own decoder design while improving the implementation:

• RMSNorm pre-normalization, using PyTorch's fused rms_norm when available.
• Rotary position encoding (RoPE).
• Grouped-query attention (GQA) with separate Q and KV head counts.
• PyTorch scaled-dot-product attention (SDPA) when available, with a math fallback.
• Q/K normalization can be enabled with qk_norm=True.
• SwiGLU feed-forward blocks.
• Weight-tied input/output embeddings by default.
• Optional linear or NTK-style RoPE scaling controls.
• Gradient checkpointing support for memory-constrained training.
• Preallocated per-layer KV cache for low-allocation autoregressive generation.

The KV cache is not a claim that generation is magically constant-time: each new token still attends to the existing context. The important improvement is that cached K/V tensors are appended in-place rather than repeatedly concatenated into new tensors.

3. Safer checkpoints and resume

Checkpoints are written into a temporary directory and become visible only after all required files finish writing. This reduces the chance of a partially written checkpoint being treated as the latest good checkpoint.

Resume is strict by default. If the saved tensors do not match the current architecture, Anthracite raises a checkpoint error instead of silently ignoring missing or unexpected tensors. Set strict_resume=False only for an intentional migration.

Installation

Anthracite 2.0 targets modern Python and PyTorch environments.

pip install .

The package declares the tokenizer backends commonly required by external Hugging Face tokenizers (tokenizers, sentencepiece, and tiktoken).

For TPU support:

pip install .[tpu]

For Hugging Face datasets:

pip install .[hf]

Quick start

from anthracite import train, generate

train(
    model_name="Saraswati-3M",
    model_type="text_gen",
    dataset="data.txt",
    tokens="100M",
    params="3M",
    context_length=512,
    tokenizer="./tokenizer/",
    device="auto",
    precision="bf16",
)

print(generate("./models/Saraswati-3M", "Hello,", max_new_tokens=80))

When device="auto" detects more than one CUDA GPU on a single machine, Anthracite can launch one distributed worker per GPU. The distributed path uses DDP/data parallelism; the model replica must therefore fit on each GPU.

External tokenizer example

from transformers import AutoTokenizer
from anthracite import train

tok = AutoTokenizer.from_pretrained("gpt2", use_fast=False)

train(
    model_name="ExternalTokModel",
    dataset="data.txt",
    tokens="50M",
    params="20M",
    tokenizer=tok,
    device="auto",
)

A local tokenizer bundle also works:

train(
    model_name="LocalTokModel",
    dataset="data.txt",
    tokens="50M",
    params="20M",
    tokenizer="./my-tokenizer/",
)

For a multi-GPU run, the parent process persists and validates the tokenizer before workers start. The original tokenizer object is not expected to be reconstructed independently in each worker.

Using Anthracite-2.0 directly

The public training API forwards every TrainingConfig field through **kwargs, including the new attention/cache controls:

train(
    model_name="ModernModel",
    dataset="data.txt",
    params="100M",
    context_length=2048,
    attention_backend="auto",   # auto | sdpa | math
    rope_scaling="ntk",         # none | linear | ntk
    use_cache=True,
    grad_checkpointing=True,
)

attention_backend="auto" prefers PyTorch SDPA. attention_backend="math" is the explicit implementation fallback. grad_checkpointing=True trades extra recomputation for lower activation memory and is intended for training, not cached generation.

Multi-GPU execution

Automatic single-node launch:

train(
    model_name="MultiGPUModel",
    dataset="data.txt",
    tokens="1B",
    params="300M",
    device="multi-gpu",
)

Or launch your script with PyTorch's distributed launcher:

torchrun --standalone --nproc-per-node=4 train.py

Anthracite does not claim that DDP can train a model larger than a single GPU's VRAM. DDP replicates the model on every rank. For model sizes that cannot fit on one GPU, use a sharded strategy such as PyTorch FSDP2 rather than assuming DDP will solve the memory problem.

KV-cache generation

Normal generation automatically uses Anthracite-2.0's KV cache:

from anthracite import generate

text = generate(
    "./models/ModernModel",
    "Once upon a time",
    max_new_tokens=256,
    temperature=0.8,
    top_p=0.95,
)

Streaming generation also reuses the incremental cache rather than regenerating the entire prefix for every emitted token:

from anthracite import stream_text

for piece in stream_text("./models/ModernModel", "Once upon a time", max_tokens=256):
    print(piece, end="", flush=True)

Checkpoints and resume

train(
    model_name="ResumeMe",
    dataset="data.txt",
    tokens="500M",
    params="100M",
    checkpoint_interval=2000,
    resume=True,
)

To intentionally load a compatible-but-not-identical checkpoint during migration:

train(
    model_name="Migrated",
    dataset="data.txt",
    tokens="10M",
    params="100M",
    resume="./models/old/checkpoints/checkpoint-5000",
    strict_resume=False,
)

Use strict_resume=False deliberately. It can hide an architectural mismatch and should not be the default for ordinary resume.

SFT and embeddings

SFT remains part of the training pipeline and can automatically detect common instruction/chat record layouts. Input masking can be enabled with sft_mask_input=True so only answer tokens contribute to the supervised loss.

Embedding models are supported through the existing model_type="embedding" pipeline with Anthracite or BERT-style backbones and mean/CLS/max pooling.

Model size and hardware reality

params="..." is a target budget. Anthracite solves that target into concrete width/depth/head dimensions; the final exact parameter count is stored in the model config and metadata.

A model's usefulness is not determined only by parameter count. Training quality also depends on tokenizer quality, data quality, token budget, optimization settings, context length, and hardware. Anthracite does not promise that a small training run will produce a competitive general-purpose assistant.

Anthracite can build substantially larger models when the architecture dimensions and hardware support them, but this 2.0.2 release's built-in distributed training path is DDP-style replication rather than full model sharding.

Output layout

A successful model directory contains the model weights, model config, tokenizer artifacts, training config, metrics, metadata, and a generated model card. Intermediate distributed artifacts live under .anthracite-cache and include the validated shared tokenizer and, when enabled, a reusable token cache.

Validation done for this 2.0.2 drop

The repository includes a small regression suite covering:

• external tokenizer.json round-tripping without the Rust tokenizer package present;
• persisted custom tokenizer-object round-tripping;
• Anthracite-2.0 forward/loss smoke tests;
• incremental KV-cache decode;
• cache-backed generation beyond the base context window;
• SDPA/math backend shape checks;
• version/config defaults;
• strict syntax compilation across the package.

Actual NCCL multi-GPU and TPU hardware execution is environment-dependent. Those paths still need to be exercised on the target hardware before a production training job.

Architecture compatibility

Existing legacy architecture names remain accepted for compatibility:

• anthracite-1
• anthracite-1.0
• anthracite-1.2
• gpt2
• bert (embedding pipeline)

New text models default to anthracite-2.0.

License

MIT. See LICENSE.

2.0.2 tokenizer reliability

External Hugging Face tokenizer bundles are materialized and validated in a parent-process staging directory before distributed workers start. Persisted special-token IDs are authoritative, including GPT-2's shared BOS/EOS id 50256.