Anthracite

Anthracite is a compact training framework for building text-generation and embedding models from local data, Hugging Face datasets, or external tokenizers. The project is designed for simple training loops, SFT workflows, and efficient runtime behavior on CPU, CUDA, and TPU-backed environments.

The framework is intentionally lightweight: the public interface stays small, model config stays transparent, and each training run saves its tokenizer, config, checkpoints, metadata, and model card in a reproducible directory structure.

---

Overview

Anthracite supports:

• text model training with a causal transformer
• embedding model training with MLM and contrastive objectives
• dataset auto-detection for plain text and instruction-style records
• supervised fine-tuning with input masking
• tokenizer loading from bundled, local, or Hugging Face sources
• automatic device resolution across CPU, CUDA, and TPU runtimes
• contextual long-sequence support through rotary positional embeddings

The architecture is intentionally based on a standard, explainable transformer stack rather than a hidden preset system. This keeps the code readable, inference predictable, and the training recipe easy to tune by hand.

---

What makes Anthracite useful

Anthracite is built for users who want a clean training stack without over-engineering the project. It tries to make the following process straightforward:

1. point at a dataset
2. choose a target parameter count or token budget
3. start training
4. save tokenizer + config + checkpoints
5. fine-tune or generate from the saved model

The framework emphasizes four practical goals:

• plain-language training configuration
• good support for instruction tuning and chat-style SFT data
• robust tokenization for arbitrary text data
• compatibility with modern accelerator hardware without requiring a large preset library

---

Installation

pip install anthracite
pip install anthracite[hf]
pip install anthracite[all]

For a fuller tokenization and dataset workflow, the Hugging Face extras are recommended:

pip install transformers datasets huggingface_hub tokenizers

A working PyTorch installation is required. TPU support relies on torch_xla being present in the runtime environment.

---

Quick start

from anthracite import train, generate

train(
    model_name="Anthracite-20M",
    dataset="data.jsonl",
    tokens=100_000_000,
    params="20M",
    context_length=512,
    tokenizer="auto",
    device="auto",
    precision="auto",
)

text = generate("./models/Anthracite-20M", "Once upon a time")
print(text)

This preserves a normal training flow while allowing the user to choose a tokenizer automatically or provide an explicit one.

---

Tokenizer behavior

Anthracite supports several tokenizer entry points:

1. Automatic tokenizer selection

train(
    model_name="DemoModel",
    dataset="data.jsonl",
    tokens=50_000_000,
    params="10M",
    tokenizer="auto",
)

When tokenizer="auto" is used, Anthracite prefers a Hugging Face tokenizer if it is available and falls back to the bundled tokenizer when it is not. This keeps the initialization path robust across machines.

2. Bundled tokenizer

train(..., tokenizer="bundled")

This uses Anthracite's built-in tokenization path and avoids separate tokenizer training.

3. Local tokenizer path

train(..., tokenizer="./tokenizers/mytok")
train(..., tokenizer="./tokenizer.json")

4. Hugging Face repo or repo file

train(..., tokenizer="gpt2")
train(..., tokenizer="meta-llama/Llama-2-7b-hf")
train(..., tokenizer="meta-llama/Llama-2-7b-hf/tokenizer.json")

5. Live tokenizer object

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
train(..., tokenizer=tok)

Special tokens are part of the Anthracite tokenizer vocabulary. This includes:

• <BOS>
• <EOS>
• <PAD>
• <UNK>
• <START>
• <END>
• <|reasoning_start|>
• <|reasoning_end|>

These markers help model training and SFT workflows where reasoning or structured thought sections need to be bounded and separated from final outputs.

---

Base training and raw text extraction

Anthracite is designed to avoid training on metadata objects when the dataset is structured. If a record contains an id, label, or other metadata, the framework keeps the content field and ignores the extra fields when choosing text.

A record like this:

{"id": "abc-001", "text": "The quick brown fox jumps over the lazy dog."}

is treated as a plain-text training sample, and only the actual text payload is used.

This matters for base training because the model should learn from the text content, not from JSON metadata or bookkeeping keys. If the record has a field such as conversation, messages, instruction, output, or text, the loader extracts the relevant body content and drops the rest.

That makes the base training path more robust for mixed-format or loosely structured datasets.

---

SFT and reasoning support

Anthracite supports supervised fine-tuning using instruction-output records. During SFT, the model learns to generate the answer while the prompt tokens are masked from the loss. This keeps the training objective aligned with instruction following rather than prompt memorization.

A typical record can look like this:

{
  "instruction": "Summarise the article.",
  "input": "Long text here",
  "output": "Short summary here"
}

Anthracite also supports reasoning-aware SFT records. If the dataset contains a reasoning, thought, analysis, or chain_of_thought field, the tokenizer markers are inserted automatically so the model learns a bounded reasoning segment and a final answer segment.

Example:

{
  "instruction": "Solve this step by step.",
  "input": "What is 12 x 8?",
  "reasoning": "Multiply 10 by 8 and then add 2 times 8.",
  "output": "96"
}

The model sees a structure like:

<|reasoning_start|>Multiply 10 by 8 and then add 2 times 8.<|reasoning_end|>96

This is useful because the output is not just raw text; it can include a short reasoning segment before a final answer. The SFT pipeline preserves that structure without forcing a preset template.

---

RoPE and context extension

Anthracite uses rotary positional embeddings for sequence modeling. This matters because RoPE is not a fixed positional table; it keeps the model length-aware and more flexible than learned absolute position embeddings.

The important design fact is this:

• the base model is trained with a chosen context length
• the RoPE cache can be extended beyond that limit with a context-extension strategy
• the model can then operate on longer windows when the user increases the target context or uses a longer dataset

The framework exposes the following knobs:

• context_length
• context_extension
• rope_theta
• rope_scale

A practical example:

train(
    model_name="LongContextModel",
    dataset="bigtext.txt",
    tokens=500_000_000,
    params="30M",
    context_length=1024,
    context_extension=4096,
    rope_theta=10000.0,
    rope_scale=1.0,
)

This means the training config is aware that user workloads may extend beyond the default context window. In rough terms, the effective RoPE horizon becomes larger than the original sequence length, which gives the model greater range during training and generation when the memory budget permits.

The practical rule is simple:

• larger context length needs more memory and more compute
• longer ranges help reasoning-heavy and long-document workloads
• the user decides how much extra context is worth paying for

Anthracite keeps this transparent instead of hiding it in a preset.

---

TPU and distributed hardware support

Anthracite resolves accelerator hardware through a device abstraction. The runtime recognizes:

• cpu
• cuda
• tpu
• xla
• auto
• multi-gpu
• multi-tpu

On TPU, the framework uses the torch_xla runtime when it is available. This gives the model a native XLA path and ensures the training stack is not tied to a single machine type.

This means the system is built to work across multiple hardware classes while keeping a single high-level API. The user can choose a specific accelerator or allow automatic resolution.

train(
    model_name="TPUModel",
    dataset="dataset.jsonl",
    tokens=500_000_000,
    params="50M",
    device="tpu",
    precision="bf16",
)

A system with a larger memory budget and enough accelerator capacity can scale the training range, token budget, and context length much farther than a small CPU-only setup. Anthracite does not place a hard cap on the model size in the code itself; the limit is primarily hardware and memory.

---

Model size and frontier ambition

Anthracite does not impose a strict artificial cap on model scale. In a real deployment, the effective upper bound is driven by:

• RAM and VRAM capacity
• available TPU or GPU devices
• context length and batch size
• dataset size and token budget
• training time budget

This matters because a large model can become more capable when the hardware allows it. The framework is therefore structured as a general training stack rather than a narrow toy model. It supports large-language-model-style training flows, long-context reasoning, instruction tuning, and large token budgets when the system has the compute behind it.

That said, the project remains a pragmatic open architecture rather than a fully specialized MoE frontier stack. The implementation already includes a modern transformer pattern and RoPE support, which are foundational for frontier-style long-context models, but the final frontier behavior still depends on the quality of the training data, the target hardware, and the chosen configuration.

---

Dataset formats supported

Anthracite auto-detects common training data layouts such as:

• plain text files
• JSONL text samples
• instruction-output records
• chat/conversation records
• ShareGPT style turns
• ChatML style messages
• OpenAI-style prompt/completion records
• QA records
• generic two-column records

The automatic detection logic examines the record shape and chooses the format that best matches the available fields.

---

Training configuration

The primary training interface accepts parameters such as:

train(
    model_name="Demo",
    dataset="data.jsonl",
    tokens=10_000_000,
    params="20M",
    context_length=512,
    vocab_size="auto",
    tokenizer="auto",
    device="auto",
    precision="auto",
    batch_size="auto",
    output_dir="./models/Demo",
)

The most important knobs are:

• model_name — output model directory name
• dataset — local file, directory, or HF dataset id
• tokens — total token budget
• params — target parameter count
• context_length — training horizon
• vocab_size — vocabulary size when training a tokenizer from scratch
• tokenizer — tokenizer source or object
• device — accelerator selection
• precision — compute precision mode
• batch_size — runtime scheduling control

---

Fine-tuning

from anthracite import finetune

finetune(
    model="./models/Anthracite-20M",
    dataset="instructions.jsonl",
    tokens=10_000_000,
    objective="sft",
    tokenizer="auto",
    output_dir="./models/Anthracite-20M-SFT",
)

The fine-tuning path reuses the original model architecture and tokenizer compatibility rules. If the user supplies a tokenizer, it must be compatible with the model vocabulary. Otherwise Anthracite raises a clear configuration error instead of silently training with mismatched tokens.

---

Generation and inference

from anthracite import generate

result = generate(
    "./models/Anthracite-20M",
    "Write a short story about a moonlit city.",
    max_new_tokens=200,
    temperature=0.8,
    top_p=0.95,
)

print(result)

This is useful for testing the trained checkpoint, confirming generation quality, and validating the model after fine-tuning or base training.

---

Notes for production use

Anthracite is deliberately modular. A user can swap:

• dataset source
• tokenizer source
• context length
• hardware target
• training objective

without rewriting the full training stack. This is useful in a real development pipeline where the same code path is reused across experiments, fine-tuning runs, and long-context evaluations.

For the best results, use:

• GPU or TPU for larger training runs
• stronger tokenization for noisy or multilingual corpora
• longer context lengths for document reasoning and long-form generation
• SFT data with explicit reasoning segments when the use case benefits from structured outputs

---

License

MIT License. See LICENSE for details.