Anthracite v1.5

Universal AI training & fine-tuning framework — train text generation and embedding models from scratch or fine-tune them with one Python call.

---

Table of Contents

1. Installation
2. Quick Start
3. Tokenizer Options
4. Supervised Fine-Tuning (SFT)
5. Dataset Formats
6. HF Dataset Streaming
7. Embedding Models
8. Generation & Inference
9. Fine-Tuning an Existing Model
10. Configuration Reference
11. PyPI Upload Guide
12. What Changed in v1.5

---

Installation

pip install anthracite                    # core (PyTorch required separately)
pip install anthracite[hf]               # + HuggingFace tokenizers, datasets, hub
pip install anthracite[all]              # everything above

PyTorch is not bundled (too large). Install it from https://pytorch.org first.

---

Quick Start

from anthracite import train, generate

# 1. Train a 20 M parameter text model
train(
    model_name  = "MyGPT-20M",
    dataset     = "my_data.jsonl",   # local file, HF dataset id, or directory
    tokens      = 100_000_000,        # training token budget
    params      = "20M",             # target parameter count
    context_length = 512,
    tokenizer   = "bundled",         # ← use built-in GPT-2 tokenizer (new v1.5)
    device      = "auto",
    precision   = "auto",
)

# 2. Generate text
text = generate("./models/MyGPT-20M", "Once upon a time")
print(text)

---

Tokenizer Options (new in v1.5)

Anthracite v1.5 gives you full control over the tokenizer. Four modes:

1. Train from scratch (default, v1.0 behaviour)

train(..., tokenizer=None)   # trains a byte-level BPE tokenizer on your dataset

2. Built-in bundled tokenizer

A GPT-2 style 50 k BPE tokenizer ships with Anthracite. Use it to skip the tokenizer training step:

train(..., tokenizer="bundled")

3. HuggingFace tokenizer by repo id

# Load by HF Hub repo id
train(..., tokenizer="gpt2")
train(..., tokenizer="mistralai/Mistral-7B-v0.1")
train(..., tokenizer="meta-llama/Llama-2-7b-hf")

# Load a specific file: repo/model/filename
train(..., tokenizer="bert-base-uncased/tokenizer.json")

Requires: pip install transformers (or tokenizers + huggingface_hub)

4. Local tokenizer directory / file

train(..., tokenizer="./my_tokenizer/")     # directory with tokenizer.json
train(..., tokenizer="./tokenizer.json")    # exact file path

5. Live tokenizer object (any library)

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2")
train(..., tokenizer=tok)

# SentencePiece, tiktoken, tokenizers, etc. all work via duck-typing:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
train(..., tokenizer=enc)

Standalone tokenizer loading

from anthracite import load_tokenizer

tok = load_tokenizer("bundled")            # built-in
tok = load_tokenizer("gpt2")              # HF Hub
tok = load_tokenizer("./my_tokenizer/")   # local path

ids = tok.encode("Hello world!")
text = tok.decode(ids)
print(text)   # "Hello world!"

---

Supervised Fine-Tuning (SFT)

SFT mode trains the model to generate the output given the input, and masks input tokens from the loss so the model doesn't just learn to repeat the prompt.

from anthracite import train

train(
    model_name     = "MyChat",
    dataset        = "instructions.jsonl",
    tokens         = 50_000_000,
    params         = "20M",
    objective      = "sft",           # ← enable SFT mode
    sft_mask_input = True,            # mask prompt tokens (default True)
    sft_format     = "auto",          # auto-detect format (default)
    tokenizer      = "bundled",
)

Explicit format

train(..., sft_format="alpaca")      # Alpaca instruction format
train(..., sft_format="sharegpt")    # ShareGPT multi-turn
train(..., sft_format="chatML")      # ChatML messages format
train(..., sft_format="openai")      # prompt / completion
train(..., sft_format="qa")          # question / answer
train(..., sft_format="text")        # raw text, no masking

---

Dataset Formats & Smart Key Detection

Anthracite v1.5 automatically detects your dataset format. You don't need to specify column names — the SmartKeyExtractor peeks at a sample of records, scores every known format, and picks the best match.

Supported formats

Format	Keys detected	Input	Output
Alpaca	instruction, input, output	instruction + input	output
Alpaca-short	instruction, output	instruction	output
OpenAI	prompt, completion	prompt	completion
Prompt-Response	prompt, response	prompt	response
QA	question, answer	question	answer
Generic IO	input, output	input	output
AI-User	user, ai	user	ai
ShareGPT	conversations list	full conversation	(no mask)
ChatML	messages list	full conversation	(no mask)
Raw text	text, content, body	full text	(no mask)
2-column generic	any 2 string keys	first key	second key

Alpaca format example

{"instruction": "Translate to French.", "input": "Hello world", "output": "Bonjour le monde"}
{"instruction": "Summarise this.", "input": "Long text...", "output": "Short summary."}

ShareGPT / ChatML format example

{"conversations": [{"from": "human", "value": "Hi"}, {"from": "gpt", "value": "Hello!"}]}
{"messages": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello!"}]}

OpenAI / prompt-completion format

{"prompt": "Tell me a joke.", "completion": "Why did the chicken..."}

Raw text format

{"text": "Once upon a time in a land far away..."}

---

HF Dataset Streaming (new in v1.5)

When you pass a Hugging Face dataset id, Anthracite now streams the data instead of downloading the whole dataset first. This means:

• Training starts immediately (no multi-GB download wait)
• Large datasets (100 GB+) fit on machines with little disk space
• Memory usage stays constant regardless of dataset size

train(
    model_name  = "MyGPT",
    dataset     = "HuggingFaceFW/fineweb",   # 15 TB — streams fine!
    tokens      = 1_000_000_000,
    params      = "100M",
    hf_streaming = True,    # default in v1.5
)

To download fully (v1.0 behaviour):

train(..., hf_streaming=False)

---

Embedding Models

from anthracite import train, embed, similarity

# Train a sentence embedding model
train(
    model_name  = "MyEmbedder",
    model_type  = "embedding",
    dataset     = "corpus.txt",
    tokens      = 20_000_000,
    params      = "30M",
    context_length = 256,
    objective   = "contrastive",  # or "mlm" for pre-training stage
    tokenizer   = "bundled",
)

# Use it
vecs = embed("./models/MyEmbedder", ["sentence A", "sentence B"])
score = similarity(vecs[0], vecs[1])
print(f"Similarity: {score:.3f}")

---

Generation & Inference

from anthracite import generate

# Simple generation
text = generate("./models/MyGPT-20M", "The quick brown fox")
print(text)

# With options
text = generate(
    "./models/MyGPT-20M",
    "Tell me about space",
    max_new_tokens = 300,
    temperature    = 0.8,
    top_p          = 0.9,
)

---

Fine-Tuning an Existing Model

from anthracite import finetune

finetune(
    model      = "./models/MyGPT-20M",     # base model path or HF repo id
    dataset    = "instructions.jsonl",
    tokens     = 10_000_000,
    objective  = "sft",
    tokenizer  = None,   # None = use base model's tokenizer (default)
    output_dir = "./models/MyGPT-20M-SFT",
)

You can also fine-tune from a HuggingFace Hub model:

finetune(
    model   = "your-username/MyGPT-20M",   # HF Hub repo id
    dataset = "instructions.jsonl",
    tokens  = 5_000_000,
)

---

Configuration Reference

All parameters for train() and finetune():

Parameter	Default	Description
model_name	"anthracite-model"	Output model name / directory
model_type	"text_gen"	"text_gen" or "embedding"
dataset	—	File path, HF dataset id, or list of strings
tokens	10_000_000	Training token budget (int or "100M")
params	"20M"	Target parameters (int or "20M")
context_length	512	Sequence length
vocab_size	32768	Vocabulary size (when training tokenizer from scratch)
tokenizer	None	None / "bundled" / HF id / path / object
device	"auto"	"auto" / "cuda" / "mps" / "cpu"
precision	"auto"	"auto" / "fp32" / "bf16" / "fp16"
batch_size	"auto"	Effective batch size or "auto"
learning_rate	3e-4	Peak learning rate
output_dir	./models/<name>	Where to save the model
objective	"auto"	"auto" / "causal_lm" / "mlm" / "contrastive" / "sft"
sft_mask_input	True	Mask prompt tokens from loss (SFT only)
sft_format	"auto"	Dataset format hint (SFT only)
hf_streaming	True	Stream HF datasets (don't download fully)
checkpoint_interval	5000	Save checkpoint every N steps
keep_last_checkpoints	3	Number of recent checkpoints to keep
seed	42	Random seed
val_split	0.01	Fraction of data held out for validation
gradient_accumulation	None	Accumulation steps (auto if None)
weight_decay	0.1	AdamW weight decay
grad_clip	1.0	Gradient clipping norm
warmup_ratio	0.02	Fraction of steps used for LR warm-up
num_workers	0	DataLoader worker processes

---

PyPI Upload Guide

Follow these steps to publish your own fork or a new release to PyPI.

1. Install build tools

pip install build twine

2. Build the package

cd anthracite-pkg        # the directory containing pyproject.toml
python -m build          # creates dist/anthracite_ai-1.5.0.tar.gz and .whl

3. Test on TestPyPI first (recommended)

# Upload to TestPyPI
twine upload --repository testpypi dist/*

# Install from TestPyPI to verify
pip install --index-url https://test.pypi.org/simple/ anthracite

4. Upload to PyPI

twine upload dist/*
# Enter your PyPI username and password (or API token)

5. Using an API token (safer than password)

# Create a token at https://pypi.org/manage/account/token/
twine upload dist/* -u __token__ -p pypi-<your-token-here>

6. Automate with GitHub Actions

Create .github/workflows/publish.yml:

name: Publish to PyPI
on:
  release:
    types: [published]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install build twine
      - run: python -m build
      - run: twine upload dist/*
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}

---

What's New in v1.5

Tokenizer overhaul

• tokenizer="bundled" — GPT-2 tokenizer ships with Anthracite, no training needed.
• External tokenizer support — load from HF Hub, local path, or pass any live tokenizer object (HuggingFace, tiktoken, SentencePiece, etc.).
• Bug fix — encode() and decode() no longer require self to be passed manually; the tokenizer works correctly as a standalone object.

Smart SFT format detection (SmartKeyExtractor)

• Automatically detects Alpaca, ShareGPT, ChatML, OpenAI, QA, and 10+ other formats.
• Correctly identifies input (prompt) vs output (answer) fields.
• Masks input tokens from the training loss so the model learns to generate answers, not repeat prompts.

HF Dataset streaming

• Hugging Face datasets stream by default (hf_streaming=True).
• Training starts immediately without downloading multi-GB datasets.
• Works with any HF dataset, including terabyte-scale ones.

PyPI-ready packaging

• pyproject.toml with proper metadata, optional dependencies, and entry points.
• pip install anthracite and pip install anthracite[hf] work out of the box.

---

License

MIT License — see LICENSE for details.