Skip to main content

teapottrainer

teapottrainer is a config-driven library for pretraining and supervised fine-tuning of causal language models. You describe the model, the dataset mix, and the token budgets. TeapotTrainer runs pretraining, then supervised training, and can resume from the last checkpoint.

pip install teapottrainer

Train

from teapottrainer import (
    DatasetSpec,
    FINEWEB_EDU,
    GSM8K,
    ModelConfig,
    TeapotTrainer,
    TrainingConfig,
)

config = TrainingConfig(
    model=ModelConfig(
        model_type="llama",
        hidden_size=768,
        layers=12,
        heads=12,
        kv_heads=4,
        intermediate=2048,
        seq_len=2048,
    ),
    pretrain_datasets=[
        DatasetSpec(source=FINEWEB_EDU, target_tokens=1_000_000_000),
    ],
    sft_datasets=[
        DatasetSpec(source=GSM8K, target_tokens=4_000_000),
    ],
    tokenizer="HuggingFaceTB/SmolLM2-135M",
    batch_size=4,
    grad_accum=8,
    lr=5e-4,
)

trainer = TeapotTrainer(config)
trainer.train(fresh=True)

train(fresh=False) continues from checkpoints/resume in the run directory: weights, optimizer, and the sample cursor. Pass init_from to start from an existing Hugging Face model or a local checkpoint instead of randomly initialized weights.

Set sft_at_checkpoints=True to fine-tune a copy of the base model at each pretrain eval and save that copy beside the base weights. Pretraining keeps going from the base model.

Runs are written under ./runs/. Loss and task scores go to TensorBoard and to metrics.jsonl in the run directory.

Chat format

chat_format wraps supervised examples. Pretrain rows stay raw text, including a raw source that you also put in the supervised mix. The default is plain, which is the System / User / Assistant text. The same template is saved with a supervised checkpoint, so Transformers can build the prompt from messages.

config = TrainingConfig(
    chat_format="chatml",
    tokenizer="HuggingFaceTB/SmolLM2-135M",
    sft_datasets=[
        DatasetSpec(source=GSM8K, target_tokens=4_000_000),
    ],
)

chatml is the SmolLM-Instruct template. SmolLM2 already contains <|im_start|> and <|im_end|>, and a supervised checkpoint uses <|im_end|> as the stop token. smollm is the SmolLM2-Instruct template, including its default system line. The other names are llama2, llama3, gemma, mistral, phi, zephyr, alpaca, and vicuna. qwen is another name for chatml.

Model

model_type chooses how the trainer builds a new model. llama and gpt2 are built in. Any other Hugging Face architecture is custom: pass the config class and the model class. The trainer calls the config class with vocab_size, hidden_size, num_hidden_layers, num_attention_heads, intermediate_size, and max_position_embeddings. It does not pass kv_heads. A wrapper can set fields the class needs beyond those six.

from transformers import Qwen2Config, Qwen2ForCausalLM

# Llama. kv_heads defaults to 4. heads must divide hidden_size.
ModelConfig(
    model_type="llama",
    hidden_size=768,
    layers=16,
    heads=12,
    kv_heads=4,
    intermediate=2048,
    seq_len=2048,
)

# GPT-2. intermediate and kv_heads are ignored. heads must divide hidden_size.
ModelConfig(
    model_type="gpt2",
    hidden_size=768,
    layers=12,
    heads=12,
    seq_len=1024,
)

# Qwen2, or any other Hugging Face pair with the same constructor fields.
def qwen_config(**kwargs):
    kwargs["num_key_value_heads"] = 2
    return Qwen2Config(**kwargs)

ModelConfig(
    model_type="custom",
    hidden_size=896,
    layers=24,
    heads=14,
    intermediate=4864,
    seq_len=2048,
    custom_config_cls=qwen_config,
    custom_model_cls=Qwen2ForCausalLM,
)

init_from starts from weights you already have, either a Hugging Face id or a local directory. The ModelConfig dimensions have to match that checkpoint.

TrainingConfig(
    model=ModelConfig(
        model_type="llama",
        hidden_size=576,
        layers=30,
        heads=9,
        kv_heads=3,
        intermediate=1536,
        seq_len=2048,
    ),
    init_from="HuggingFaceTB/SmolLM2-135M",
    tokenizer="HuggingFaceTB/SmolLM2-135M",
)

Inference

The finished weights are a normal Transformers checkpoint. After both phases, the run directory is the supervised model. checkpoints/pretrain is the base model from before supervised training, still without a chat template.

With chat_format="chatml" or chat_format="smollm", load that supervised directory and pass messages. The pipeline applies the template that was used for supervised training.

from transformers import pipeline

pipe = pipeline("text-generation", model="./runs/teapot-SmolLM2-135M-4c2004")
pipe([{"role": "user", "content": "Who are you?"}])

plain does not add those control tokens. Format the prompt yourself:

from transformers import AutoModelForCausalLM, AutoTokenizer

ckpt = "./runs/teapot-SmolLM2-135M-4c2004"
tokenizer = AutoTokenizer.from_pretrained(ckpt)
model = AutoModelForCausalLM.from_pretrained(ckpt)

prompt = "User: What is 2 + 2?\n\nAssistant:\n"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Datasets

Preset sources live on the package: FINEWEB_EDU, DCLM_EDU, STACK_EDU, FINEMATH, GSM8K, SQUAD_V2, HOTPOT, SYNTHQA, TEAPOT_CHAT, the SmolTalk subsets, and the reading, tool-calling, and instruction-following evals. A DatasetSpec assigns a source a token budget and a mix weight.

To add a source, build a DatasetSourceConfig with a chat_template_fn that returns a prompt and an answer. The trainer supervises the answer tokens.

Develop

The train.py script in this repository is one example run. It is not part of the installed package.

pip install -e ".[dev]"
pytest

Metadata

Release files for teapottrainer 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for teapottrainer 0.1.2
File Size Uploaded Provenance Actions
teapottrainer-0.1.2.tar.gz 51.1 kB With provenance Details for teapottrainer-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for teapottrainer 0.1.2
File Interpreter ABI Platform Actions
teapottrainer-0.1.2-py3-none-any.whl Python 3 none any Details for teapottrainer-0.1.2-py3-none-any.whl

Total release size: 104.0 kB

Release files / teapottrainer-0.1.2.tar.gz

Download URL teapottrainer-0.1.2.tar.gz
Size 51.1 kB
Tags Source
SHA-256 checksum
How to use checksums
dcfddd35f35c0f47ad5782204ebba672eb7ed44ad754a300cfbc9a11655742da
BLAKE2b-256 checksum
How to use checksums
a36215b2b8f843abc3ebb31b1614bfcddc4240480c3907ff66b18119d2ed8023
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / teapottrainer-0.1.2-py3-none-any.whl

Download URL teapottrainer-0.1.2-py3-none-any.whl
Size 52.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9dfac2f81fcee6a88a93f524ef442b70a5bd323b9c57e64132a5037e1a1f48bb
BLAKE2b-256 checksum
How to use checksums
2d69f588cb4a5c436f68cc981b2a100ce7b917e9e1581fe7846a463a2a201bd0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Security for teapottrainer 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for teapottrainer-0.1.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page