teapottrainer
teapottrainer is a config-driven library for pretraining and supervised fine-tuning of causal language models. You describe the model, the dataset mix, and the token budgets. TeapotTrainer runs pretraining, then supervised training, and can resume from the last checkpoint.
pip install teapottrainer
Train
from teapottrainer import (
DatasetSpec,
FINEWEB_EDU,
GSM8K,
ModelConfig,
TeapotTrainer,
TrainingConfig,
)
config = TrainingConfig(
model=ModelConfig(
model_type="llama",
hidden_size=768,
layers=12,
heads=12,
kv_heads=4,
intermediate=2048,
seq_len=2048,
),
pretrain_datasets=[
DatasetSpec(source=FINEWEB_EDU, target_tokens=1_000_000_000),
],
sft_datasets=[
DatasetSpec(source=GSM8K, target_tokens=4_000_000),
],
tokenizer="HuggingFaceTB/SmolLM2-135M",
batch_size=4,
grad_accum=8,
lr=5e-4,
)
trainer = TeapotTrainer(config)
trainer.train(fresh=True)
train(fresh=False) continues from checkpoints/resume in the run directory: weights, optimizer, and the sample cursor. Pass init_from to start from an existing Hugging Face model or a local checkpoint instead of randomly initialized weights.
Set sft_at_checkpoints=True to fine-tune a copy of the base model at each pretrain eval and save that copy beside the base weights. Pretraining keeps going from the base model.
Runs are written under ./runs/. Loss and task scores go to TensorBoard and to metrics.jsonl in the run directory.
Chat format
chat_format wraps supervised examples. Pretrain rows stay raw text, including a raw source that you also put in the supervised mix. The default is plain, which is the System / User / Assistant text. The same template is saved with a supervised checkpoint, so Transformers can build the prompt from messages.
config = TrainingConfig(
chat_format="chatml",
tokenizer="HuggingFaceTB/SmolLM2-135M",
sft_datasets=[
DatasetSpec(source=GSM8K, target_tokens=4_000_000),
],
)
chatml is the SmolLM-Instruct template. SmolLM2 already contains <|im_start|> and <|im_end|>, and a supervised checkpoint uses <|im_end|> as the stop token. smollm is the SmolLM2-Instruct template, including its default system line. The other names are llama2, llama3, gemma, mistral, phi, zephyr, alpaca, and vicuna. qwen is another name for chatml.
Model
model_type chooses how the trainer builds a new model. llama and gpt2 are built in. Any other Hugging Face architecture is custom: pass the config class and the model class. The trainer calls the config class with vocab_size, hidden_size, num_hidden_layers, num_attention_heads, intermediate_size, and max_position_embeddings. It does not pass kv_heads. A wrapper can set fields the class needs beyond those six.
from transformers import Qwen2Config, Qwen2ForCausalLM
# Llama. kv_heads defaults to 4. heads must divide hidden_size.
ModelConfig(
model_type="llama",
hidden_size=768,
layers=16,
heads=12,
kv_heads=4,
intermediate=2048,
seq_len=2048,
)
# GPT-2. intermediate and kv_heads are ignored. heads must divide hidden_size.
ModelConfig(
model_type="gpt2",
hidden_size=768,
layers=12,
heads=12,
seq_len=1024,
)
# Qwen2, or any other Hugging Face pair with the same constructor fields.
def qwen_config(**kwargs):
kwargs["num_key_value_heads"] = 2
return Qwen2Config(**kwargs)
ModelConfig(
model_type="custom",
hidden_size=896,
layers=24,
heads=14,
intermediate=4864,
seq_len=2048,
custom_config_cls=qwen_config,
custom_model_cls=Qwen2ForCausalLM,
)
init_from starts from weights you already have, either a Hugging Face id or a local directory. The ModelConfig dimensions have to match that checkpoint.
TrainingConfig(
model=ModelConfig(
model_type="llama",
hidden_size=576,
layers=30,
heads=9,
kv_heads=3,
intermediate=1536,
seq_len=2048,
),
init_from="HuggingFaceTB/SmolLM2-135M",
tokenizer="HuggingFaceTB/SmolLM2-135M",
)
Inference
The finished weights are a normal Transformers checkpoint. After both phases, the run directory is the supervised model. checkpoints/pretrain is the base model from before supervised training, still without a chat template.
With chat_format="chatml" or chat_format="smollm", load that supervised directory and pass messages. The pipeline applies the template that was used for supervised training.
from transformers import pipeline
pipe = pipeline("text-generation", model="./runs/teapot-SmolLM2-135M-4c2004")
pipe([{"role": "user", "content": "Who are you?"}])
plain does not add those control tokens. Format the prompt yourself:
from transformers import AutoModelForCausalLM, AutoTokenizer
ckpt = "./runs/teapot-SmolLM2-135M-4c2004"
tokenizer = AutoTokenizer.from_pretrained(ckpt)
model = AutoModelForCausalLM.from_pretrained(ckpt)
prompt = "User: What is 2 + 2?\n\nAssistant:\n"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Datasets
Preset sources live on the package: FINEWEB_EDU, DCLM_EDU, STACK_EDU, FINEMATH, GSM8K, SQUAD_V2, HOTPOT, SYNTHQA, TEAPOT_CHAT, the SmolTalk subsets, and the reading, tool-calling, and instruction-following evals. A DatasetSpec assigns a source a token budget and a mix weight.
To add a source, build a DatasetSourceConfig with a chat_template_fn that returns a prompt and an answer. The trainer supervises the answer tokens.
Develop
The train.py script in this repository is one example run. It is not part of the installed package.
pip install -e ".[dev]"
pytest
Metadata
Release files for teapottrainer 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| teapottrainer-0.1.2.tar.gz | 51.1 kB | With provenance | Details for teapottrainer-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| teapottrainer-0.1.2-py3-none-any.whl | Python 3 | none | any | Details for teapottrainer-0.1.2-py3-none-any.whl |
Total release size: 104.0 kB
Release files / teapottrainer-0.1.2.tar.gz
| Download URL | teapottrainer-0.1.2.tar.gz |
|---|---|
| Size | 51.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dcfddd35f35c0f47ad5782204ebba672eb7ed44ad754a300cfbc9a11655742da
|
|
BLAKE2b-256 checksum How to use checksums |
a36215b2b8f843abc3ebb31b1614bfcddc4240480c3907ff66b18119d2ed8023
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logRelease files / teapottrainer-0.1.2-py3-none-any.whl
| Download URL | teapottrainer-0.1.2-py3-none-any.whl |
|---|---|
| Size | 52.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9dfac2f81fcee6a88a93f524ef442b70a5bd323b9c57e64132a5037e1a1f48bb
|
|
BLAKE2b-256 checksum How to use checksums |
2d69f588cb4a5c436f68cc981b2a100ce7b917e9e1581fe7846a463a2a201bd0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logSecurity for teapottrainer 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for teapottrainer-0.1.2.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.