zen-trainer 0.1.0

Training framework for Zen Coder models - fine-tune 4B to 1T parameter models on agentic coding data

Zen Trainer

Training framework for Zen Coder models - fine-tune 4B to 1T parameter models on agentic coding data.

Supported Models

Model	Base	Size	VRAM (QLoRA)	Context	License
Zen Coder 4B	Qwen3-4B-Instruct	4B	8 GB	32K	Apache 2.0
Zen Coder 24B	Devstral-Small-2-24B	24B	24 GB	256K	Apache 2.0
Zen Coder 123B	Devstral-2-123B	123B	128 GB	256K	Mistral Research
Zen Coder MAX	GLM-4.7 (358B MoE)	358B	180 GB	200K	GLM-4 License
Zen Coder ULTRA	Kimi-K2 (1T MoE)	1T	400 GB	128K	MIT

Installation

# Basic installation
pip install zen-trainer

# With MLX support (Apple Silicon)
pip install zen-trainer[mlx]

# With Unsloth (2x faster NVIDIA training)
pip install zen-trainer[unsloth]

# With DeepSpeed (multi-GPU)
pip install zen-trainer[deepspeed]

# With evaluation suite
pip install zen-trainer[eval]

# Everything
pip install zen-trainer[all]

Quick Start

Training

from zen_trainer import ZenTrainer

# Train Zen Coder 4B on your data
trainer = ZenTrainer(
    model_key="qwen3-4b",
    dataset_path="path/to/your/dataset",
    output_dir="./output/zen-coder-4b",
)
trainer.train()

Benchmarking

from zen_trainer import ZenBenchmark

# Benchmark against SoTA models
bench = ZenBenchmark(
    model_path="./output/zen-coder-4b",
    model_key="qwen3-4b",
)
results = bench.run_all()
bench.compare_to_baseline()

Command Line

# Train
zen-train --model qwen3-4b --dataset ./data --output ./output

# Benchmark
zen-benchmark --model ./output/zen-coder-4b --suite all

Training Costs

For 3.35M samples (8.47B tokens) on 8xH200 @ $35/hr:

Model	Cloud Hours	Cloud Cost	Local (Mac Studio 512GB)
Zen Coder 4B	9h	$326	2 days (FREE)
Zen Coder 24B	23h	$814	5 days (FREE)
Zen Coder 123B	62h	$2,171	13 days (FREE)
Zen Coder MAX	116h	$4,071	19 days (FREE)
Zen Coder ULTRA	310h	$10,856	Too large

✓ = Fits in 128GB (M3 Ultra / single GPU node) ◆ = Fits Mac Studio 512GB or 8xH200

Backends

The trainer automatically selects the best backend:

Backend	Hardware	Speed	Models
MLX	Apple Silicon	1x	4B, 24B, MAX
Unsloth	NVIDIA GPU	2x	4B, 24B, 123B, MAX
DeepSpeed	Multi-GPU	1x	All (required for ULTRA)

12 ARC Benchmarks

Evaluation suite based on GLM-4.5:

Agentic:

• TAU-Bench (tool-agent-user interaction)
• BFCL V3 (Berkeley Function Call Leaderboard)
• BrowseComp (web browsing agent)

Reasoning:

• MMLU-Pro, AIME-24, MATH-500, SciCode
• GPQA, HLE (Humanity's Last Exam)
• LiveCodeBench

Coding:

• SWE-bench Verified (real GitHub issues)
• Terminal-Bench (terminal environment tasks)

Dataset

Models are designed for the Zen Agentic Dataset:

Metric	Value
Total Tokens	8.47 billion
Training Samples	3.35 million
Validation Samples	100,000
Size	~27 GB
Time Span	15 years (2010-2025)

Data Composition

• 29% Claude Code debug sessions (real agentic programming)
• 23% Claude conversations and interactions
• 48% Git history (commits, diffs, source files)

Model Architecture

Hyperparameters

Model	LoRA r	LoRA α	Batch	LR	Epochs
4B	64	128	4	2e-4	2
24B	32	64	2	1e-4	2
123B	16	32	1	5e-5	1
MAX	16	32	1	5e-6	1
ULTRA	8	16	1	1e-6	1

API Reference

ZenTrainer

trainer = ZenTrainer(
    model_key: str,              # Model identifier (qwen3-4b, devstral-24b, etc.)
    dataset_path: str,           # HuggingFace dataset or local path
    output_dir: str,             # Output directory for checkpoints
    backend: str = "auto",       # mlx, unsloth, deepspeed, or auto
    epochs: int = None,          # Override default epochs
    batch_size: int = None,      # Override default batch size
    learning_rate: float = None, # Override default learning rate
)

trainer.train()           # Start training
trainer.save_model()      # Save final checkpoint
trainer.push_to_hub(repo) # Push to HuggingFace

ZenBenchmark

bench = ZenBenchmark(
    model_path: str,             # Path to model checkpoint
    model_key: str,              # Model identifier for config
    benchmarks: list = None,     # Specific benchmarks or None for all
)

results = bench.run_all()        # Run all 12 ARC benchmarks
bench.run_agentic()              # TAU-Bench, BFCL V3, BrowseComp
bench.run_reasoning()            # MMLU, AIME, MATH, etc.
bench.run_coding()               # SWE-bench, Terminal-Bench
bench.compare_to_baseline()      # Compare to SoTA models

Model Configurations

from zen_trainer import get_model_config, list_models_by_vram, estimate_training_cost

# Get model config
cfg = get_model_config("glm47-358b")
print(cfg.vram_qlora)  # 180 GB

# List models for your hardware
models = list_models_by_vram(128)  # Models that fit 128GB

# Estimate training cost
cost = estimate_training_cost("devstral-24b", num_samples=100000)
print(cost)  # {'hours_estimate': (2.0, 4.0), 'cost_estimate_usd': (70, 140), ...}

Related Projects

• Zen Agentic Dataset - Training data
• Zen Coder Models - Fine-tuned models
• GLM Simple Evals - Evaluation toolkit
• Hanzo MCP - Model Context Protocol tools
• Hanzo AI - AI infrastructure platform

Citation

@software{zen_trainer,
  author = {Kelling, Zach},
  title = {Zen Trainer: Fine-tuning Framework for Agentic Coding Models},
  year = {2025},
  publisher = {Zoo Labs Foundation},
  url = {https://github.com/zenlm/zen-trainer}
}

License

Apache 2.0

---

Maintainer: [e-mail]