GrillyDistil

Temperature-scaled knowledge distillation with SA-KD — optional grilly extension.

Features

• SA-KD Temperature — simulated annealing-based adaptive temperature
• Linear Annealing — fallback T=8 -> T=2 over first 30% of steps
• Expanded Prompts — 50 seed prompts per domain (200 total seeds)
• Distillation Trainer — full training loop with KL-divergence loss
• Compression Synergy — high-T phases produce 2-3x better compression ratios

Quick Start

pip install grillydistil

from grillydistil import SAKDTemperature, DistillationTrainer, PromptGenerator

# SA-KD Temperature
temp = SAKDTemperature(T_init=8.0, alpha=0.97)
for step in range(1000):
    loss = train_step(temperature=temp.current_temperature)
    new_T = temp.step(loss)

# Generate training prompts
gen = PromptGenerator(prompts_per_domain=500)
prompts = gen.generate()  # 2000 total prompts across 4 domains

# Full distillation
trainer = DistillationTrainer(student_model, teacher_model, tokenizer)
losses = trainer.train(dataset, epochs=3)

SA-KD Algorithm

1. Start with T_init=8.0 (high temperature -> smooth teacher logits)
2. Propose T' = T + random perturbation
3. Metropolis acceptance: P = min(1, exp(-dE / T_SA))
4. SA cooling: T_SA *= 0.97 each step
5. Converges to optimal temperature for student capacity

Requirements

• Python 3.12+
• grilly >= 0.4.0
• grillyinference >= 0.1.0
• numpy

License

MIT