traintrack-ai 0.1.6

TrainTrack Client: PyTorch training-time evaluation

TrainTrack

Training-time evaluation and win-rate tracking for LLMs.

TrainTrack helps you monitor model behavior during training by running automated LLM-as-a-Judge evaluations on every checkpoint.

Features

• 🚀 Real-time Metrics: Get immediate feedback on conciseness, helpfulness, and reasoning quality.
• 📊 Win-rate Tracking: Automatically track win-rates against an anchor checkpoint (baseline) or the previous step.
• 📚 Built-in Benchmarks: Integrated support for GPQA, MMLU-Pro, IFEval, and TruthfulQA.
• 🛠️ Seamless Integration: Works with standard PyTorch loops and HuggingFace Trainer.

Quick Installation

pip install traintrack-ai

Minimal Example

from traintrack import TrainTrackHook

# 1. Initialize the hook
hook = TrainTrackHook(
    model=model,
    tokenizer=tokenizer,
    run_name="my-first-run",
    datasets=["reasoning", "helpfulness"]
)

# 2. Capture a baseline (optional)
hook.capture_anchor()

# 3. Add to your training loop
for step, batch in enumerate(train_dataloader):
    # ... training logic ...

    hook.step(step)

Documentation

For full documentation and advanced configuration (custom metrics, rubrics, and category-based evaluation), visit: github.com/traintrack/traintrack

Tinker Integration

TrainTrack includes a Tinker inline evaluator integration that can be used with evaluator_builders:

from traintrack import BuildTrainTrackTinkerEvaluator

traintrack_builder = BuildTrainTrackTinkerEvaluator(
    run_name="my-tinker-run",
    categories=["reasoning", "helpfulness"],
    model_name="Qwen/Qwen2.5-1.5B-Instruct",
    eval_every_steps=10,
)

# Add `traintrack_builder` to your Tinker `evaluator_builders` list.

If you're using a low-level custom training loop, call: evaluator.evaluate_training_step(training_client=..., step=...) to let TrainTrack handle scheduling + snapshot + sampling + ingest.

For a runnable single-eval smoke test, see: Canary/examples/tinker_sampling_evaluator_example.py

For low blocking overhead during training, wire a small frequent evaluator into evaluator_builders and larger suites into infrequent_evaluator_builders.

For a full training run example (real training + eval every N steps), see: Canary/examples/tinker_supervised_traintrack_training.py

For a low-level Tinker quickstart-style example (Pig Latin with manual forward/backward updates + TrainTrack eval every N steps), see: Canary/examples/tinker_pig_latin_with_traintrack.py

For a real NoRobots finetune with TrainTrack evaluation on the NoRobots test set (helpfulness metric, criteria + pairwise_anchor), see: Canary/examples/tinker_norobots_traintrack_finetune.py