Token Factory training utilities

Platform-owned training utilities for Saturn Cloud's Token Factory product — a no-code LoRA fine-tuning service.

This package runs inside a Token Factory fine-tuning job pod. It is not intended as a general-purpose fine-tuning library; the runtime contract (env vars, NFS layout, Atlas callback) is specific to the Token Factory platform. See DESIGN.md for the full specification.

Status: early development. The API and CLI surface are not yet stable.

What it does

A single Python entrypoint (saturn-tokenfactory-train, or python -m saturn_tokenfactory) that:

• Reads hyperparameters from environment variables set by Atlas.
• Loads a base model (Llama 3 / Mistral / Qwen) with family-aware defaults.
• Applies LoRA adapters via PEFT.
• Loads a dataset from an NFS-mounted directory (.jsonl, conversational / instruction / text formats).
• Runs SFT with trl.SFTTrainer.
• Writes config.json, LoRA checkpoints, and manifest.json to an NFS-mounted output directory.
• Posts an artifact-registration callback to Atlas on completion.
• Optionally logs to MLflow if MLFLOW_TRACKING_URI is set.

Out of scope

LoRA SFT only. No full fine-tunes, no DPO/RLHF, no multi-GPU, no serving. See DESIGN.md §14 for the explicit out-of-scope list.

Development

make conda-update      # create/update the conda env
make check-format      # black + isort (read-only check)
make format-backend    # black + isort (apply)
make flake8 mypy       # lint + type-check
make test-backend      # unit tests
make lint-backend      # full lint chain