cairn-pack-finetune

Fine-tune a model for cairn with LoRA or 4-bit QLoRA, on prepared data only: registered train, val and test versions, as cairn-pack-dataset-prep registers them (or your own JSONL splits registered with those roles). A rerun or a sweep reads the same frozen versions.

A domain pack: YAML, a profile and presets, no code. The training stages come from cairn-operator-training, the evaluation stage (eval.lm_eval: benchmarks and held-out loss for the base model and the candidate, on the GPU box) from cairn-operator-eval-lm.

Which pipeline

sft-own-box is sft without its three box steps (acquire, release-box and the release finalizer) and without the compute_connection input; its gates, legs, timeouts, retries, budget and outputs are the same (a test pins the two files together). On sft-own-box cairn never starts or stops the box, and a cancel does not stop a trainer already running on it: stop the job or the box yourself. Both price their ml legs on a connection with hourly_usd, and both show the estimate on approve-run. The rest of this README describes sft; for sft-own-box skip the box steps and compute_connection, and launch finetune/sft-own-box. eval-adapter-own-box is eval-adapter without its box steps in the same way (section 9).

The sft pipeline

approve-run ⏸ → acquire → smoke → smoke-ok → train → eval → release-box → improved → promote-model ⏸ → promote ⏸
                                                            finally (done, failed or cancelled): release

• Admission, at launch. Before any record exists, cairn's launch door checks the three versions:

  • each has the role its input declares (ADR-0146), so a test set given as training data, or a path instead of a registered version, refuses the launch;
  • the train and val versions carry an intake verdict, and a blocked or invalid one refuses;
  • training data that meets a registered test or golden version refuses (the holdout door);
  • the base model's licence.

  The decision is on the pipeline record and on the first gate's card.

• approve-run: a person reads it, and the GPU estimate, before GPU time is spent.

• acquire: compute/acquire starts the box compute_connection names (an EC2 instance), only after the approval. It stamps the cairn:managed and cairn:release-after lease tags first, then starts the instance and waits up to 600 s for its SSH port (if the port is still closed, acquire succeeds with ssh_ready: false and smoke fails on its own).

• smoke: the sft-train flow on a recorded subsample (smoke_rows, default 64) for smoke_steps (default 5). It proves the data, the tokenizer, the trainer and the packaging cheaply.

• train: the full run. At the end the trainer evaluates on val_version twice: the candidate (eval_loss), and the base model with the adapter switched off (baseline_eval_loss) — the same rows, the same trainer. max_eval_loss fails it fast. package.adapter registers the adapter as a model.

  Both ml legs get a second attempt when the first one's process is killed, and it resumes from that run's checkpoints (see Checkpoints, retries and cancel).

• eval: the sft-eval flow on the same box (see The evaluation): lm-evaluation-harness for the base model and for the base model plus the new adapter with identical settings (eval_preset, default forgetting), and both models' loss on val_version and test_version by one named method. 3-hour timeout, a second attempt after a kill, and a time probe that refuses the leg up front when the run would not fit.

• release-box: compute/release stops the box as soon as eval, the last GPU leg, is done, so it does not bill while a person decides on promotion.

• improved: the candidate's validation loss must be below the base model's.

• promote-model: a person decides, with the losses and the evaluation on the card: the benchmark deltas (candidate − base) with their standard errors, the worst delta, how many metrics regressed beyond their standard error, and both models' validation and test loss. Shown, not gated: a predicate on the deltas comes once real runs have sized the noise.

• promote: artifact/promote points tag at the adapter, after its own approval.

• release: the same compute/release, declared finalizer: true: it runs once the pipeline is done, failed or cancelled, whatever the chain reached, and a box already stopped is a success. It is the backstop for a run that never reached release-box.

The evaluation

The eval leg (cairn-extensions#42) runs cairn-operator-eval-lm's eval.lm_eval stage on the box, after train and before the box stops:

• Identical settings. The base model is loaded once (4-bit for qlora, as it was trained) and scored with lm-evaluation-harness's hf backend; the new adapter is attached to the same weights and scored again with the same tasks, documents, few-shot count, batch size and seeds (eval_seed, default 1234). One harness call per task.
• The preset (eval_preset): forgetting by default — ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande and PIQA at 500 documents each, and MMLU at 10 per subject (570 questions: lm-eval applies a limit per subject). At an accuracy near 0.7, 500 documents give a standard error of about 0.02 per model. forgetting-ifeval adds IFEval; it needs a longer leg than sft's three hours, so it is eval-adapter's default.
• Held-out loss: both models on val_version and test_version, by one method named on the record (eval_lm_eval_split_loss_method: full-sequence mean token NLL, the training chat template, one row per forward pass). It is not the trainer's eval_loss, which batches and pads.
• The time probe. About 50 documents on the base model first, timed apart from lm-eval's own work; if the projected run (both models, both losses) does not fit the leg's 3 hours less 10 %, the leg fails at once with the figures, instead of being killed at the timeout. A clean failure is not retried. Lower eval_preset's size, or copy the pipeline with a longer eval timeout (and time_budget_s, which must equal it, and the budget).
• Results: the leg's findings (cairn remote findings <pl_id>): per task and metric the base and candidate scores with standard errors, the delta and its standard error, the documents scored; and a results file, uploaded beside the run's checkpoints when the box sets CAIRN_CHECKPOINT_BUCKET (s3://<bucket>/<prefix>/<eval run>/eval/lm_eval_results.json) and registered as the leg's report artifact.
• When it cannot measure (the [box] extra missing on the box, a CUDA out-of-memory, a task that will not load), the leg still succeeds with checked: false and the reason on the card: a training run is never lost to its evaluation. A split whose bytes changed since it was registered fails the leg.

Not yet: a generation-quality evaluation, generated answers scored for base and candidate (cairn-extensions#41). The verdict is still validation loss against the base model's.

Run it

Two machines: the control plane (the engine, the data leg, the gates and every record; no torch) and a GPU box reached over SSH, which runs only the trainer. Every command below is the cairn CLI; <pl_id> is a pipeline id (pl_…), <run_id> one leg's run.

Which cairndev. This pack (0.2.0) requires cairndev>=0.3.1,<0.4: 0.3.1 ships compute/acquire and compute/release, prices a leg on a box with hourly_usd, and cancels the job on the box before a cancelled pipeline's release stops it. CI proves it on the newest release in that range.

1. Install

pip install 'cairndev[ssh]' cairn-pack-dataset-prep cairn-pack-finetune   # the control plane (brings boto3)
pip install 'cairn-operator-training[gpu]' 'cairn-operator-eval-lm[box]'  # the GPU box; add ,track-mlflow for MLflow

cairn-operator-eval-lm[box] is lm-evaluation-harness with its hf backend and IFEval's checkers, for the eval leg; the trainer's extra pins the torch, transformers, peft and bitsandbytes both use. The box downloads the base model and the benchmark datasets from the Hugging Face Hub.

To run unreleased changes, install from a checkout of this repository: on the control plane pip install 'cairndev[ssh]' ./packages/cairn-contracts-* ./packages/cairn-operator-* ./packs/cairn-pack-dataset-prep ./packs/cairn-pack-finetune, and on the box pip install ./packages/cairn-contracts-training ./packages/cairn-contracts-training-stages './packages/cairn-operator-training[gpu]' './packages/cairn-operator-eval-lm[box]'. The control plane starts and stops the box through boto3, a dependency of cairn-pack-finetune (cairndev declares it only in an extra). The box must run the same cairn-contracts-training* versions as the control plane, and needs cairn-dataflow-run on its non-interactive SSH PATH (ssh <box> 'which cairn-dataflow-run'), or CAIRN_DATAFLOW_REMOTE_CMD=<venv>/bin/cairn-dataflow-run on the control plane.

2. Durability

export CAIRN_STORAGE=sqlite:///$HOME/.cairn/train.db         # one host, one process
export CAIRN_STORAGE=postgres://user:pass@host:5432/cairn     # several processes: pip install 'cairndev[storage-postgres]'
pip install 'cairndev[runtime-dbos]' && export CAIRN_RUNTIME=dbos   # optional: resume a run at the next boot

Without CAIRN_STORAGE runs live in memory and are gone on restart. The schema migrates when the server first opens the store; upgrade every process that shares one database together. Not Temporal: sft has task: legs (acquire, the two releases, promote), which the Temporal pipeline owner does not run, and a finalizer: step, which it does not implement (sft-own-box has the promote task leg). The launch is refused under CAIRN_RUNTIME=temporal before any record exists (cairn remote launch exits 1 with the reason; cairn pipeline run exits 3). Use inprocess (the default) or dbos. cairn dev sets its own: SQLite at --db (default ~/.cairn/dev.db) and inprocess.

3. The profile, the buckets and the box

cairn ls finetune                                  # lists profiles/finetune-ssh.toml with its installed path
export CAIRN_PROFILE=<that path>
export CAIRN_ARTIFACT_S3_BUCKET=<bucket>           # the registered dataset versions (CAIRN_ARTIFACT_S3_PREFIX optional)
export CAIRN_AUDIT_LOG=$HOME/.cairn/audit.jsonl     # the profile's hash-chained audit log: one file per engine process (default: ./audit.jsonl)
export GPU_SSH_KEY="$(cat ~/.ssh/id_ed25519)"      # read by the engine; cairn never writes it to disk
cairn connect add ssh --name gpu --set host=<box> --set user=<user> \
  --set known_hosts=$HOME/.ssh/known_hosts --secret PRIVATE_KEY=GPU_SSH_KEY \
  --set instance_id=<i-…> --set region=<aws-region> --set hourly_usd=<rate>
cairn connect ls && cairn connect test gpu         # the box must be running for the test

finetune-ssh sends the ml legs to the compute.ssh connection named gpu, in the box's virtualenv, and selects the s3 artifact store, because two machines need a store both can reach. Its header lists every setting. Cancel and the checkpoint pull-back use the connection's PRIVATE_KEY (SSH_PRIVATE_KEY is needed only for a job started under cairndev 0.2.1).

The box is an EC2 instance that sft starts and stops. instance_id and region name it (give it an Elastic IP, so host stays the same across a stop); hourly_usd prices its legs. The control plane needs AWS credentials that allow ec2:DescribeInstances, ec2:CreateTags, ec2:StartInstances and ec2:StopInstances on that instance (the standard AWS chain: an instance role, AWS_PROFILE, or AWS_* variables). sft's compute_connection input names the box to start and stop, and it must be the connection ml_target selects (gpu in finetune-ssh, the input's default). cairndev does not check that the two match yet (cairndev-sh/cairn#216); step 4's explain prints both (acquire's with.connection and each ml leg's compute:). With a mismatch, acquire starts one box while the ml legs run on another, and a cancel does not stop the job on the box the legs ran on. A box without an instance_id fails acquire, and the finalizer's release fails the same way; use sft-own-box for such a box.

On the box the trainer reads CAIRN_CHECKPOINT_BUCKET (checkpoints, and the adapter with artifact_target: s3), CAIRN_CHECKPOINT_PREFIX, AWS_DEFAULT_REGION and, for MinIO, AWS_ENDPOINT_URL. Set them in the box's environment, or set them on the control plane and name them in the profile's remote_env (the header shows the line); the engine forwards each one that is set and refuses a name containing KEY, SECRET, TOKEN or PASSWORD. Credentials stay on the box (an instance role, or its own AWS_*): read access to the artifact bucket, write access to the checkpoint bucket. MLFLOW_TRACKING_URI set on the control plane is forwarded by the engine itself.

4. Start the control plane and preflight

cairn dev --engine-only --port 8471 &                       # a laptop: sign-in dev / dev
cairn remote login -u dev -p dev -e http://localhost:8471
cairn doctor --strict                                       # configuration and reachability
cairn pipeline show finetune/sft                            # legs, gates, and which runtime runs each leg
cairn pipeline explain finetune/sft -i train_version=dataset:ifc-bim-alpaca-train@latest   # acquire's connection = the ml legs' compute

A server runs cairn-server with the same environment plus sign-in (CAIRN_JWT_SECRET, CAIRN_API_USER, CAIRN_API_PASSWORD_HASH from cairn remote hash-password). explain is a dry run: nothing starts and nothing is written.

5. Prepare the data, then a dry run without a GPU

cairn remote launch dataset-prep/prepare --preset ifc-bim-alpaca --watch   # once; see cairn-pack-dataset-prep
cairn remote advance <pl_id> source-ok                                      # a person confirms the source
cairn remote findings <pl_id>                                               # the three versions it registered
cairn remote launch finetune/sft --preset ifc-bim-qwen3-4b -i stub=true --watch

The stub trainer (stub=true) runs every leg, gate and record on the box without a GPU and trains nothing; the eval leg's stub loads no model and reports placeholder numbers marked stub, never checked. Its validation loss is not below its baseline, so improved stops it: a stub adapter never reaches a person who could promote it. With CAIRN_CHECKPOINT_BUCKET set on the box, the smoke leg writes a stub checkpoint there, which shows the box can write to the bucket. It is a real run on the box, so after approve-run it starts the box (real EC2 calls, a few minutes of box time) and stops it after eval; the finalizer finds it stopped.

6. Train, and the three approvals

cairn remote presets finetune/sft
cairn remote launch finetune/sft --preset ifc-bim-qwen3-4b --watch
cairn remote advance <pl_id> approve-run                    # 1. before any GPU time, with the estimate on the card
cairn remote advance <pl_id> promote-model                  # 2. after `improved` (the box is already stopped), with the losses and the eval deltas on the card
cairn remote advance <pl_id> promote                        # 3. artifact/promote's own approval

approve-run's card carries estimate_usd and estimate_basis in its evidence: the worst case the launch door checked, per ml leg (for example train: $0.66/h × 12h on gpu × 2 (2 attempts) = $15.84). acquire is paid_compute: under a profile that requires approval for paid compute it pauses for its own approval after approve-run (cairn remote advance <pl_id> acquire).

Override any input with -i key=value and keep the result with --save-preset <name>. A list input takes JSON (-i 'report_to=["mlflow"]') or a comma string (-i report_to=mlflow). cairn remote pipeline <pl_id> shows which step is waiting. --watch returns when the pipeline pauses at a gate; approvers: ml-owners names who decides, and with sign-in on the signed-in user is recorded.

The launch door refuses, before any record exists (cairn remote launch exits 1 with the reason): a test version given as train_version, a train or val version without an intake verdict, training data that meets a registered test or golden version, and a gated base model without -i accept_license=true.

7. Watch, logs, findings, replay

cairn remote watch                                  # a live view of the runs (Ctrl-C)
cairn remote pipeline <pl_id>                       # the durable record: plan, step states, admission, job handles
cairn remote status <run_id>                        # one leg's run
cairn remote logs <run_id> -f                       # its event log, followed live (--after-seq N resumes)
cairn remote findings <pl_id>                       # every step's findings: losses, steps, adapter URI, versions
cairn replay <run_id> --source db                   # the recorded events as a timeline; runs nothing again

8. Checkpoints, retries and cancel

• The trainer saves a checkpoint every save_steps (50), keeps save_total_limit (2), and mirrors each one to s3://<CAIRN_CHECKPOINT_BUCKET>/<prefix>/<run-id>/. It writes under the leg's job directory on the box (train-output/<output_name>), so two runs never share checkpoints. The published adapter is the adapter files, without the checkpoints.
• smoke, train and eval declare retry: {max_attempts: 2} (an ml leg otherwise gets one attempt). A killed eval starts its evaluation again; it has nothing to resume from. When an attempt's process dies without writing its exit code (the OOM killer, a kill on the box), the engine reports it retriable and the leg runs once more under the same run; the trainer pulls that run's newest checkpoint from CAIRN_CHECKPOINT_BUCKET and continues from its step. A leg that fails on its own error (a clean non-zero exit) is not retried, and neither is a poll that cannot reach the box. Without a checkpoint bucket the retry starts again from step 0.
• Budget: governance.budget: {usd: 25} on the sft series, sized to the priced legs' worst case. Before any record exists, the launch door checks hourly_usd × timeout × 2 attempts for each ml leg against what the series has left: (30 min for smoke + 12 h for train + 3 h for eval) × 2 = 31 box-hours, $20.46 at $0.66/h. A box dearer than $0.80/h is refused at launch with the figures; give it a larger budget, or a copy of the pipeline with shorter timeouts. Each attempt reserves one attempt's figure before it is submitted, so an attempt the budget cannot cover fails the leg instead of starting, and every attempt settles its own box time (Run.cost.gpu_usd), failed or not. Not priced: the minutes between acquire and smoke (the boot) and between eval and release-box; a connection without hourly_usd is "not priced", never $0. The launch checks the estimate against what the series has left, live holds included: while one sft attempt holds its train figure ($7.92), a second sft launch in the same series sees $17.08 left and is refused. Run one at a time, or give the series a larger budget.
• Cancel: cairn remote cancel-pipeline <pl_id> stops the pipeline: no further leg starts. If smoke, train or eval is running on the box, its job is cancelled first (the trainer is killed, and the leg settles cancelled, not retried), then the finalizer release stops the box. A job cancel that fails is logged and audited (leg_job_cancel, sent: false), and the box is stopped anyway. The record reads cancelled once the release has settled.
• The box: sft starts it after approve-run and stops it after eval, and the finalizer stops it on every outcome that reaches it. Two gaps: a cancel that lands while acquire is still starting the box can leave it running (core cancels flow legs, not a task leg in flight), and a release-box that fails after eval (a transient EC2 error) fails the pipeline before improved: the candidate is registered, but promotion is manual. A failed eval (the time probe's refusal, or a killed second attempt) fails the pipeline the same way, after train registered the adapter; finetune/eval-adapter can evaluate it later. In every case the finalizer or the lease (below) stops the box. acquire stamps a lease of 32 hours (cairn:release-after) before the start: the retried worst case, 2 attempts × (30 min + 12 h + 3 h) = 31 h, plus an hour, so a sweep never stops the box under a retried train or eval. If the control plane dies, cairn compute sweep lists the box once the lease has passed and --apply stops it. The trade-off: until then an orphaned box keeps billing, up to 32 h (about $21 at $0.66/h), so run the sweep from a cron, or stop the box yourself. cairn compute status gpu shows whether the box is up and what it costs per hour.

9. Evaluate an adapter you already have: finetune/eval-adapter

adapter-ref → approve-run ⏸ → acquire → eval → release-box → evaluated
                                     finally (done, failed or cancelled): release

The same eval leg on an adapter trained earlier, with no training: adapter_uri (an s3://…/ prefix as artifact_target: s3 publishes it, hf://<repo>, a Hugging Face Hub id, or a directory on the box), base_model, method, and the registered val_version and test_version (their roles are checked at launch, as in sft).

cairn remote presets finetune/eval-adapter
cairn remote launch finetune/eval-adapter --preset ifc-bim-qwen3-4b --watch   # the published IFC/BIM adapter
cairn remote advance <pl_id> approve-run                    # before any GPU time, with the estimate on the card
cairn remote findings <pl_id>                               # scores, deltas, standard errors, sample sizes, losses

• adapter-ref: a promoted tag (<name>@<tag>, e.g. ifc-bim-qwen3-4b@candidate) is refused before any box time: cairndev resolves dataset: refs in a leg's inputs but not artifact tags yet, and the box has no registry. Pass the adapter's URI instead (cairn remote findings of the run that trained it: train's train_qlora_adapter_uri).
• eval: a 6-hour timeout with forgetting-ifeval (the forgetting tasks plus IFEval, all 541 prompts) by default, two attempts, the same time probe.
• evaluated: the run fails unless the leg measured both models (checked): a missing harness on the box, a failure while scoring, or the stub tier ends here.
• Budget: {usd: 10} on the eval-adapter series: 6 h × 2 attempts = 12 box-hours, $7.92 at $0.66/h (a box dearer than $0.83/h needs more); the lease is 13 hours.

finetune/eval-adapter-own-box is the same without acquire, release-box and release, for a box you start and stop yourself.

10. Optional: an admission policy

docs/policies/require-dedup.yaml is an example, not active: in a production environment, each dataset version's intake verdict must show that exact and near deduplication ran. The holdout door needs no policy: it runs on every launch.

mkdir -p ~/.cairn/policies                       # or $CAIRN_POLICIES_DIR
cp <the pack>/docs/policies/require-dedup.yaml ~/.cairn/policies/
cairn validate ~/.cairn/policies/require-dedup.yaml

<the pack> is this directory in a checkout, or the installed cairn_pack_finetune directory (the one cairn ls finetune shows the profile in).

What the door decided is on the pipeline record (admission, with policy@<digest>) and on the first gate's card. A check a policy names must be one the door knows; it refuses every launch under an unknown one, holdout included.

MLflow (optional): MLFLOW_TRACKING_URI on the control plane with -i 'report_to=["mlflow"]' (or -i report_to=mlflow) records the run; the box installs cairn-operator-training[gpu,track-mlflow] and must reach the URI. On one machine: use finetune/sft-own-box (sft needs an EC2 instance to start and stop) with the default profile and no ssh settings; dataset-prep still registers its versions through the s3 store. The default profile binds no audit sink: bind "audit.sink" = "audit/store" in your own profile, or use the core secure profile, to keep the audit log.

Presets and profiles

Support

Apache-2.0 — see LICENSE.