syvain-training-utils

Internal Syvain helpers for small, explicit ML training runs. No secret sauce here, just shared runtime, device-diagnostic, and checkpoint patterns.

Install

uv add syvain-training-utils

Runtime setup

from syvain_training_utils import (
    generate_run_id,
    require_torch_compile_toolchain,
    select_device,
)

run_id = generate_run_id()
device = select_device()
require_torch_compile_toolchain()

Load a project-042 initialization bundle

Create the uncompiled BF16 model first, then verify and load the immutable model-only state before constructing the optimizer:

import torch

from syvain_training_utils import load_initialization_bundle_variant

# `storage_config`, the manifest pins, and `model_config` come from the sealed
# project-042 run contract. Construct `model` in BF16 without compiling it.
initialization = load_initialization_bundle_variant(
    storage_config=storage_config,
    manifest_key=INITIALIZATION_MANIFEST_KEY,
    expected_manifest_size=INITIALIZATION_MANIFEST_SIZE,
    expected_manifest_sha256=INITIALIZATION_MANIFEST_SHA256,
    variant_id=INITIALIZATION_VARIANT_ID,
    expected_config=model_config,
    model=model,
)
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)

The loader verifies the manifest, frozen recipe and source pins, variant ledger, model object, and destination model. It accepts only a bare BF16 state dict, loads it strictly, and returns an immutable receipt for the Metrics config.

Device smoke test

import json

from syvain_training_utils import run_device_smoke_test

report = run_device_smoke_test(require_cuda=True)
print(json.dumps({"smoke": report}, indent=2, sort_keys=True))

Checkpoint a training run

from syvain_training_utils import (
    StorageConfig,
    TrainingLoopState,
    load_training_checkpoint_if_available,
    save_model_checkpoint,
)

storage_config = StorageConfig(
    bucket="my-training-bucket",
    s3_base_url="https://t3.storage.dev",
    region="auto",
    access_key_id="...",
    secret_access_key="...",
)
checkpoint_base_path = "models/my-model"

loop_state = TrainingLoopState(
    global_step=global_step,
    curriculum_stage=curriculum_stage,
    curriculum_step=curriculum_step,
)

save_model_checkpoint(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    experiment_slug=experiment_slug,
    run_id=run_id,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    loop_state=loop_state,
    checkpoint_label=f"step-{global_step:012d}",
)

resume = load_training_checkpoint_if_available(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    device=device,
)

The library owns the object-store clients. Every transient retry opens a fresh client, and expired Tigris multipart sessions restart the complete upload at the same checkpoint key. Checkpoint bodies are written through a temporary local file, uploaded with an adaptive multipart size, and downloaded with resumable range reads. Ensure the machine has temporary disk capacity for one checkpoint.

The manifest remains a pointer to the current checkpoint key within the configured bucket and also records its byte size, ETag, and SHA-256 digest. The manifest is published only after the checkpoint upload succeeds. Loading verifies the complete digest before deserializing the model, optimizer, optional scheduler, and PyTorch RNG state.

The storage transport uses a 10-second connect timeout, a 60-second read-inactivity timeout, and a 10-minute overall request timeout. A bounded 15-minute outer no-progress retry window owns recovery and client replacement.