syvain-training-utils

Internal Syvain helpers for small, explicit ML training runs. No secret sauce here, just shared runtime, device-diagnostic, and checkpoint patterns.

Install

uv add syvain-training-utils

Runtime setup

from syvain_training_utils import (
    generate_run_id,
    require_torch_compile_toolchain,
    select_device,
)

run_id = generate_run_id()
device = select_device()
require_torch_compile_toolchain()

Device smoke test

import json

from syvain_training_utils import run_device_smoke_test

report = run_device_smoke_test(require_cuda=True)
print(json.dumps({"smoke": report}, indent=2, sort_keys=True))

Checkpoint a training run

from syvain_training_utils import (
    StorageConfig,
    TrainingLoopState,
    load_training_checkpoint_if_available,
    save_model_checkpoint,
)

storage_config = StorageConfig(
    bucket="my-training-bucket",
    s3_base_url="https://t3.storage.dev",
    region="auto",
    access_key_id="...",
    secret_access_key="...",
)
checkpoint_base_path = "models/my-model"

loop_state = TrainingLoopState(
    global_step=global_step,
    curriculum_stage=curriculum_stage,
    curriculum_step=curriculum_step,
)

save_model_checkpoint(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    experiment_slug=experiment_slug,
    run_id=run_id,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    loop_state=loop_state,
    checkpoint_label=f"step-{global_step:012d}",
)

resume = load_training_checkpoint_if_available(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    device=device,
)

The library owns the object-store clients. Every transient retry opens a fresh client, and expired Tigris multipart sessions restart the complete upload at the same checkpoint key. Checkpoint bodies are written through a temporary local file, uploaded with an adaptive multipart size, and downloaded with resumable range reads. Ensure the machine has temporary disk capacity for one checkpoint.

The manifest remains a pointer to the current checkpoint key within the configured bucket and also records its byte size, ETag, and SHA-256 digest. The manifest is published only after the checkpoint upload succeeds. Loading verifies the complete digest before deserializing the model, optional optimizer, scheduler, and PyTorch RNG state.

Observation-only runs can pass optimizer=None and scheduler=None to both checkpoint functions. Model parameters and buffers, PyTorch RNG state, run identity, and loop progress are still saved and restored. A buffer-only module needs no dummy parameter or optimizer. Save and resume must agree on whether the optimizer and scheduler exist; incompatible configurations fail before restoring model state. A scheduler always requires an optimizer. To load only model state across configurations, use load_model_checkpoint_from_key.

The storage transport uses a 10-second connect timeout, a 60-second read-inactivity timeout, and a 10-minute overall request timeout. A bounded 15-minute outer no-progress retry window owns recovery and client replacement.