syvain-training-utils

Internal Syvain helpers for small, explicit ML training runs. No secret sauce here, just shared runtime, device-diagnostic, and checkpoint patterns.

Install

uv add syvain-training-utils

Runtime setup

from syvain_training_utils import (
    generate_run_id,
    require_torch_compile_toolchain,
    select_device,
)

run_id = generate_run_id()
device = select_device()
require_torch_compile_toolchain()

Device smoke test

import json

from syvain_training_utils import run_device_smoke_test

report = run_device_smoke_test(require_cuda=True)
print(json.dumps({"smoke": report}, indent=2, sort_keys=True))

Checkpoint a training run

from syvain_training_utils import (
    StorageConfig,
    TrainingLoopState,
    load_training_checkpoint_if_available,
    save_model_checkpoint,
)

storage_config = StorageConfig(
    bucket="my-training-bucket",
    s3_base_url="https://t3.storage.dev",
    region="auto",
    access_key_id="...",
    secret_access_key="...",
)
checkpoint_base_path = "models/my-model"

loop_state = TrainingLoopState(
    global_step=global_step,
    curriculum_stage=curriculum_stage,
    curriculum_step=curriculum_step,
)

save_model_checkpoint(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    experiment_slug=experiment_slug,
    run_id=run_id,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    loop_state=loop_state,
    checkpoint_label=f"step-{global_step:012d}",
)

resume = load_training_checkpoint_if_available(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    device=device,
)

The library owns the object-store clients. Every transient retry opens a fresh client, and expired Tigris multipart sessions restart the complete upload at the same checkpoint key. Checkpoint bodies are written through a temporary local file, uploaded with an adaptive multipart size, and downloaded with resumable range reads. Ensure the machine has temporary disk capacity for one checkpoint.

The manifest remains a pointer to the current checkpoint key within the configured bucket and also records its byte size, ETag, and SHA-256 digest. The manifest is published only after the checkpoint upload succeeds. Loading verifies the complete digest before deserializing the model, optimizer, optional scheduler, and PyTorch RNG state.

The storage transport uses a 10-second connect timeout, a 60-second read-inactivity timeout, and a 10-minute overall request timeout. A bounded 15-minute outer no-progress retry window owns recovery and client replacement.