denpex-sdk

Zero-third-party Python helpers for SDC detection (per-layer weight/gradient norms, NaN/Inf checks, optimizer state validation) and asynchronous checkpoint validation (manifest integrity + content checksums) for S3/Lustre.

Cloud diagnosis client

from denpex_sdk import DenpexClient

client = DenpexClient(api_key="dpx_...")
result = client.diagnose(
    open("training.log").read(),
    jobName="llama3-405b-pretrain",
    telemetry={"jobId": "slurm-1842"},
)
print(result["meta"]["failureType"], result["meta"].get("confidence"))
print(result["fix"]["summary"])
print(result["fix"]["action"])
client.feedback(result["diagnosisId"], "worked")

The client reads DENPEX_API_KEY when api_key is omitted and calls the live /api/diagnose contract. Its default 320-second timeout covers the API's 290-second deep-diagnosis window. Optional arguments such as telemetry, codeContext, topologyMap, runtimeHours, and gpuRate pass through.

SDC detection

from denpex_sdk import SdcHook, install_sdc_hooks

hook = install_sdc_hooks(model, SdcHook(max_weight_norm=1e4, max_grad_norm=1e4))
weight_report = hook.inspect()
grad_report = hook.inspect_gradients()
opt_report = hook.inspect_optimizer(optimizer)

Checkpoint validation

from denpex_sdk import build_manifest, manifest_to_json, manifest_from_json, validate_checkpoint

manifest = build_manifest(["/checkpoints/step-1000/"], metadata={"job_id": "..."})
with open("/checkpoints/step-1000/manifest.json", "w") as handle:
    handle.write(manifest_to_json(manifest))

with open("/checkpoints/step-1000/manifest.json") as handle:
    expected = manifest_from_json(handle.read())

result = validate_checkpoint(["/checkpoints/step-1000/"], expected_manifest=expected)
assert result.valid