HF Job Control

HF Job Control is a Python library and CLI for cooperative control of detached Hugging Face Jobs. A running job saves its work at a safe boundary before it responds to run, pause, stop, or abort.

The control history stays in a versioned Hugging Face dataset. Checkpoints use content-addressed keys in a Hugging Face Bucket, while a separate dataset stores observed status and immutable receipts.

Installation

HF Job Control requires Python 3.11 or newer. Install the CLI from PyPI:

uv tool install hf-job-control

A submitted job should pin an exact package version and verify the built wheel's SHA-256. A Git commit remains available before the first release or when testing an unreleased change:

uv tool install "hf-job-control @ git+https://github.com/osolmaz/hf-job-control@<commit>"

Agent skill

The repository includes a comprehensive Agent Skills package under skills/hf-job-control. It covers operator workflows, worker integration, protocol records, incident recovery, and launch or completion audits.

Pi can load only the public HF Job Control skill from a pinned Git revision:

pi install git:github.com/osolmaz/hf-job-control@<tag-or-commit>

The package.json manifest excludes the repository's internal documentation helper. Other Agent Skills-compatible tools can load or copy the skills/hf-job-control directory directly.

Operator workflow

Set the private Hub resources used by the run:

export HF_JOB_CONTROL_REPO=owner/jobs-control
export HF_JOB_STATUS_REPO=owner/job-status
export HF_JOB_ARTIFACT_BUCKET=owner/job-artifacts

Create a logical run. The default ID comes from @osolmaz/petname:

RUN_ID="$(hf-job-control create --reason "Start training" | jq -r '.control.run_id')"

Launch one physical attempt from a checked-in launch specification:

hf-job-control launch "$RUN_ID" launch.json

The first launch stores this specification under the logical run. Later attempts must match it byte for byte after canonical JSON serialization.

A pause takes effect after the next safe checkpoint:

hf-job-control pause "$RUN_ID" --reason "Release the worker"
hf-job-control watch "$RUN_ID"

Resume verifies the paused checkpoint and publishes the next run generation. Exact and boundary adapters carry the checkpoint reference forward. Restart adapters begin again from their immutable inputs. Launching creates a new physical job under the same logical run ID:

hf-job-control resume "$RUN_ID" --reason "Continue from the checkpoint"
hf-job-control launch "$RUN_ID" launch.json

Stop completes the logical run at its next safe boundary:

hf-job-control stop "$RUN_ID" --reason "The registered metric has converged"
hf-job-control watch "$RUN_ID"

Pass --expected-generation to a mutating command when another operator or process could write concurrently.

Remote canary

The built-in canary runs a small counter on cpu-basic. It exercises the same Hub reads, checkpoint uploads, status writes, and receipts as a real worker:

hf-job-control canary "$RUN_ID" \
  --status-repo "$HF_JOB_STATUS_REPO" \
  --artifact-bucket "$HF_JOB_ARTIFACT_BUCKET" \
  --package-ref "git+https://github.com/osolmaz/hf-job-control.git@<tag>"

Use the normal pause, resume, launch, and stop commands against the canary run.

Worker integration

A worker supplies a checkpoint adapter and calls the controller at each safe boundary:

from hf_job_control import (
    Boundary,
    Controller,
    ControllerConfig,
    HubBucketArtifactStore,
    HubControlStore,
    HubStatusStore,
)

controller = Controller(
    ControllerConfig.from_environment(),
    control_store=HubControlStore("owner/jobs-control"),
    status_store=HubStatusStore("owner/job-status"),
    artifact_store=HubBucketArtifactStore("owner/job-artifacts"),
)
controller.start(checkpoint_adapter)

for step in training_steps:
    train(step)
    if is_safe_boundary(step):
        decision = controller.boundary(
            boundary=Boundary(name="half-epoch", sequence=step),
            adapter=checkpoint_adapter,
            metrics={"exact": evaluate()},
        )
        if decision.should_exit:
            finalize_outputs()
            controller.finish(decision)
            raise SystemExit(decision.exit_code)

The adapter writes and restores the application's checkpoint payload. The controller verifies the bundle and Hub revisions, tracks command generations, and writes both receipts and observed state. On Hugging Face Jobs, it also copies the built-in JOB_ID into those records.

Resume guarantees

Adapters declare one resume mode. exact restores every state item needed to match uninterrupted execution. boundary restarts from the last committed unit. restart repeats the job from immutable inputs. unsupported rejects a pause at the safe boundary and exits failed because it cannot promise a usable checkpoint.

For PyTorch training, an exact adapter normally includes model parameters, optimizer and scheduler state, mixed-precision state, random-number generator state, data order, global step, and model-selection counters.

Monitoring

hf-job-control watch reads durable project status. W&B can receive the same metrics through WandbMetricSink, but W&B is optional and never controls checkpoint or resume state.

Safety

Hugging Face Jobs defaults to a 30-minute timeout. Every launch specification must set an explicit timeout long enough for the workload.

The controller writes a checkpoint before reading control. It writes an applied-control receipt before changing lifecycle state. If control remains unavailable after retries, the worker pauses instead of continuing forever.

See the protocol for file layouts and action semantics. The implementation plan records the design and test requirements.

License

MIT