HF Job Control

HF Job Control provides Python and TypeScript libraries plus an operator CLI for detached Hugging Face Jobs. A running Job can publish durable progress and save its work at a safe boundary before it responds to run, pause, stop, or abort.

The control history stays in a versioned Hugging Face dataset. Checkpoints use content-addressed keys in a Hugging Face Bucket, while a separate dataset stores observed status and immutable receipts.

Installation

HF Job Control requires Python 3.11 or newer. Install the CLI from PyPI:

uv tool install hf-job-control

A submitted job should pin an exact package version and verify the built wheel's SHA-256. A Git commit remains available before the first release or when testing an unreleased change:

uv tool install "hf-job-control @ git+https://github.com/osolmaz/hf-job-control@<commit>"

Agent skill

The Python package includes the comprehensive Agent Skills package under skills/hf-job-control. The CLI exposes it through the Skillflag interface:

hf-job-control --skill list
hf-job-control --skill show hf-job-control
hf-job-control --skill export hf-job-control | skill-install --agent codex --scope user

The skill covers operator workflows, worker integration, protocol records, incident recovery, and launch or completion audits. The export contains its references, schemas, agent metadata, and launch-specification example.

Pi can also load the public HF Job Control skill from a pinned Git revision:

pi install git:github.com/osolmaz/hf-job-control@<tag-or-commit>

The package.json manifest and the Python CLI expose only the public skill. The repository's internal documentation helper is excluded.

Operator workflow

Set the private Hub resources used by the run:

export HF_JOB_CONTROL_REPO=owner/jobs-control
export HF_JOB_STATUS_REPO=owner/job-status
export HF_JOB_ARTIFACT_BUCKET=owner/job-artifacts

Create a logical run. The default ID comes from @osolmaz/petname:

RUN_ID="$(hf-job-control create --reason "Start training" | jq -r '.control.run_id')"

Launch one physical attempt from a checked-in launch specification:

hf-job-control launch "$RUN_ID" launch.json

The first launch stores this specification under the logical run. Later attempts must match it byte for byte after canonical JSON serialization.

A pause takes effect after the next safe checkpoint:

hf-job-control pause "$RUN_ID" --reason "Release the worker"
hf-job-control watch "$RUN_ID"

Resume verifies the paused checkpoint and publishes the next run generation. Exact and boundary adapters carry the checkpoint reference forward. Restart adapters begin again from their immutable inputs. Launching creates a new physical job under the same logical run ID:

hf-job-control resume "$RUN_ID" --reason "Continue from the checkpoint"
hf-job-control launch "$RUN_ID" launch.json

Stop completes the logical run at its next safe boundary:

hf-job-control stop "$RUN_ID" --reason "The registered metric has converged"
hf-job-control watch "$RUN_ID"

Pass --expected-generation to a mutating command when another operator or process could write concurrently.

Remote canary

The built-in canary runs a small counter on cpu-basic. It exercises the same Hub reads, checkpoint uploads, status writes, and receipts as a real worker:

hf-job-control canary "$RUN_ID" \
  --status-repo "$HF_JOB_STATUS_REPO" \
  --artifact-bucket "$HF_JOB_ARTIFACT_BUCKET" \
  --package-ref "git+https://github.com/osolmaz/hf-job-control.git@<tag>"

Use the normal pause, resume, launch, and stop commands against the canary run.

Worker integration

A worker supplies a checkpoint adapter and calls the controller at each safe boundary:

from hf_job_control import (
    Boundary,
    Controller,
    ControllerConfig,
    HubBucketArtifactStore,
    HubControlStore,
    HubStatusStore,
)

controller = Controller(
    ControllerConfig.from_environment(),
    control_store=HubControlStore("owner/jobs-control"),
    status_store=HubStatusStore("owner/job-status"),
    artifact_store=HubBucketArtifactStore("owner/job-artifacts"),
)
controller.start(checkpoint_adapter)

for step in training_steps:
    train(step)
    if is_safe_boundary(step):
        decision = controller.boundary(
            boundary=Boundary(name="half-epoch", sequence=step),
            adapter=checkpoint_adapter,
            metrics={"exact": evaluate()},
        )
        if decision.should_exit:
            finalize_outputs()
            controller.finish(decision)
            raise SystemExit(decision.exit_code)

The adapter writes and restores the application's checkpoint payload. The controller verifies the bundle and Hub revisions, tracks command generations, and writes both receipts and observed state. On Hugging Face Jobs, it also copies the built-in JOB_ID into those records.

Resume guarantees

Adapters declare one resume mode. exact restores every state item needed to match uninterrupted execution. boundary restarts from the last committed unit. restart repeats the job from immutable inputs. unsupported rejects a pause at the safe boundary and exits failed because it cannot promise a usable checkpoint.

For PyTorch training, an exact adapter normally includes model parameters, optimizer and scheduler state, mixed-precision state, random-number generator state, data order, global step, and model-selection counters.

Progress reporting

Python and TypeScript workers can publish exact committed progress without using the full lifecycle controller. Each application defines its phases and counts. HF Job Control validates the tracks, prevents regression, and writes content-addressed snapshots to the application's existing Bucket.

Python workers use ProgressReporter with HubBucketProgressStore. TypeScript workers install this repository from a pinned tag or commit and use ProgressReporter with ObjectProgressStore plus their existing Bucket IO. Both implementations use the same progress-v1 format.

A worker fixes a plan_id, total, and unit before reporting a measurable phase. Changing the input or denominator starts a new plan instead of moving a counter backward. Workers report committed output only and flush at safe boundaries. Monitoring systems calculate rates and finish-time ranges from the immutable snapshot chain.

See the progress reporting plan for storage, recovery, and application integration rules.

Monitoring

hf-job-control watch reads durable project status. W&B can receive the same metrics through WandbMetricSink, but W&B is optional and never controls checkpoint or resume state. The optional typed progress field carries the latest validated progress snapshot separately from scientific metrics.

Safety

Hugging Face Jobs defaults to a 30-minute timeout. Every launch specification must set an explicit timeout long enough for the workload.

The controller writes a checkpoint before reading control. It writes an applied-control receipt before changing lifecycle state. If control remains unavailable after retries, the worker pauses instead of continuing forever.

See the protocol for file layouts and action semantics. The implementation plan records the design and test requirements.

License

MIT