hf-ark

The models your code depends on live on servers Nvidia now owns. Snapshot them before the terms change.

hf-ark finds the Hugging Face models and datasets a codebase (or machine) depends on and snapshots them into a portable, checksummed, offline-usable archive: an "ark". It's deliberately boring under the hood. An ark is a valid huggingface_hub cache directory plus a manifest with checksums and license tags. No custom format. Anything that reads an HF cache can read an ark.

Quickstart

pip install hf-ark
hf-ark snapshot --from . --include-cache
hf-ark verify ./hf-ark

Later, on any machine:

hf-ark restore ./hf-ark
HF_HUB_OFFLINE=1 python your_script.py

What an ark is

An ark is just a huggingface_hub cache directory (models--org--name/, blobs/, snapshots/) plus one ark.json file recording, per repo: the resolved commit sha, license tag, gated status, and a sha256 for every file. hf-ark computes those checksums itself; it doesn't trust the hub's metadata.

Because an ark is a real HF cache, you don't need to restore it anywhere to use it. Point HF_HUB_CACHE straight at it:

HF_HUB_CACHE=/path/to/ark HF_HUB_OFFLINE=1 python your_script.py

hf-ark restore exists for when you'd rather merge an ark into your normal cache location instead.

Commands

• hf-ark scan [PATHS...] finds HF repo references in your source (from_pretrained(...), hf_hub_download(...), huggingface.co URLs, and an optional hf-ark.txt file). Add --include-cache to also list what's already in your local HF cache, --check to confirm each repo exists on the hub, --json for machine-readable output.
• hf-ark snapshot [REPO_IDS...] [-o ARK] downloads repos into an ark and records them in ark.json. Takes explicit repo IDs, --from PATHS (runs the scanner), and/or --include-cache. --dry-run prints the download size without writing anything.
• hf-ark verify ARK recomputes every file's sha256 and checks it against the manifest.
• hf-ark list ARK prints a table of what's in an ark: repo, type, sha, license, size. No network.
• hf-ark restore ARK [--target DIR] merges an ark into a local HF cache (default: wherever huggingface_hub already looks).

Run hf-ark <command> --help for the full flag list.

FAQ

Does this redistribute model weights? No. It downloads what you already have access to and stores it under your control. Nothing gets uploaded anywhere.

What about licenses? The manifest records each model's license tag as reported by the hub. The contents stay under their own license. Snapshotting a repo doesn't change what you're allowed to do with it.

Gated models? You still need a token with access (HF_TOKEN, same as huggingface_hub always used). An ark of a gated repo doesn't grant access to anyone else; it's just your already-authorized download saved locally.

Private hub / mirror? hf-ark respects HF_ENDPOINT, same as huggingface_hub.

Is Hugging Face actually going to break? Nobody knows. That's the point. Snapshots are cheap insurance either way.

Not this

hf-ark isn't a mirror server (see olah for that), a torrent client, or a package manager. It snapshots and restores. That's it.

License

MIT