cog-hf-template 0.0.12

Cog template for Hugging Face.

cog-huggingface-template

Tools to help deploy Hugging Face models to replicate.

Requirements

• GPU environment
• Cog

Installation

PyPI Install

pip install cog_hf_template

Dev Install

Clone the repo:

git clone https://github.com/replicate/cog-huggingface-template
cd cog-huggingface-template
pip install -e .

Run tests

pip install pytest

pytest -vv tests

Usage

NOTE (Oct 19) - examples are out of date. Will update soon.

For now, see the examples directory for a working example.

Mirroring Weights from HF to Replicate Storage

Here's a quick tip on how to mirror weights from Hugging Face to Replicate Storage on GCP.

You can use a minimal install of this package to avoid the extra dependencies...

# required deps for mirroring weights/CLI
pip install --no-cache huggingface-hub==0.18.0 google-cloud-storage==2.12.0 hf-transfer==0.1.3 fire==0.5.0

# Minimal version of the package
pip install --no-deps --no-cache --upgrade cog-hf-template

You will also need pget installed

curl -o /usr/local/bin/pget -L "https://github.com/replicate/pget/releases/download/v0.0.2/pget" && chmod +x /usr/local/bin/pget

You'll need to make sure you have access to the bucket. You can do this by running gcloud auth login and following the instructions, assuming you've been granted access to the [e-mail] group on GCP.

Then you can run the following command to mirror the weights:

cog-hf-template mirror-weights \
    openai/whisper-large-v3 \
    replicate-hf-weights \
    --gcp-project-name replicate-production \
    --config_save_path whisper_large_v3_config.json

In the example above, we mirror openai/whisper-large-v3 to the bucket replicate-hf-weights in the project replicate-production. We also save the model config to whisper_large_v3_config.json so that we can use it later to load the model (as you'll need to know the files to download).

The config looks like this:

{
  "hf_model_id": "openai/whisper-large-v3",
  "task": "automatic-speech-recognition",
  "gcp_bucket_weights": "gs://replicate-hf-weights/openai/whisper-large-v3/1940b900b8d4d8fdf72a31be85a0d223b9488e00",
  "trust_remote_code": true,
  "remote_filenames": [
    ".gitattributes",
    "README.md",
    "added_tokens.json",
    "config.json",
    "flax_model.msgpack",
    "generation_config.json",
    "merges.txt",
    "model.fp32-00001-of-00002.safetensors",
    "model.fp32-00002-of-00002.safetensors",
    "model.safetensors",
    "model.safetensors.index.fp32.json",
    "preprocessor_config.json",
    "pytorch_model.bin",
    "pytorch_model.bin.index.fp32.json",
    "pytorch_model.fp32-00001-of-00002.bin",
    "pytorch_model.fp32-00002-of-00002.bin",
    "special_tokens_map.json",
    "tokenizer_config.json",
    "vocab.json"
  ]
}

Note the files are accessible through the delivery URLs, and will the gs://replicate-hf-weights prefix will be replaced by the appropriate prefix before being downloaded.

import json
from pathlib import Path

from cog_hf_template.download_utils import maybe_pget_weights

config = json.loads(Path("whisper_large_v3_config.json").read_text())
maybe_pget_weights(
    path="whisper_large_v3",
    remote_path=config["gcp_bucket_weights"],
    remote_filenames=config["remote_filenames"],
)