Client library to download and publish models on the huggingface.co hub
Client library to download and publish models and other files on the huggingface.co hub
Do you have an open source ML library? We're looking to partner with a small number of other cool open source ML libraries to provide model hosting + versioning. https://twitter.com/julien_c/status/1336374565157679104 https://twitter.com/mnlpariente/status/1336277058062852096
- versioning is built-in (as hosting is built around git and git-lfs), no lock-in, you can just
- anyone can upload a new model for your library, just need to add the corresponding tag for the model to be discoverable – no more need for a hardcoded list in your code
- Fast downloads! We use Cloudfront (a CDN) to geo-replicate downloads so they're blazing fast from anywhere on the globe
- Usage stats and more features to come
Ping us if interested 😎
♻️ Partial list of implementations in third party libraries:
- http://github.com/asteroid-team/asteroid [initial PR 👀]
- https://github.com/pyannote/pyannote-audio [initial PR 👀]
- https://github.com/flairNLP/flair [work-in-progress, initial PR 👀]
- https://github.com/espnet/espnet [initial PR 👀]
Download files from the huggingface.co hub
Integration inside a library is super simple. We expose two functions,
- a repo id (e.g. a model id like
julien-c/EsperBERTo-smalli.e. a user or organization name and a repo name, separated by
- a filename (like
- and an optional git revision id (can be a branch name, a tag, or a commit hash)
and returns the url we'll use to download the actual files:
If you check out this URL's headers with a
HEAD http request (which you can do from the command line with
curl -I) for a few different files, you'll see that:
- small files are returned directly
- large files (i.e. the ones stored through git-lfs) are returned via a redirect to a Cloudfront URL. Cloudfront is a Content Delivery Network, or CDN, that ensures that downloads are as fast as possible from anywhere on the globe.
cached_download() takes the following parameters, downloads the remote file, stores it to disk (in a versioning-aware way) and returns its local file path.
- a remote
- your library's name and version (
library_version), which will be added to the HTTP requests' user-agent so that we can provide some usage stats.
cache_dirwhich you can specify if you want to control where on disk the files are cached.
Check out the source code for all possible params (we'll create a real doc page in the future).
Publish models to the huggingface.co hub
Uploading a model to the hub is super simple too:
- create a model repo directly from the website, at huggingface.co/new (models can be public or private, and are namespaced under either a user or an organization)
- clone it with git
- download and install git lfs if you don't already have it on your machine (you can check by running a simple
- add, commit and push your files, from git, as you usually do.
We are intentionally not wrapping git too much, so that you can go on with the workflow you’re used to and the tools you already know.
👀 To see an example of how we document the model sharing process in
transformers, check out https://huggingface.co/transformers/model_sharing.html
Users add tags into their README.md model cards (e.g. your
library_name, a domain tag like
audio, etc.) to make sure their models are discoverable.
Documentation about the model hub itself is at https://huggingface.co/docs
API utilities in
You don't need them for the standard publishing workflow, however, if you need a programmatic way of creating a repo, deleting it (
⚠️ caution), or listing models from the hub, you'll find helpers in
We also have an API to query models by specific tags (e.g. if you want to list models compatible to your library)
Those API utilities are also exposed through a CLI:
huggingface-cli login huggingface-cli logout huggingface-cli whoami huggingface-cli repo create
Need to upload large (>5GB) files?
To upload large files (>5GB 🔥), you need to install the custom transfer agent for git-lfs, bundled in this package.
To install, just run:
$ huggingface-cli lfs-enable-largefiles
This should be executed once for each model repo that contains a model file >5GB. If you just try to push a file bigger than 5GB without running that command, you will get an error with a message reminding you to run it.
Finally, there's a
huggingface-cli lfs-multipart-upload command but that one is internal (called by lfs directly) and is not meant to be called by the user.
Visual integration into the huggingface.co hub
Finally, we'll implement a few tweaks to improve the UX for your models on the website – let's use Asteroid as an example:
Model authors add an
asteroid tag to their model card and they get the advantages of model versioning built-in
We add a custom "Use in Asteroid" button.
When clicked you get a library-specific code sample that you'll be able to specify. 🔥
Feedback (feature requests, bugs, etc.) is super welcome 💙💚💛💜♥️🧡
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size huggingface_hub-0.0.6-py3-none-any.whl (28.5 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size huggingface_hub-0.0.6.tar.gz (22.8 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for huggingface_hub-0.0.6-py3-none-any.whl