hfutils
Useful utilities for huggingface
Quick Start
To get started with hfutils, you can install it using PyPI:
pip install hfutils
Alternatively, you can install it from the source code:
git clone https://github.com/deepghs/hfutils.git
cd hfutils
pip install .
Verify the installation by checking the version:
hfutils -v
If Python is not available in your local environment, we recommend downloading the pre-compiled executable version from the releases.
Upload Data
Upload data to repositories using the following commands:
# Upload a single file to the repository
hfutils upload -r your/repository -i /your/local/file -f file/in/your/repo
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip
# Upload files in a directory as a directory tree to the repository
hfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo
You can achieve the same using the Python API:
from hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory
# Upload a single file to the repository
upload_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
upload_directory_as_archive(
local_directory='/your/local/directory',
repo_id='your/repository',
archive_in_repo='archive/file/in/your/repo.zip'
)
# Upload files in a directory as a directory tree to the repository
upload_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
path_in_repo='dir/in/your/repo'
)
Explore additional options for uploading:
hfutils upload -h
Download Data
Download data from repositories using the following commands:
# Download a single file from the repository
hfutils download -r your/repository -o /your/local/file -f file/in/your/repo
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip
# Download files from the repository as a directory tree
hfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo
Use the Python API for the same functionality:
from hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory
# Download a single file from the repository
download_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
download_archive_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
file_in_repo='archive/file/in/your/repo.zip',
)
# Download files from the repository as a directory tree
download_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
dir_in_repo='dir/in/your/repo'
)
Explore additional options for downloading:
hfutils download -h
List Files in Repository
List files in repositories
hfutils ls -r your/repository -o /your/local/file -d subdir/in/repo
Supported Formats
By default, we support the zip and tar formats, including .zip, .tar, .tar.gz, .tar.bz2, and .tar.xz.
If you require support for rar and 7z files, install the extra dependencies using the following command:
pip install hfutils[rar,7z]
NOTE: Creating RAR archive files is not supported. We use the rarfile library, which lacks the functionality for creating RAR files.
How to Access Private Repositories
Simply configure the HF_TOKEN environment variable by using your HuggingFace access token.
Note that write permissions are required if you plan to upload any content.
How to Use Hf-Transfer for Acceleration
If you are using the PyPI CLI, you need to install hfutils with the following command:
pip install hfutils[transfer]
If you are using a precompiled executable file, the transfer module is integrated inside; simply use it.
Enable Hf-Transfer acceleration by setting the environment variable HF_HUB_ENABLE_HF_TRANSFER to 1.
How to Change the Temporary Directory
The temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk space.
If your disk, especially the C drive on Windows, does not have sufficient space, simply set the TMPDIR to designate
another directory as the temporary directory.
1 maintainer
Metadata
1 maintainer
Release files for hfutils 0.13.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| hfutils-0.13.0.tar.gz | 86.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| hfutils-0.13.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 201.6 kB
Release files / hfutils-0.13.0.tar.gz
| Download URL | hfutils-0.13.0.tar.gz |
|---|---|
| Size | 86.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d771b95236c84be4b79332835c6ef8b7d16e6ea5a0148c403a5e53be6fdc925f
|
|
BLAKE2b-256 checksum How to use checksums |
8d009c6fc84f4db593ad6275386116fe652e88ffeb6823aba7a5b4d18fe89f53
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.14
|
Release files / hfutils-0.13.0-py3-none-any.whl
| Download URL | hfutils-0.13.0-py3-none-any.whl |
|---|---|
| Size | 115.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
db852e4b266018ee5218749b8c46980de17e3f9675dd6e73cd2e884aa767393f
|
|
BLAKE2b-256 checksum How to use checksums |
3a964586598b438de26bffd6c327eef7338de922d913286542ba682edb14e7e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.14
|
