Useful utilities for huggingface
Project description
hfutils
Useful utilities for huggingface
Quick Start
To get started with hfutils
, you can install it using PyPI:
pip install hfutils
Alternatively, you can install it from the source code:
git clone https://github.com/deepghs/hfutils.git
cd hfutils
pip install .
Verify the installation by checking the version:
hfutils -v
If Python is not available in your local environment, we recommend downloading the pre-compiled executable version from the releases.
Upload Data
Upload data to repositories using the following commands:
# Upload a single file to the repository
hfutils upload -r your/repository -i /your/local/file -f file/in/your/repo
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip
# Upload files in a directory as a directory tree to the repository
hfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo
You can achieve the same using the Python API:
from hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory
# Upload a single file to the repository
upload_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
upload_directory_as_archive(
local_directory='/your/local/directory',
repo_id='your/repository',
archive_in_repo='archive/file/in/your/repo.zip'
)
# Upload files in a directory as a directory tree to the repository
upload_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
path_in_repo='dir/in/your/repo'
)
Explore additional options for uploading:
hfutils upload -h
Download Data
Download data from repositories using the following commands:
# Download a single file from the repository
hfutils download -r your/repository -o /your/local/file -f file/in/your/repo
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip
# Download files from the repository as a directory tree
hfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo
Use the Python API for the same functionality:
from hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory
# Download a single file from the repository
download_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
download_archive_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
file_in_repo='archive/file/in/your/repo.zip',
)
# Download files from the repository as a directory tree
download_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
dir_in_repo='dir/in/your/repo'
)
Explore additional options for downloading:
hfutils download -h
Supported Formats
By default, we support the zip
and tar
formats, including .zip
, .tar
, .tar.gz
, .tar.bz2
, and .tar.xz
.
If you require support for rar
and 7z
files, install the extra dependencies using the following command:
pip install hfutils[rar,7z]
NOTE: Creating RAR archive files is not supported. We use the rarfile library, which lacks the functionality for creating RAR files.
How to Access Private Repositories
Simply configure the HF_TOKEN
environment variable by using your HuggingFace access token.
Note that write permissions are required if you plan to upload any content.
How to Use Hf-Transfer for Acceleration
If you are using the PyPI CLI, you need to install hfutils
with the following command:
pip install hfutils[transfer]
If you are using a precompiled executable file, the transfer module is integrated inside; simply use it.
Enable Hf-Transfer acceleration by setting the environment variable HF_HUB_ENABLE_HF_TRANSFER
to 1
.
How to Change the Temporary Directory
The temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk space.
If your disk, especially the C drive on Windows, does not have sufficient space, simply set the TMPDIR
to designate
another directory as the temporary directory.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.