Useful utilities for huggingface
Project description
hfutils
Useful utilities for huggingface
Quick Start
To get started with hfutils
, you can install it using PyPI:
pip install hfutils
Alternatively, you can install it from the source code:
git clone https://github.com/deepghs/hfutils.git
cd hfutils
pip install .
Verify the installation by checking the version:
hfutils -v
If Python is not available in your local environment, we recommend downloading the pre-compiled executable version from the releases.
Upload Data
Upload data to repositories using the following commands:
# Upload a single file to the repository
hfutils upload -r your/repository -i /your/local/file -f file/in/your/repo
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip
# Upload files in a directory as a directory tree to the repository
hfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo
You can achieve the same using the Python API:
from hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory
# Upload a single file to the repository
upload_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
upload_directory_as_archive(
local_directory='/your/local/directory',
repo_id='your/repository',
archive_in_repo='archive/file/in/your/repo.zip'
)
# Upload files in a directory as a directory tree to the repository
upload_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
path_in_repo='dir/in/your/repo'
)
Explore additional options for uploading:
hfutils upload -h
Download Data
Download data from repositories using the following commands:
# Download a single file from the repository
hfutils download -r your/repository -o /your/local/file -f file/in/your/repo
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip
# Download files from the repository as a directory tree
hfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo
Use the Python API for the same functionality:
from hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory
# Download a single file from the repository
download_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
download_archive_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
file_in_repo='archive/file/in/your/repo.zip',
)
# Download files from the repository as a directory tree
download_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
dir_in_repo='dir/in/your/repo'
)
Explore additional options for downloading:
hfutils download -h
List Files in Repository
List files in repositories
hfutils ls -r your/repository -o /your/local/file -d subdir/in/repo
Supported Formats
By default, we support the zip
and tar
formats, including .zip
, .tar
, .tar.gz
, .tar.bz2
, and .tar.xz
.
If you require support for rar
and 7z
files, install the extra dependencies using the following command:
pip install hfutils[rar,7z]
NOTE: Creating RAR archive files is not supported. We use the rarfile library, which lacks the functionality for creating RAR files.
How to Access Private Repositories
Simply configure the HF_TOKEN
environment variable by using your HuggingFace access token.
Note that write permissions are required if you plan to upload any content.
How to Use Hf-Transfer for Acceleration
If you are using the PyPI CLI, you need to install hfutils
with the following command:
pip install hfutils[transfer]
If you are using a precompiled executable file, the transfer module is integrated inside; simply use it.
Enable Hf-Transfer acceleration by setting the environment variable HF_HUB_ENABLE_HF_TRANSFER
to 1
.
How to Change the Temporary Directory
The temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk space.
If your disk, especially the C drive on Windows, does not have sufficient space, simply set the TMPDIR
to designate
another directory as the temporary directory.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file hfutils-0.5.1.tar.gz
.
File metadata
- Download URL: hfutils-0.5.1.tar.gz
- Upload date:
- Size: 61.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1d5440a3d346e4148ace5d812c6af26ccb905d82924358138512efa5a30c1d1 |
|
MD5 | cfbd584b07fa161d659aa836a35fd2d9 |
|
BLAKE2b-256 | ce2d2e508a77bc39def76208c03d92d1367649e56797f0511eb5c44aa15670c1 |
File details
Details for the file hfutils-0.5.1-py3-none-any.whl
.
File metadata
- Download URL: hfutils-0.5.1-py3-none-any.whl
- Upload date:
- Size: 84.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a35fd0b9487ac44e8ad7035887b23a01b61a7d005badd964947ca8eceaa3720 |
|
MD5 | e8c2f42ef3e4410209d5211d4c03ac9c |
|
BLAKE2b-256 | 64d136def4fd469facea4c76949ae0ea9cec420c39a74d313713dcd2eb95b132 |