Skip to main content

Useful utilities for huggingface

Project description

hfutils

PyPI PyPI - Python Version Loc Comments

Code Test Package Release codecov

Discord GitHub Org's stars GitHub stars GitHub forks GitHub commit activity GitHub issues GitHub pulls Contributors GitHub license

Useful utilities for huggingface

Quick Start

To get started with hfutils, you can install it using PyPI:

pip install hfutils

Alternatively, you can install it from the source code:

git clone https://github.com/deepghs/hfutils.git
cd hfutils
pip install .

Verify the installation by checking the version:

hfutils -v

If Python is not available in your local environment, we recommend downloading the pre-compiled executable version from the releases.

Upload Data

Upload data to repositories using the following commands:

# Upload a single file to the repository
hfutils upload -r your/repository -i /your/local/file -f file/in/your/repo

# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip

# Upload files in a directory as a directory tree to the repository
hfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo

You can achieve the same using the Python API:

from hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory

# Upload a single file to the repository
upload_file_to_file(
    local_file='/your/local/file',
    repo_id='your/repository',
    file_in_repo='file/in/your/repo'
)

# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
upload_directory_as_archive(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    archive_in_repo='archive/file/in/your/repo.zip'
)

# Upload files in a directory as a directory tree to the repository
upload_directory_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    path_in_repo='dir/in/your/repo'
)

Explore additional options for uploading:

hfutils upload -h

Download Data

Download data from repositories using the following commands:

# Download a single file from the repository
hfutils download -r your/repository -o /your/local/file -f file/in/your/repo

# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip

# Download files from the repository as a directory tree
hfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo

Use the Python API for the same functionality:

from hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory

# Download a single file from the repository
download_file_to_file(
    local_file='/your/local/file',
    repo_id='your/repository',
    file_in_repo='file/in/your/repo'
)

# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
download_archive_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    file_in_repo='archive/file/in/your/repo.zip',
)

# Download files from the repository as a directory tree
download_directory_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    dir_in_repo='dir/in/your/repo'
)

Explore additional options for downloading:

hfutils download -h

List Files in Repository

List files in repositories

hfutils ls -r your/repository -o /your/local/file -d subdir/in/repo

Supported Formats

By default, we support the zip and tar formats, including .zip, .tar, .tar.gz, .tar.bz2, and .tar.xz.

If you require support for rar and 7z files, install the extra dependencies using the following command:

pip install hfutils[rar,7z]

NOTE: Creating RAR archive files is not supported. We use the rarfile library, which lacks the functionality for creating RAR files.

How to Access Private Repositories

Simply configure the HF_TOKEN environment variable by using your HuggingFace access token. Note that write permissions are required if you plan to upload any content.

How to Use Hf-Transfer for Acceleration

If you are using the PyPI CLI, you need to install hfutils with the following command:

pip install hfutils[transfer]

If you are using a precompiled executable file, the transfer module is integrated inside; simply use it.

Enable Hf-Transfer acceleration by setting the environment variable HF_HUB_ENABLE_HF_TRANSFER to 1.

How to Change the Temporary Directory

The temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk space.

If your disk, especially the C drive on Windows, does not have sufficient space, simply set the TMPDIR to designate another directory as the temporary directory.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

hfutils-0.5.1.tar.gz (61.8 kB view details)

Uploaded Source

Built Distribution

hfutils-0.5.1-py3-none-any.whl (84.9 kB view details)

Uploaded Python 3

File details

Details for the file hfutils-0.5.1.tar.gz.

File metadata

  • Download URL: hfutils-0.5.1.tar.gz
  • Upload date:
  • Size: 61.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.10

File hashes

Hashes for hfutils-0.5.1.tar.gz
Algorithm Hash digest
SHA256 e1d5440a3d346e4148ace5d812c6af26ccb905d82924358138512efa5a30c1d1
MD5 cfbd584b07fa161d659aa836a35fd2d9
BLAKE2b-256 ce2d2e508a77bc39def76208c03d92d1367649e56797f0511eb5c44aa15670c1

See more details on using hashes here.

File details

Details for the file hfutils-0.5.1-py3-none-any.whl.

File metadata

  • Download URL: hfutils-0.5.1-py3-none-any.whl
  • Upload date:
  • Size: 84.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.10

File hashes

Hashes for hfutils-0.5.1-py3-none-any.whl
Algorithm Hash digest
SHA256 8a35fd0b9487ac44e8ad7035887b23a01b61a7d005badd964947ca8eceaa3720
MD5 e8c2f42ef3e4410209d5211d4c03ac9c
BLAKE2b-256 64d136def4fd469facea4c76949ae0ea9cec420c39a74d313713dcd2eb95b132

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page