Skip to main content

hfutils

PyPI PyPI - Python Version Loc Comments

Code Test Package Release codecov

Discord GitHub Org's stars GitHub stars GitHub forks GitHub commit activity GitHub issues GitHub pulls Contributors GitHub license

Useful utilities for huggingface

Quick Start

To get started with hfutils, you can install it using PyPI:

pip install hfutils

Alternatively, you can install it from the source code:

git clone https://github.com/deepghs/hfutils.git
cd hfutils
pip install .

Verify the installation by checking the version:

hfutils -v

If Python is not available in your local environment, we recommend downloading the pre-compiled executable version from the releases.

Upload Data

Upload data to repositories using the following commands:

# Upload a single file to the repository
hfutils upload -r your/repository -i /your/local/file -f file/in/your/repo

# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip

# Upload files in a directory as a directory tree to the repository
hfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo

You can achieve the same using the Python API:

from hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory

# Upload a single file to the repository
upload_file_to_file(
    local_file='/your/local/file',
    repo_id='your/repository',
    file_in_repo='file/in/your/repo'
)

# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
upload_directory_as_archive(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    archive_in_repo='archive/file/in/your/repo.zip'
)

# Upload files in a directory as a directory tree to the repository
upload_directory_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    path_in_repo='dir/in/your/repo'
)

Explore additional options for uploading:

hfutils upload -h

Download Data

Download data from repositories using the following commands:

# Download a single file from the repository
hfutils download -r your/repository -o /your/local/file -f file/in/your/repo

# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip

# Download files from the repository as a directory tree
hfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo

Use the Python API for the same functionality:

from hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory

# Download a single file from the repository
download_file_to_file(
    local_file='/your/local/file',
    repo_id='your/repository',
    file_in_repo='file/in/your/repo'
)

# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
download_archive_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    file_in_repo='archive/file/in/your/repo.zip',
)

# Download files from the repository as a directory tree
download_directory_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    dir_in_repo='dir/in/your/repo'
)

Explore additional options for downloading:

hfutils download -h

List Files in Repository

List files in repositories

hfutils ls -r your/repository -o /your/local/file -d subdir/in/repo

Read a Single File Out of a Zip Archive

A zip archive carries its own index (the central directory at its end), so no index file is needed: only the bytes of the directory and of the member you ask for are fetched, with Range requests, from a repository or from the disk.

from hfutils.index import hf_zip_list_files, hf_zip_file_read, hf_zip_file_download, zip_file_read

print(hf_zip_list_files('your/repository', 'path/in/repo/archive.zip'))
data = hf_zip_file_read('your/repository', 'path/in/repo/archive.zip', 'dir/file.txt', password='secret')  # password is optional
hf_zip_file_download('your/repository', 'path/in/repo/archive.zip', 'dir/file.txt', '/your/local/file.txt')
head = zip_file_read('/your/local/archive.zip', 'dir/file.txt', limit=1024)  # the same functions without `hf_` read local files
  • Split archives (X.zip.001, X.zip.002, ... made by 7-Zip, or X.z01, ..., X.zip made by Info-ZIP) open from any of their volumes.
  • The format is recognized by its content, never by the extension (.jar, .docx, a self-extractor or a file with no extension are fine). A file that is not a zip raises NotAZipError that says what it looks like ('tar', 'git-lfs-pointer', 'html', ...).
  • Methods: stored, deflate, deflate64, bzip2, lzma, xz, zstd, and the legacy shrink / reduce / implode. Encryption: ZipCrypto and WinZip AES. PPMd, PKWARE strong encryption and an encrypted central directory are reported as unsupported.
  • pip install hfutils[zip] adds what a few of them need: cryptography (AES), inflate64 (deflate64) and zstandard (zstd before Python 3.14).
  • Names that are not Unicode are decoded with a guessed code page; give encoding= ('shift_jis', 'gbk', ...) or lang= to decide.
  • Every error is a ZipIndexError with a kind: fetch, format, password or unsupported.

hfutils ils -r your/repository -a path/in/repo/archive.zip lists a zip like it lists an indexed tar (-l, -I, sorting).

Supported Formats

By default, we support the zip and tar formats, including .zip, .tar, .tar.gz, .tar.bz2, and .tar.xz.

If you require support for rar and 7z files, install the extra dependencies using the following command:

pip install hfutils[rar,7z]

NOTE: Creating RAR archive files is not supported. We use the rarfile library, which lacks the functionality for creating RAR files.

How to Access Private Repositories

Simply configure the HF_TOKEN environment variable by using your HuggingFace access token. Note that write permissions are required if you plan to upload any content.

How to Use Hf-Transfer for Acceleration

If you are using the PyPI CLI, you need to install hfutils with the following command:

pip install hfutils[transfer]

If you are using a precompiled executable file, the transfer module is integrated inside; simply use it.

Enable Hf-Transfer acceleration by setting the environment variable HF_HUB_ENABLE_HF_TRANSFER to 1.

How to Change the Temporary Directory

The temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk space.

If your disk, especially the C drive on Windows, does not have sufficient space, simply set the TMPDIR to designate another directory as the temporary directory.

Metadata

Release files for hfutils 0.15.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hfutils 0.15.0
File Size Uploaded Provenance Actions
hfutils-0.15.0.tar.gz 129.9 kB Without provenance Details for hfutils-0.15.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hfutils 0.15.0
File Interpreter ABI Platform Actions
hfutils-0.15.0-py3-none-any.whl Python 3 none any Details for hfutils-0.15.0-py3-none-any.whl

Total release size: 295.0 kB

Release files / hfutils-0.15.0.tar.gz

Download URL hfutils-0.15.0.tar.gz
Size 129.9 kB
Tags Source
SHA-256 checksum
How to use checksums
7d4c47c4031e2867279bb618f029dd6a97315a7aaca2af1851b3924aadb4f719
BLAKE2b-256 checksum
How to use checksums
f1f4d9ddaa059b6f01282a9c805e5d455b053b8c8dd32f531b50a4571d8794fa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.18

Release files / hfutils-0.15.0-py3-none-any.whl

Download URL hfutils-0.15.0-py3-none-any.whl
Size 165.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8e51e9cb4752fed1f460a3f42bc37c412c1b195cce47332bae9bbe10d1dc6f98
BLAKE2b-256 checksum
How to use checksums
5d7fab8d7029f2a9392dfadef9d38376fcc58d9149bf697ec317b448120ab396
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.18

Security for hfutils 0.15.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.15.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page