hfutils 0.0.2

Useful utilities for huggingface

hfutils

Useful utilities for huggingface

Quick Start

To get started with hfutils, you can install it using PyPI:

pip install hfutils

Alternatively, you can install it from the source code:

git clone https://github.com/deepghs/hfutils.git
cd hfutils
pip install .

Verify the installation by checking the version:

hfutils -v

If Python is not available in your local environment, we recommend downloading the pre-compiled executable version from the releases.

Upload Data

Upload data to repositories using the following commands:

# Upload a single file to the repository
hfutils upload -r your/repository -i /your/local/file -f file/in/your/repo

# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip

# Upload files in a directory as a directory tree to the repository
hfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo

You can achieve the same using the Python API:

from hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory

# Upload a single file to the repository
upload_file_to_file(
    local_file='/your/local/file',
    repo_id='your/repository',
    file_in_repo='file/in/your/repo'
)

# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
upload_directory_as_archive(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    archive_in_repo='archive/file/in/your/repo.zip'
)

# Upload files in a directory as a directory tree to the repository
upload_directory_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    path_in_repo='dir/in/your/repo'
)

Explore additional options for uploading:

hfutils upload -h

Download Data

Download data from repositories using the following commands:

# Download a single file from the repository
hfutils download -r your/repository -o /your/local/file -f file/in/your/repo

# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip

# Download files from the repository as a directory tree
hfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo

Use the Python API for the same functionality:

from hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory

# Download a single file from the repository
download_file_to_file(
    local_file='/your/local/file',
    repo_id='your/repository',
    file_in_repo='file/in/your/repo'
)

# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
download_archive_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    file_in_repo='archive/file/in/your/repo.zip',
)

# Download files from the repository as a directory tree
download_directory_as_directory(
    local_directory='/your/local/directory',
    repo_id='your/repository',
    dir_in_repo='dir/in/your/repo'
)

Explore additional options for downloading:

hfutils download -h