Toolkit for managing huggingface models and datasets
Project description
huggingface_tool
Tools for loading, upload, managing huggingface models and datasets
Installation
pip install huggingface-tool
Usage
Firstly, you need to login with huggingface-cli login
(you can create or find your token at settings).
- Download and save a repo with:
htool save-repo <repo_id> <save_dir> -r <model/dataset>
.-r
means the repo is a model or dataset repo. By default, it is a model repo.- For example:
htool save-repo OpenRL/tizero ./tizero
- For example:
htool save-repo OpenRL/DeepFakeFace ./DeepFakeFace -r dataset
- For example:
- Download and save a file with:
htool save-file <repo_id>:<remote_filepath> <save_dir> -r <model/dataset>
.-r
means the repo is a model or dataset repo. By default, it is a model repo.- For example:
htool save-file OpenRL/tizero:actor.pt ./tizero
- For example:
htool save-file OpenRL/DeepFakeFace:README.md ./DeepFakeFace -r dataset
- For example:
- Download and save transformer models with:
htool save-model <model_class> <model_name> <save_dir>
- For example:
htool save-model AutoModelForCausalLM gpt2 ./gpt2
- For example:
- Download and save tokenizer with:
htool save-tk <tokenizer_name> <save_dir>
- For example:
htool save-tk gpt2 ./gpt2
- For example:
- Download and save dataset with:
htool save-data <dataset_name> <save_dir>
- For example:
htool save-data daily_dialog ./daily_dialog
- For example:
- Download and save diffusion models with:
htool save-dm <model_name> <save_dir>
- For example:
htool save-dm google/ddpm-cat-256 ./google/
- For example:
You can also use htool to upload datasets and models to huggingface.
- Upload dataset with:
htool upload-data <local_dataset_dir> <organization_or_username/dataset_name>
- For example:
htool upload-data ./daily_dialog OpenRL/daily_dialog
- For example:
- Upload model with:
htool upload-model <local_model_dir> <organization_or_username/model_name>
- For example:
htool upload-model ./tizero OpenRL/tizero
- For example:
Citing huggingface_tool
If our work has been helpful to you, please feel free to cite us:
@misc{huggingface_tool2023,
title={huggingface_tool},
author={OpenRL Contributors},
publisher = {GitHub},
howpublished = {\url{https://github.com/OpenRL-Lab/huggingface_tool}},
year={2023},
}
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
huggingface_tool-0.0.5.tar.gz
(13.1 kB
view hashes)
Built Distribution
Close
Hashes for huggingface_tool-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50f17905ca78254fb11a1092d74c632c528c11eaf1bd34179fbc8e90937e6416 |
|
MD5 | 6dd6ec270a05a167eacba9162ec89f58 |
|
BLAKE2b-256 | 43b202956d87043fcb180675e5c661d0b8a1004dd13cf9625b95093dfc4f2471 |