Toolkit for managing huggingface models and datasets
Project description
huggingface_tool
Tools for loading, upload, managing huggingface models and datasets
Installation
pip install huggingface-tool
Usage
- Download and save transformer models with:
htool save-model <model_class> <model_name> <save_dir>
- For example:
htool save-model AutoModelForCausalLM gpt2 ./gpt2
- For example:
- Download and save tokenizer with:
htool save-tk <tokenizer_name> <save_dir>
- For example:
htool save-tk gpt2 ./gpt2
- For example:
- Download and save dataset with:
htool save-data <dataset_name> <save_dir>
- For example:
htool save-data daily_dialog ./daily_dialog
- For example:
- Download and save diffusion models with:
htool save-dm <model_name> <save_dir>
- For example:
htool save-dm google/ddpm-cat-256 ./google/
- For example:
You can also use htool to upload datasets and models to huggingface.
Firstly, you need to login with huggingface-cli login
(you can create or find your token at settings).
- Upload dataset with:
htool upload-data <local_dataset_dir> <organization_or_username/dataset_name>
- For example:
htool upload-data ./daily_dialog OpenRL/daily_dialog
- For example:
Citing huggingface_tool
If our work has been helpful to you, please feel free to cite us:
@misc{huggingface_tool2023,
title={huggingface_tool},
author={OpenRL Contributors},
publisher = {GitHub},
howpublished = {\url{https://github.com/OpenRL-Lab/huggingface_tool}},
year={2023},
}
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
huggingface_tool-0.0.3.tar.gz
(11.6 kB
view hashes)
Built Distribution
Close
Hashes for huggingface_tool-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a21062618aede8371ada10026e4b93884fde5aabdb74c33d53bac70d68daa3a |
|
MD5 | c14696f2380520e3663a3e97dfd70966 |
|
BLAKE2b-256 | 6df7b0133bb980c232274515d1c4b7501439639e03166f3780036261e121ded7 |