HF-fastup
Pushes a HF dataset to the HF hub as a Parquet dataset, allowing streaming. The dataset is processed to shards and uploaded in parallel. It useful for large datasets, for example, with embedded data.
Usage
Make sure hf_transfer is installed and HF_HUB_ENABLE_HF_TRANSFER is set to 1.
import hffastup
import datasets
datasets.logging.set_verbosity_info()
# load any HF dataset
dataset = datasets.load_dataset("my_large_dataset.py")
hffastup.upload_to_hf_hub(dataset, "Org/repo") # upload to HF Hub
hffastup.push_dataset_card(dataset, "Org/repo") # Makes a dataset card and pushes it to HF Hub
1 maintainer
Release files for hf-fastup 0.0.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| hf-fastup-0.0.7.tar.gz | 5.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| hf_fastup-0.0.7-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 12.2 kB
Release files / hf-fastup-0.0.7.tar.gz
| Download URL | hf-fastup-0.0.7.tar.gz |
|---|---|
| Size | 5.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fda4046498680ab173ed5147d847b85657434331900e787d9da73f297c3bca10
|
|
BLAKE2b-256 checksum How to use checksums |
eca539d0568aae1a34384011294f041d4e1b967d1c97f2f65f5734f2c58d5bac
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/4.0.2 CPython/3.11.8
|
Release files / hf_fastup-0.0.7-py3-none-any.whl
| Download URL | hf_fastup-0.0.7-py3-none-any.whl |
|---|---|
| Size | 6.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
861a57cc1b690de39ffdbdda1d77b3c3f28beb180a7e88df560cf5e51eb87e6f
|
|
BLAKE2b-256 checksum How to use checksums |
2466c04abf09fa7a2945f83d449b7a51bc6658852563af91e77787ef24604287
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/4.0.2 CPython/3.11.8
|
