Skip to main content

HF-fastup

Pushes a HF dataset to the HF hub as a Parquet dataset, allowing streaming. The dataset is processed to shards and uploaded in parallel. It useful for large datasets, for example, with embedded data.

Usage

Make sure hf_transfer is installed and HF_HUB_ENABLE_HF_TRANSFER is set to 1.

import hffastup
import datasets
datasets.logging.set_verbosity_info()

# load any HF dataset
dataset = datasets.load_dataset("my_large_dataset.py")

hffastup.upload_to_hf_hub(dataset, "Org/repo") # upload to HF Hub
hffastup.push_dataset_card(dataset, "Org/repo") # Makes a dataset card and pushes it to HF Hub

Metadata

Release files for hf-fastup 0.0.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hf-fastup 0.0.7
File Size Uploaded Provenance Actions
hf-fastup-0.0.7.tar.gz 5.9 kB Without provenance Details for hf-fastup-0.0.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hf-fastup 0.0.7
File Interpreter ABI Platform Actions
hf_fastup-0.0.7-py3-none-any.whl Python 3 none any Details for hf_fastup-0.0.7-py3-none-any.whl

Total release size: 12.2 kB

Release files / hf-fastup-0.0.7.tar.gz

Download URL hf-fastup-0.0.7.tar.gz
Size 5.9 kB
Tags Source
SHA-256 checksum
How to use checksums
fda4046498680ab173ed5147d847b85657434331900e787d9da73f297c3bca10
BLAKE2b-256 checksum
How to use checksums
eca539d0568aae1a34384011294f041d4e1b967d1c97f2f65f5734f2c58d5bac
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/4.0.2 CPython/3.11.8

Release files / hf_fastup-0.0.7-py3-none-any.whl

Download URL hf_fastup-0.0.7-py3-none-any.whl
Size 6.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
861a57cc1b690de39ffdbdda1d77b3c3f28beb180a7e88df560cf5e51eb87e6f
BLAKE2b-256 checksum
How to use checksums
2466c04abf09fa7a2945f83d449b7a51bc6658852563af91e77787ef24604287
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/4.0.2 CPython/3.11.8

Security for hf-fastup 0.0.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page