scigantic-transfer 0.1.0

Mirror a Hugging Face dataset into a region-local S3 bucket and load it into memory fast (with HF fallback + benchmark).

scigantic-transfer

Mirror a Hugging Face dataset into a region-local S3 bucket, then load it into memory fast on a GPU instance in that region.

Install

pip install scigantic-transfer

Mirror a dataset (Hugging Face to S3)

scigantic-transfer --region REGION --name HF_DATASET --bucket-name BUCKET [--include GLOB]

Flags:

--region        AWS region of the bucket and the GPU instance (must match)
--name          Hugging Face dataset id
--bucket-name   destination S3 bucket; data lands at s3://BUCKET/NAME/
--include GLOB  files to mirror, repeatable; default '*' (whole repo)

Examples:

scigantic-transfer --region us-east-1 --name org/ds --bucket-name my-mirror
scigantic-transfer --region us-east-1 --name org/ds --bucket-name my-mirror --include 'data/*'

Load into memory

from scigantic_transfer import load

result = load("org/ds", region="us-east-1", bucket="my-mirror")
data = result.blob

Benchmark

scigantic-transfer-bench --size-gib 1
scigantic-transfer-bench --live --region us-east-1 --name org/ds --bucket-name my-mirror

License

Proprietary, evaluation use only. No redistribution or derivative works. See LICENSE. Contact [e-mail].