scigantic-transfer 0.1.1

Mirror a Hugging Face dataset into a region-local S3 bucket and load it into memory fast (with HF fallback + benchmark).

scigantic-transfer

Mirror a Hugging Face dataset into a region-local S3 bucket, then load it into memory fast on a GPU instance in that region.

Install

pip install scigantic-transfer

Mirror a dataset (Hugging Face to S3)

scigantic-transfer --region REGION --name HF_DATASET --bucket-name BUCKET [--include GLOB] [--endpoint-url URL]

Flags:

--region         region of the bucket and the GPU instance (must match)
--name           Hugging Face dataset id
--bucket-name    destination S3 bucket; data lands at s3://BUCKET/NAME/
--include GLOB   files to mirror, repeatable; default '*' (whole repo)
--endpoint-url   S3-compatible endpoint for neoclouds; omit on AWS (see below)

Examples:

scigantic-transfer --region us-east-1 --name org/ds --bucket-name my-mirror
scigantic-transfer --region us-east-1 --name org/ds --bucket-name my-mirror --include 'data/*'

Neoclouds (Nebius and other non-AWS GPU clouds)

--endpoint-url is for neoclouds. The speed win only exists when the bucket and the GPU instance are in the same cloud and region, so on a neocloud you mirror into that cloud's own S3-compatible object storage and point the tool at its endpoint. On AWS, omit the flag.

scigantic-transfer --region eu-north1 --name org/ds --bucket-name my-mirror \
    --endpoint-url https://storage.eu-north1.nebius.cloud

Works against any S3-compatible store (Nebius, MinIO, Cloudflare R2). Credentials are read the usual way (env vars / config) and apply to that endpoint.

Load into memory

from scigantic_transfer import load

result = load("org/ds", region="us-east-1", bucket="my-mirror")
data = result.blob

Benchmark

scigantic-transfer-bench --size-gib 1
scigantic-transfer-bench --live --region us-east-1 --name org/ds --bucket-name my-mirror

License

Proprietary, evaluation use only. No redistribution or derivative works. See LICENSE. Contact [e-mail].