dexverse 0.1.0

DexVerse dataset downloader for S3-compatible robot training datasets

DexVerse Downloader (dexverse)

S3-compatible batch downloader for robot-training datasets.

manifest → dexverse-dl → s5cmd → MinIO / S3 → original-path landing

The downloader never reorganizes your dataset directory. Local paths come from the manifest's relativePath (falling back to originalPath, then objectKey):

localPath = out / (file.relativePath || file.originalPath || file.objectKey)

Install

pip install "dexverse[s3,mock]"          # boto3 + fastapi mock server
# optional, recommended download kernel:
#   install s5cmd (https://github.com/peak/s5cmd) and put it on PATH

Development:

uv venv && . .venv/bin/activate          # Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"

Quick start — Local Manifest mode (Phase 0)

export AWS_ACCESS_KEY_ID="minio_access_key"
export AWS_SECRET_ACCESS_KEY="minio_secret_key"
export AWS_REGION="us-east-1"

dexverse-dl download --manifest ./manifest.json --out /data/dexverse

Result:

/data/dexverse/datasets/aaa_test/v1/episodes/episode_000001.hdf5

If s5cmd is not installed, fall back to the pure-Python backend:

dexverse-dl download --manifest ./manifest.json --out /data/dexverse --backend python

Quick start — Presigned-URL mode (no login, no S3 creds)

If your backend issues short-lived presigned URLs per file (self-authenticating HTTPS links), put them in the manifest and use the http backend. No login, no AWS_* credentials, no s5cmd:

dexverse-dl download --manifest ./manifest_presigned.json --out D:\DexVerse \
  --backend http --insecure

--insecure (-k) skips TLS verification for self-signed/internal endpoints like https://192.168.x.x:8443 — drop it when the endpoint has valid TLS.

Each manifest file just needs objectKey, relativePath, size, and presignedUrl:

{
  "taskId": "dl_presigned",
  "files": [{
    "objectKey": "datasets/.../20260429_180359.hdf5",
    "relativePath": "datasets/.../20260429_180359.hdf5",
    "size": 14994636,
    "presignedUrl": "https://host/bucket/datasets/.../20260429_180359.hdf5?X-Amz-Signature=..."
  }]
}

Presigned URLs expire (often 1h). Have the backend regenerate the manifest right before each download; expired URLs surface as per-file HTTP 403.

Quick start — Mock Backend mode (Phase 1)

# terminal 1 — run the mock server
dexverse-mock-server --host 0.0.0.0 --port 18080

# terminal 2 — login + download by taskId
dexverse-dl login --server http://localhost:18080 --token dev-token
dexverse-dl download --task dl_demo_001 --out /data/dexverse

CLI

dexverse-dl login   --server https://dexverse.xxx.com [--token TOKEN]
dexverse-dl download --manifest ./manifest.json --out /data/dexverse [--parallel 16] [--backend s5cmd|python]
dexverse-dl download --task dl_xxx --out /data/dexverse [--server URL]
dexverse-dl verify  --manifest PATH --out /data/dexverse [--deep]
dexverse-dl repair  --manifest PATH --out /data/dexverse [--backend python]

Python SDK

from dexverse import DexVerseClient, download_manifest

client = DexVerseClient(server="https://dexverse.xxx.com", token="xxx")
local_root = client.download_task(task_id="dl_xxx", out="/data/dexverse", parallel=16)

# or from a local manifest file
download_manifest(manifest_path="./manifest.json", out="/data/dexverse", backend="s5cmd")

Layout of the download root

State files live under <out>/.dexverse/ and never mix into the dataset tree:

/data/dexverse/
  .dexverse/tasks/<taskId>/
    manifest.json
    download.s5cmd
    download-state.json
    download.log
  datasets/aaa_test/v1/episodes/episode_000001.hdf5   # original paths preserved

Backends

backend	engine	notes
s5cmd	peak/s5cmd binary	default, high-throughput, needs binary
python	boto3	fallback, no external binary required
http	httpx	presigned URLs only — no creds, no login