dexverse 0.1.1

DexVerse dataset downloader for S3-compatible robot training datasets

DexVerse Downloader (dexverse)

Presigned-URL batch downloader for robot-training datasets.

manifest -> dexverse-dl -> httpx -> presigned URLs -> local file landing

The primary input is the DexVerse backend's raw presigned-URL JSON. For those payloads, storage_path is the remote object key and file_name is the local landing path under --out.

Canonical manifests are still supported. Their local paths come from relativePath, falling back to originalPath, then objectKey:

localPath = out / (file.relativePath || file.originalPath || file.objectKey)

Install

pip install "dexverse[s3,mock]"          # boto3 + fastapi mock server
# optional legacy S3 backend:
#   install s5cmd (https://github.com/peak/s5cmd) and put it on PATH

Development:

uv venv && . .venv/bin/activate          # Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"

Quick Start

If your backend issues short-lived presigned URLs per file, save that response as download.json and run:

dexverse-dl download --manifest ./download.json --out D:\DexVerse

No login, no AWS_* credentials, no s5cmd, no backend flag, and no TLS flag are required for the common internal/self-signed endpoint case. The default backend is http, and TLS verification is disabled by default for http/presigned downloads. Use --secure when the endpoint has a valid certificate and you want certificate validation.

Raw backend output is accepted directly:

[{
  "download_url": "https://host/bucket/datasets/.../20260429_180359.hdf5?X-Amz-Signature=...",
  "storage_bucket": "dexverse",
  "storage_path": "datasets/.../20260429_180359.hdf5",
  "file_name": "20260429_180359.hdf5",
  "file_size": 14994636
}]

Result:

D:\DexVerse\20260429_180359.hdf5

Presigned URLs expire, often after 1 hour. Have the backend regenerate the manifest right before each download; expired URLs surface as per-file HTTP 403 errors.

CLI

dexverse-dl login   --server https://dexverse.xxx.com [--token TOKEN]
dexverse-dl download --manifest ./download.json --out /data/dexverse [--parallel 16] [--secure] [--backend http|python|s5cmd]
dexverse-dl download --task dl_xxx --out /data/dexverse [--server URL]
dexverse-dl verify  --manifest PATH --out /data/dexverse [--deep]
dexverse-dl repair  --manifest PATH --out /data/dexverse [--backend http|python|s5cmd]

Python SDK

from dexverse import DexVerseClient, download_manifest

client = DexVerseClient(server="https://dexverse.xxx.com", token="xxx")
local_root = client.download_task(task_id="dl_xxx", out="/data/dexverse", parallel=16)

# or from a local manifest file
download_manifest(manifest_path="./download.json", out="/data/dexverse")

Layout

State files live under <out>/.dexverse/ and never mix into the dataset tree:

/data/dexverse/
  .dexverse/tasks/<taskId>/
    manifest.json
    download-state.json
    download.log
  20260429_180359.hdf5

Backends

backend	engine	notes
http	httpx	default; presigned URLs, no creds/login
python	boto3	legacy S3 fallback
s5cmd	peak/s5cmd binary	legacy S3 high-throughput backend