DexVerse Downloader (dexverse)

Presigned-URL batch downloader for robot-training datasets.

manifest -> dexverse-dl -> httpx -> presigned URLs -> local file landing

The primary input is the DexVerse backend's raw presigned-URL JSON. For those payloads, storage_path is the remote object key and file_name is the local landing path under --out.

Canonical manifests are still supported. Their local paths come from relativePath, falling back to originalPath, then objectKey:

localPath = out / (file.relativePath || file.originalPath || file.objectKey)

Install

pip install "dexverse[s3,mock]"          # boto3 + fastapi mock server
# optional legacy S3 backend:
#   install s5cmd (https://github.com/peak/s5cmd) and put it on PATH

Development:

uv venv && . .venv/bin/activate          # Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"

Quick Start

If your backend issues short-lived presigned URLs per file, save that response as download.json and run:

dexverse-dl download --manifest ./download.json --out D:\DexVerse

No login, no AWS_* credentials, no s5cmd, no backend flag, and no TLS flag are required for the common internal/self-signed endpoint case. The default backend is http, and TLS verification is disabled by default for http/presigned downloads. Use --secure when the endpoint has a valid certificate and you want certificate validation.

Raw backend output is accepted directly:

[{
  "download_url": "https://host/bucket/datasets/.../20260429_180359.hdf5?X-Amz-Signature=...",
  "storage_bucket": "dexverse",
  "storage_path": "datasets/.../20260429_180359.hdf5",
  "file_name": "20260429_180359.hdf5",
  "file_size": 14994636,
  "task_instruction": {
    "aaa": "111",
    "aaaa": 222,
    "ewrwrwe": 1.1111111111111112e+54
  }
}]

Result:

D:\DexVerse\20260429_180359.hdf5

task_instruction is optional and must be a JSON object. If multiple manifest items resolve to the same local path (for example, repeated file_name values), the first item wins and later duplicates are ignored.

Include file_size (and preferably sha256) for large files. The HTTP backend also learns the total size from Content-Length/Content-Range, but manifest metadata lets the CLI verify an existing local file before making a request.

Transient timeouts and HTTP 408/425/429/5xx responses are retried five times by default. Bytes already received stay in a sibling <file>.part; retries and later CLI runs request the remaining bytes with HTTP Range, while the final destination path only appears after the size check succeeds. The object server must support Range requests for efficient resume.

Presigned URLs expire, often after 1 hour. Have the backend regenerate the manifest right before each download; expired URLs surface as per-file HTTP 403 errors. After generating a fresh download.json, run the same command with the same --out path to continue from the retained .part.

CLI

dexverse-dl login   --server https://dexverse.xxx.com [--token TOKEN]
dexverse-dl download --manifest ./download.json --out /data/dexverse [--parallel 16] [--secure] [--backend http|python|s5cmd]
dexverse-dl download --manifest ./download.json --out /data/dexverse [--http-read-timeout 120] [--http-retries 5]
dexverse-dl download --task dl_xxx --out /data/dexverse [--server URL]
dexverse-dl verify  --manifest PATH --out /data/dexverse [--deep]
dexverse-dl repair  --manifest PATH --out /data/dexverse [--backend http|python|s5cmd]

还原 HDF5 → 可训练数据

下载完成后，把落盘的 .hdf5 还原为原始录制结构（逐帧 JPEG + pose_record_*.json

• metadata.jsonl）。如果对应 manifest 带有 task_instruction，还会自动生成 task_instruction.json：

# 单文件
dexverse-dl convert -i /data/dexverse/20260429_180359.hdf5 -o /data/dexverse/restored
# 批量（递归处理目录下所有 *.hdf5，保持目录结构）
dexverse-dl convert --input-dir /data/dexverse -o /data/dexverse/restored

还原输出形如：

/data/dexverse/restored/20260429_180359/
  head/left/000000.jpg ...
  head/right/000000.jpg ...
  hand/left/000000.jpg ...
  hand/right/000000.jpg ...
  pose_record_20260429_180359.json
  metadata.jsonl
  task_instruction.json

task_instruction.json 的内容就是 manifest 中的对象本身，不增加外层包装。 下载器会把转换所需的映射保存在 .dexverse/task-instructions/，无需为 convert 增加额外参数。独立来源且没有该隐藏元数据的 HDF5 仍按原行为还原。

该命令依赖 h5py / numpy / av / pillow，已列入核心依赖，安装时随包装上。

Python SDK

from dexverse import DexVerseClient, download_manifest

client = DexVerseClient(server="https://dexverse.xxx.com", token="xxx")
local_root = client.download_task(task_id="dl_xxx", out="/data/dexverse", parallel=16)

# or from a local manifest file
download_manifest(manifest_path="./download.json", out="/data/dexverse")

Layout

State files live under <out>/.dexverse/ and never mix into the dataset tree:

/data/dexverse/
  .dexverse/task-instructions/
    20260429_180359.hdf5.json
  .dexverse/tasks/<taskId>/
    manifest.json
    download-state.json
    download.log
  20260429_180359.hdf5

Backends