dexverse 0.2.2

DexVerse dataset downloader for S3-compatible robot training datasets

DexVerse Downloader (dexverse)

Presigned-URL batch downloader for robot-training datasets.

manifest -> dexverse-dl -> httpx -> presigned URLs -> local file landing

The primary input is the DexVerse backend's raw presigned-URL JSON. For those payloads, storage_path is the remote object key and file_name is the local landing path under --out.

Canonical manifests are still supported. Their local paths come from relativePath, falling back to originalPath, then objectKey:

localPath = out / (file.relativePath || file.originalPath || file.objectKey)

Install

pip install "dexverse[s3,mock]"          # boto3 + fastapi mock server
# optional legacy S3 backend:
#   install s5cmd (https://github.com/peak/s5cmd) and put it on PATH

Development:

uv venv && . .venv/bin/activate          # Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"

Quick Start

If your backend issues short-lived presigned URLs per file, save that response as download.json and run:

dexverse-dl download --manifest ./download.json --out D:\DexVerse

No login, no AWS_* credentials, no s5cmd, no backend flag, and no TLS flag are required for the common internal/self-signed endpoint case. The default backend is http, and TLS verification is disabled by default for http/presigned downloads. Use --secure when the endpoint has a valid certificate and you want certificate validation.

Raw backend output is accepted directly:

[{
  "download_url": "https://host/bucket/datasets/.../20260429_180359.hdf5?X-Amz-Signature=...",
  "storage_bucket": "dexverse",
  "storage_path": "datasets/.../20260429_180359.hdf5",
  "file_name": "20260429_180359.hdf5",
  "file_size": 14994636,
  "task_instruction": {
    "aaa": "111",
    "aaaa": 222,
    "ewrwrwe": 1.1111111111111112e+54
  }
}]

Result:

D:\DexVerse\20260429_180359.hdf5

task_instruction is optional and must be a JSON object. If multiple manifest items resolve to the same local path (for example, repeated file_name values), the first item wins and later duplicates are ignored.

Presigned URLs expire, often after 1 hour. Have the backend regenerate the manifest right before each download; expired URLs surface as per-file HTTP 403 errors.

CLI

dexverse-dl login   --server https://dexverse.xxx.com [--token TOKEN]
dexverse-dl download --manifest ./download.json --out /data/dexverse [--parallel 16] [--secure] [--backend http|python|s5cmd]
dexverse-dl download --task dl_xxx --out /data/dexverse [--server URL]
dexverse-dl verify  --manifest PATH --out /data/dexverse [--deep]
dexverse-dl repair  --manifest PATH --out /data/dexverse [--backend http|python|s5cmd]

还原 HDF5 → 可训练数据

下载完成后，把落盘的 .hdf5 还原为原始录制结构（逐帧 JPEG + pose_record_*.json

• metadata.jsonl）。如果对应 manifest 带有 task_instruction，还会自动生成 task_instruction.json：

# 单文件
dexverse-dl convert -i /data/dexverse/20260429_180359.hdf5 -o /data/dexverse/restored
# 批量（递归处理目录下所有 *.hdf5，保持目录结构）
dexverse-dl convert --input-dir /data/dexverse -o /data/dexverse/restored

还原输出形如：

/data/dexverse/restored/20260429_180359/
  head/left/000000.jpg ...
  head/right/000000.jpg ...
  hand/left/000000.jpg ...
  hand/right/000000.jpg ...
  pose_record_20260429_180359.json
  metadata.jsonl
  task_instruction.json

task_instruction.json 的内容就是 manifest 中的对象本身，不增加外层包装。 下载器会把转换所需的映射保存在 .dexverse/task-instructions/，无需为 convert 增加额外参数。独立来源且没有该隐藏元数据的 HDF5 仍按原行为还原。

该命令依赖 h5py / numpy / av / pillow，已列入核心依赖，安装时随包装上。

Python SDK

from dexverse import DexVerseClient, download_manifest

client = DexVerseClient(server="https://dexverse.xxx.com", token="xxx")
local_root = client.download_task(task_id="dl_xxx", out="/data/dexverse", parallel=16)

# or from a local manifest file
download_manifest(manifest_path="./download.json", out="/data/dexverse")

Layout

State files live under <out>/.dexverse/ and never mix into the dataset tree:

/data/dexverse/
  .dexverse/task-instructions/
    20260429_180359.hdf5.json
  .dexverse/tasks/<taskId>/
    manifest.json
    download-state.json
    download.log
  20260429_180359.hdf5

Backends

backend	engine	notes
http	httpx	default; presigned URLs, no creds/login
python	boto3	legacy S3 fallback
s5cmd	peak/s5cmd binary	legacy S3 high-throughput backend