dexverse-cli

dexverse-cli downloads DexVerse dataset manifests over HTTP(S), packs strict raw sessions into versioned HDF5 files, and unpacks HDF5 sessions. It requires Python 3.10 or newer.

Install

pip install dexverse-cli

Download

Read a manifest from a file:

dexverse-dl download --manifest manifest.json --out ./dataset

Or pipe a short-lived manifest directly to stdin:

type manifest.json | dexverse-dl download --manifest - --out ./dataset

A manifest is either a JSON file list or an object containing files. Each file requires download_url and file_name. Optional fields are storage_path, file_size, sha256, meta_data, and data_protocol. Unknown top-level keys next to files (for example the query_conditions / export_info blocks written by the web batch export) are ignored. data_protocol is retained in the downloaded metadata sidecar. Downloaded metadata is stored in a versioned sidecar and becomes task_instruction.json when the corresponding HDF5 file is unpacked; only top-level data_protocol is removed from that algorithm-facing projection.

Downloads use bounded parallelism, retries, and compatible partial-file resume. Within a manifest, the first occurrence of a target path keeps its name; later occurrences receive -1, -2, etc. before the final extension (for example, episode1.hdf5, episode1-1.hdf5, episode1-2.hdf5). Numbering skips names and paths already claimed by other entries, including partial files and metadata. Different relative directories remain independent. Names are assigned in manifest order, with Windows case-insensitive matching, before any downloads start. Rerunning the same manifest preserves the assigned names for skipping and resume; existing files on disk do not trigger new suffixes. This is not cross-batch name allocation: changing the manifest entries or their order can change the mapping. When sha256 is present it is always checked before the final file is replaced. TLS certificate verification is disabled by default for internal endpoints; pass --secure to enable certificate verification.

Pack

Pack one raw session, or the immediate session directories under a parent:

dexverse-dl pack --input ./raw/session --output ./packed --format jpeg

JPEG is the platform-independent default. Packing validates the complete raw inventory, every image, all camera stream relationships, and pose data before transactionally replacing an output. Camera and pose counts may differ; pose is linearly aligned to the deterministic reference camera timeline.

Jetson MP4 packing is optional:

pip install "dexverse-cli[jetson]"
dexverse-dl pack --input ./raw/session --output ./packed --format video

The Python extra does not install native Jetson GStreamer components. Video packing reports an explicit error unless the required hardware pipeline is available. JPEG packing, download, help, and unpack do not import GStreamer.

Unpack

Unpack one HDF5 file:

dexverse-dl unpack --input ./packed/session.hdf5 --output ./restored

Or recursively unpack *.hdf5 files while preserving their relative paths:

dexverse-dl unpack --input-dir ./packed --output ./restored

Unpacking validates the complete versioned schema before transactionally publishing a restored session. It also retains the supported unversioned HDF5 read path.

Convert

Convert one supported HDF5 file:

dexverse-dl convert --input ./dataset/session.hdf5 --output ./restored

Or recursively convert *.hdf5 files while preserving their relative paths:

dexverse-dl convert --input-dir ./dataset --output ./restored

convert is the compatibility alias for unpack; it retains the same options, target mapping, legacy HDF5 support, exit codes, and converted=... summary. Batches continue after an item failure but exit with status 1 if any item fails.