dexverse-cli

dexverse-cli downloads DexVerse dataset manifests over HTTP(S), packs strict raw sessions into versioned HDF5 files, and unpacks HDF5 sessions. It requires Python 3.10 or newer.

Install

pip install dexverse-cli

Download

Read a manifest from a file:

dexverse-dl download --manifest manifest.json --out ./dataset

Or pipe a short-lived manifest directly to stdin:

type manifest.json | dexverse-dl download --manifest - --out ./dataset

A manifest is either a JSON file list or an object containing files. Each file requires download_url and file_name. Optional fields are storage_path, file_size, sha256, and meta_data. Downloaded metadata is stored in a versioned sidecar and becomes task_instruction.json when the corresponding HDF5 file is unpacked; only top-level data_protocol is removed from that algorithm-facing projection.

Downloads use bounded parallelism, retries, and compatible partial-file resume. When sha256 is present it is always checked before the final file is replaced. TLS certificate verification is disabled by default for internal endpoints; pass --secure to enable certificate verification.

Pack

Pack one raw session, or the immediate session directories under a parent:

dexverse-dl pack --input ./raw/session --output ./packed --format jpeg

JPEG is the platform-independent default. Packing validates the complete raw inventory, every image, all camera stream relationships, and pose data before transactionally replacing an output. Camera and pose counts may differ; pose is linearly aligned to the deterministic reference camera timeline.

Jetson MP4 packing is optional:

pip install "dexverse-cli[jetson]"
dexverse-dl pack --input ./raw/session --output ./packed --format video

The Python extra does not install native Jetson GStreamer components. Video packing reports an explicit error unless the required hardware pipeline is available. JPEG packing, download, help, and unpack do not import GStreamer.

Unpack

Unpack one HDF5 file:

dexverse-dl unpack --input ./packed/session.hdf5 --output ./restored

Or recursively unpack *.hdf5 files while preserving their relative paths:

dexverse-dl unpack --input-dir ./packed --output ./restored

Unpacking validates the complete versioned schema before transactionally publishing a restored session. It also retains the supported unversioned HDF5 read path.

Convert

Convert one supported HDF5 file:

dexverse-dl convert --input ./dataset/session.hdf5 --output ./restored

Or recursively convert *.hdf5 files while preserving their relative paths:

dexverse-dl convert --input-dir ./dataset --output ./restored

convert is the compatibility alias for unpack; it retains the same options, target mapping, legacy HDF5 support, exit codes, and converted=... summary. Batches continue after an item failure but exit with status 1 if any item fails.