gidata 0.3.0

GI Labs Data Platform CLI — upload and manage robotics data

gidata CLI Guide

gidata is a command-line tool for uploading and downloading robotics data on the GI Labs Data Platform. It supports uploading LeRobot datasets, video files, trajectory data, and downloading complete task bundles in 7 formats — annotation containers (Parquet/JSON/RLDS) or domain-standard recording layouts (LeRobot v3.0, HDF5/ALOHA, ROS 2 Rosbag, UMI passthrough) — for offline analysis, CI pipelines, or ML training. The full format guide lives in docs/user-guides/exporting-data.md.

Installation

Requires Python 3.11+.

# Recommended (isolated CLI install):
pipx install gidata

# Or with uv:
uv tool install gidata

# Or plain pip into the active environment:
pip install gidata

Verify the installation:

gidata --version

Upgrade later with the built-in self-upgrade command — it autodetects whether you installed via pipx, uv tool, or plain pip and runs the matching upgrade for you:

gidata upgrade            # interactive: prompts before installing
gidata upgrade --yes      # skip the prompt
gidata upgrade --check    # report whether an update is available, no install

If you prefer to drive the package manager yourself: pipx upgrade gidata, uv tool upgrade gidata, or pip install --upgrade gidata.

Local development

Working from a clone of this repo? Install the editable copy instead:

# From the repository root
pip install -e ./cli
# Or: uv pip install -e ./cli

Releasing a new version

Releases are cut from a cli-vX.Y.Z git tag. The Publish gidata CLI workflow (.github/workflows/cli-release.yml) builds sdist + wheel with uv build and pushes them to PyPI via Trusted Publishing — no API tokens are stored in GitHub.

# 1. Bump the version in cli/pyproject.toml AND cli/gidata/__init__.py
# 2. Commit and push to main
# 3. Tag and push:
git tag cli-v0.1.1
git push origin cli-v0.1.1

The workflow guards against tag/version drift and runs ruff + pytest before publishing. For dry runs, trigger the workflow manually with target=testpypi.

Authentication

Before uploading, you need to authenticate. There are two methods:

API Key (Recommended)

Generate an API key from the web UI at Settings > Account > API Keys, then:

gidata auth login --api-key YOUR_KEY --org-id YOUR_ORG_UUID

--url is optional and defaults to the GILabs Data Center backend, so you normally don't need it. Pass --url only to target a local or self-hosted backend (e.g. --url http://localhost:8000); it's saved for future commands.

Where to find your organization ID

Most CLI commands need an org id. Open the dashboard, click the organization name in the top-left of the sidebar to expand the switcher — the active org's UUID appears as a small monospace string under the member count. Click it to copy.

You can pass the UUID inline (--org-id YOUR_ORG_UUID on auth login) or set it independently afterwards:

gidata config set org-id YOUR_ORG_UUID    # one-time
gidata config show                         # verify the active settings

Email / Password

You can also log in with your Supabase email/password credentials:

gidata auth login \
  --url http://localhost:8000 \
  --supabase-url http://localhost:54321 \
  --supabase-anon-key YOUR_ANON_KEY

You'll be prompted for your email and password.

Check Status

gidata auth status

Shows the current auth method, API URL, and user info.

Log Out

gidata auth logout

Removes stored credentials from ~/.config/gidata/.

Uploading Data

All upload commands require authentication. Run gidata auth login first.

Upload a LeRobot Dataset

Upload a .zip file containing a LeRobot-formatted dataset to a task:

gidata upload dataset ./my_dataset.zip --task-id TASK_ID

The zip should follow the LeRobot v3.0 format:

dataset/
├── meta/
│   ├── info.json
│   ├── episodes.jsonl
│   └── tasks.jsonl
├── data/
│   └── episode_{idx}/
│       └── trajectory.parquet
└── videos/
    └── {camera_key}/
        └── episode_{idx}/
            └── file_{idx}.mp4

Upload Videos

Upload video files for a specific episode:

# Single video
gidata upload videos ./video.mp4 --episode-id EPISODE_ID

# Directory of videos
gidata upload videos ./camera_feeds/ --episode-id EPISODE_ID

Supported formats: .mp4, .avi, .mov, .mkv, .webm.

Upload Trajectory Data

Upload a .parquet trajectory file for an episode:

gidata upload trajectory ./trajectory.parquet --episode-id EPISODE_ID

Upload a .umi Episode

Upload a raw .umi recording (the format produced by Quest devices) to a task. The task must be in data_collection mode. The CLI opens a chunked upload session against /api/v1/upload/umi/sessions, streams the file in 2 MB chunks with Content-Range for resume support, then PUTs metadata and polls until the backend finishes converting to LeRobot format (demux → MP4 + trajectory parquet → Episode row).

# Minimal: infers episode name from the file stem, device_id defaults to "cli"
gidata upload umi ./episode_0.umi --task-id TASK_ID

# With a custom episode name and explicit device identifier
gidata upload umi ./episode_0.umi \
  --task-id TASK_ID \
  --episode-name pickup_run_42 \
  --device-id lab-quest-01

# Point at an explicit metadata.json (otherwise a sibling metadata.json next to the
# .umi file is used automatically, falling back to a minimal {task_id, episode_name} dict)
gidata upload umi ./episode_0.umi --task-id TASK_ID --metadata ./meta.json

# Fire-and-forget: return as soon as bytes are uploaded, don't wait for processing
gidata upload umi ./episode_0.umi --task-id TASK_ID --no-wait

By default the command waits up to 30 minutes for the server to finish processing and prints the resulting Episode ID on success. With --no-wait it exits immediately after metadata is submitted and prints the session ID so you can check status later.

Constraints enforced by the server:

• Task must exist in the current org and be data_collection mode
• Per-device/episode-name uniqueness for active (non-terminal) sessions
• File size capped by MAX_UMI_FILE_SIZE

Upload an Ego Episode (.tar or raw rig .umi)

Upload an ego payload to an ego_collection task. Two payload types are accepted:

• a phone-recorded .tar (produced by the ego-recorder phone app — H.265 video + IMU JSONL + camera_info JSON + metadata.json bundled together); or
• a raw RV1106 ego-rig .umi straight off the device. The server validates that it is an ego recording (an EQ_EGO camera, no pose stream — a gripper .umi is refused and pointed at upload umi) and repacks it into the ego layout before ingest.

Either way the flow mirrors UMI exactly: chunked session, 2 MiB chunks with Content-Range, server-side ingest into MCAP + Episode row.

# Phone .tar — episode name from file stem, device_id "cli"
gidata upload ego ./episode_0.tar --task-id TASK_ID

# Raw rig .umi — keep the rig's sibling metadata.json next to it (auto-discovered);
# the server translates that UMI-native blob into the fields it needs.
gidata upload ego ./session_00002/part_000.umi --task-id TASK_ID

# With explicit metadata
gidata upload ego ./episode_0.tar \
  --task-id TASK_ID \
  --episode-name kitchen_pour_01 \
  --device-id ego-phone-3 \
  --metadata ./episode_0.metadata.json

Constraints enforced by the server:

• Task must be ego_collection mode
• For a .umi: must be an ego-rig recording (not a gripper) — content-checked, not by extension
• Payload size ≤ MAX_EGO_FILE_SIZE (25 GiB by default — accommodates Pure Ego TARs that carry the raw RV1106 source/episode.umi alongside the demuxed artifacts)
• Per-device/episode-name uniqueness for active sessions

Downloading Task Data

The gidata download subcommands pull a task's uploaded data from the backend and write it to your local disk. Useful for running algorithms offline, training models, or archiving completed tasks.

List Available Tasks

Browse tasks in your organization to find the one you want to pull:

gidata download list                          # first 50 tasks
gidata download list --search "pickup"        # substring match on task name
gidata download list --status labeled         # filter by status
gidata download list --project-id PROJECT_ID  # filter by project

Output shows task id (truncated), name, status, project, and verified / labeled / collected progress counts.

Download a Task

# By name (resolved via substring search)
gidata download task "aloha_pickup"

# By UUID (exact)
gidata download task 550e8400-e29b-41d4-a716-446655440000

If a name matches multiple tasks, the CLI prints a disambiguation table and asks for the exact task id.

Default behavior — the CLI:

1. Asks the backend to bundle the task into a zip (annotations + trajectories + episode metadata + signed video URL manifest)
2. Streams the zip with a progress bar
3. Extracts it into ./<task-name>_<short-id>/
4. Prints a summary (episode count, annotation count, size)

Downsampling with --sample-rate

For large trajectories you often don't need every frame. Downsample on the server before downloading:

# Keep every other frame (halves the trajectory row count)
gidata download task my_task --sample-rate 0.5

# Keep every 10th frame
gidata download task my_task --sample-rate 0.1

# Equivalent using explicit integer stride
gidata download task my_task --frame-stride 10

--sample-rate R in (0, 1] is converted to frame_stride = round(1 / R). Use --frame-stride N directly if you prefer integer semantics (takes precedence if both are set).

Sampling Episodes

If a task has far more episodes than you need, pick a subset with one of the three mutually exclusive flags below. They filter episodes before packing — annotations and trajectories for unselected episodes are not downloaded. All three are orthogonal to --frame-stride / --sample-rate and can be combined with them.

# Every 3rd episode (ordered by episode_index)
gidata download task my_task --episode-stride 3

# Just the most recent 5 episodes
gidata download task my_task --last-episodes 5

# A representative sample of at most 10 episodes, evenly spaced across the task
gidata download task my_task --max-episodes 10

Pick only one — the CLI errors out if you pass more than one. The summary line after download shows N/M episodes so you can see how much of the task was sampled.

Format, Trajectory, and Output Options

# Annotation containers (legacy)
gidata download task my_task --format parquet           # default
gidata download task my_task --format json
gidata download task my_task --format rlds              # for TFDS / RLDS pipelines

# Recording formats (the data, not just the annotations)
gidata download task my_task --format lerobot           # HuggingFace LeRobot v3.0 dir tree
gidata download task my_task --format rosbag            # ROS 2 .db3 per episode (refs mode)
gidata download task my_task --format umi               # passthrough of original .umi files

# Async formats — CLI polls until the bundle is ready
gidata download task my_task --format hdf5              # ALOHA-style .h5 per episode
gidata download task my_task --format rosbag --inline-videos
gidata download task my_task --format hdf5 --inline-videos

# Skip trajectory parquets (metadata + annotations only)
gidata download task my_task --no-trajectory

# Custom output directory
gidata download task my_task --out /data/offline/my_task

# Keep the raw zip alongside the extracted tree
gidata download task my_task --keep-zip

The CLI runs a pre-flight availability check before submitting. If the format isn't applicable to the task (e.g. UMI on a LeRobot-only task) it hard-fails with the server's reason rather than silently producing an empty bundle:

$ gidata download task my-lerobot-task -f umi
'umi' export not available: UMI export requires UMI-collected episodes
with a stored raw .umi file; this task has none.

Bulk-download all passed / failed QC episodes

gidata download qc downloads every QC-passed (or QC-failed) episode across your whole organization, divided into per-task folders. Unlike the web QC page's Export button (which bundles your checkbox selection, or the whole filtered set capped at 2000 episodes), the CLI has no cap — it lists the episodes, groups them by task, and downloads each task's bundle one at a time, so it scales to any size.

# All passed episodes, default parquet container
gidata download qc --status passed

# All failed episodes as LeRobot, into a chosen directory
gidata download qc --status failed --format lerobot --out ./qc_failed

# Limit to tasks matching a search (name / canonical id / id)
gidata download qc --status passed --search "ballsincontainer"

Options: --status passed|failed (required), --format (parquet | json | rlds | lerobot | umi | rosbag — same sync formats as download task; async HDF5 / inline-Rosbag aren't offered here), --search, --no-trajectory, --frame-stride, --out (default ./qc_<status>/), --force. Output layout:

qc_passed/
└── tasks/
    ├── <task-name>_<id8>/   # that task's passed episodes, in the chosen format
    └── …

Requires the reviewer role (same gate as the web QC pages). Videos are referenced via signed URLs (manifest), matching download task.

Bundle Layouts

Annotation containers (parquet / json / rlds) — default

my_task_550e8400/
├── task.json                          # task + project metadata, progress, export params
├── annotations.parquet                # (or .json / .rlds.json depending on --format)
├── episodes/
│   ├── episode_0/
│   │   ├── metadata.json              # fps, length, duration, cameras, storage paths
│   │   └── trajectory.parquet         # downsampled per --sample-rate / --frame-stride
│   └── episode_1/
│       └── ...
└── videos_manifest.json               # signed MP4 URLs (valid 24h), per camera

LeRobot v3.0 (--format lerobot)

my_task_550e8400/
├── task.json
├── meta/info.json                     # robot_type, fps, features (cameras)
├── meta/episodes.jsonl                # one row per episode {episode_index, tasks, length}
├── meta/tasks.jsonl                   # action descriptions from annotations
├── data/chunk-000/episode_<idx>.parquet  # trajectory + task_index column
├── videos_manifest.json
└── skipped_episodes.json              # ego (mcap) eps that LeRobot can't carry

Round-trip-ingestible by the platform's own LeRobotIngestor.

HDF5 / ALOHA (--format hdf5, async)

my_task_550e8400/
├── task.json
├── episodes/
│   ├── episode_0.h5                   # /observations/qpos, /action, /timestamps, /annotations/segments
│   └── episode_1.h5                   # plus /observations/images/<cam> in --inline-videos mode
└── videos_manifest.json

ROS 2 Rosbag (--format rosbag)

my_task_550e8400/
├── task.json
├── episodes/
│   ├── episode_0/
│   │   ├── episode_0.db3              # ROS 2 sqlite3 bag
│   │   └── metadata.yaml              # rosbag2 sidecar
│   └── episode_1/...
└── videos_manifest.json

Topics: /joint_states, /imu (ego only), /annotations, /video_refs. With --inline-videos, also /<camera>/image_raw/compressed (JPEG per frame). Replay with ros2 bag play episode_0/.

UMI (--format umi)

my_task_550e8400/
├── task.json
├── episodes/
│   ├── episode_0.umi                  # byte-copy of raw_umi_path
│   └── episode_0.annotations.json     # per-episode annotations (when present)
└── missing_episodes.json              # episodes without a raw .umi

In refs mode (default), videos are not inlined — the zip contains videos_manifest.json with signed URLs you can fetch separately (e.g. with curl or wget). This keeps bundle sizes small. Pass --inline-videos for HDF5 / Rosbag to embed frames; that path runs as a background job (CLI polls automatically).

Configuration

Credentials and settings are stored in ~/.config/gidata/:

File	Contents
config.json	API URL and active organization id
credentials.json	API key or bearer token + (for email/password logins) supabase URL/anon key for auto-refresh (file permissions: 600)

Inspect or update settings via the config subcommand instead of editing the JSON by hand:

gidata config show                                              # print current settings
gidata config set api-url http://localhost:8000                 # override backend URL (default: GILabs Data Center)
gidata config set org-id 550e8400-e29b-41d4-a716-446655440000   # set active org
gidata config unset org-id                                      # clear an existing value

Troubleshooting

Permission denied: Not a member of this organization on any upload/download — the org id stored in ~/.config/gidata/config.json is wrong (a stale slug, a typo, or set to a non-UUID like testtest). The backend looks up OrganizationMember by exact UUID and a non-match returns 403 even though your API key / bearer token verified fine. Fix:

gidata config show                                  # confirm current org id
gidata config set org-id 550e8400-e29b-41d4-a716-446655440000

Find the correct UUID in the dashboard by clicking the organization name in the top-left of the sidebar — the active org's UUID appears as a small monospace string under the member count.

gidata auth status says "Authenticated" but uploads still 403 — auth status only verifies the credential against /users/me (which is org-agnostic). It does not validate the configured org id. A stale or wrong org-id will pass auth status and fail every org-scoped call.

Bearer token auto-refresh

Email/password logins persist supabase_url and supabase_anon_key alongside the access/refresh token pair. When a request returns 401 the CLI calls Supabase's /auth/v1/token?grant_type=refresh_token once, swaps in the new access token, and retries the original request — so a long-running shell stays authenticated past the access-token TTL without re-prompting. Streamed multipart uploads can't be replayed transparently; if a 401 lands mid-upload you'll need to re-run the command. API-key logins (the recommended path) never expire and skip the refresh path entirely.

Command Reference

gidata --version                              Show version
gidata auth login --api-key KEY [--url URL]   Log in with API key
gidata auth login [--url URL] [--supabase-*]  Log in with email/password (auto-refresh enabled)
gidata auth logout                            Clear stored credentials
gidata auth status                            Show auth state and user info
gidata config show                            Print current settings + file paths
gidata config set KEY VALUE                   Update api-url or org-id
gidata config unset KEY                       Remove a setting
gidata upload dataset PATH --task-id ID       Upload .zip LeRobot dataset (streamed)
gidata upload videos PATH --episode-id ID     Upload video file(s) (streamed)
gidata upload trajectory PATH --episode-id ID Upload .parquet trajectory
gidata upload umi PATH --task-id ID [opts]    Upload .umi episode (chunked, processed server-side)
gidata upload ego PATH --task-id ID [opts]    Upload ego .tar episode (chunked, ego_collection tasks only)
gidata download list [--search TEXT]          List tasks in the current org
gidata download task TASK [options]           Download a task's uploaded data
gidata download qc --status passed|failed     Download all passed/failed QC episodes (by task, uncapped)
gidata firmware publish APK ...               Publish a firmware APK release (platform admin)

gidata upload umi options

Flag	Description	Default
--task-id, -t	Target task ID (must be data_collection mode)	required
--device-id, -d	Source device identifier	cli
--episode-name, -n	Episode name — unique per device for active sessions	file stem
--metadata, -m	Path to metadata.json	sibling metadata.json, else minimal dict
--no-wait	Return after upload without polling for processing status	off

gidata upload ego options

Same flags as upload umi, but the task must be ego_collection mode and the file must be .tar.

gidata download task options

Flag	Description	Default
--format, -f	Export format: parquet, json, rlds (annotation containers) or lerobot, hdf5, rosbag, umi (recording formats). HDF5 + inline-mode Rosbag run as async jobs that the CLI polls automatically.	parquet
--inline-videos / --no-inline-videos	Embed video frames inside the bundle (HDF5/Rosbag only). Default off — videos referenced via signed URLs in videos_manifest.json. Enabling this switches to the async export-job path.	--no-inline-videos
--sample-rate, -r	Keep fraction of frames in (0, 1] — converted to integer stride	1.0 (all frames)
--frame-stride, -S	Explicit integer stride; takes precedence over --sample-rate	1
--episode-stride	Keep every Nth episode. Mutually exclusive with the two below.	off (all episodes)
--last-episodes	Keep only the last N episodes. Mutually exclusive with the two others.	off
--max-episodes	Cap at N episodes, evenly spaced. Mutually exclusive with the two others.	off
--no-trajectory	Skip trajectory parquets	off
--out, -o	Output directory	./<task-name>_<short-id>/
--keep-zip	Keep the downloaded zip file	off
--force, -y	Overwrite a non-empty --out directory without prompting	off

Examples

# Full workflow: authenticate and upload a dataset
gidata auth login --api-key gidata_abc123...
gidata upload dataset ./aloha_pickup.zip --task-id 550e8400-e29b-41d4-a716-446655440000

# Check who you're logged in as
gidata auth status

# Upload videos from a directory
gidata upload videos ./recordings/cam_left/ --episode-id ep-001

# Upload trajectory separately
gidata upload trajectory ./data/episode_0/trajectory.parquet --episode-id ep-001

# List labeled tasks and pull one for offline ML training
gidata download list --status labeled
gidata download task "aloha_pickup_v2" --sample-rate 0.5 --out ./training_data/

# Batch-pull several tasks with a shell loop
for t in pickup stack place; do
  gidata download task "$t" --format parquet --out ./data/$t
done