gi-ingest

Offload robotics recordings from TF cards and deliver them to GILabs.

Install

Not on PyPI yet — there is no release workflow for this package ( .github/workflows/cli-release.yml publishes gidata only). Install from a checkout:

git clone https://github.com/General-Intelligence-Labs/gilabs-data-center.git
uv tool install ./gilabs-data-center/vendor-cli    # or: pipx install ./…/vendor-cli
gi-ingest --help

Needs Python ≥ 3.11. Both commands put gi-ingest on your PATH in its own isolated environment; uv tool install --force … upgrades in place after a git pull.

Once the package is published this becomes uv tool install gi-ingest (or pipx install gi-ingest) with no path — until then that command fails with "No matching distribution found", which means it is not published, not that anything is wrong with your setup.

Then authenticate:

gi-ingest login --api-key gik_...
gi-ingest doctor      # verifies connectivity, clock skew, disk, credentials

Get your token from the vendor portal at vendor.gilabs.xyz → Upload tokens.

The three phases

They are three commands because they run on two different timescales, and fusing them turns a 30-minute attended task into an all-night one.

# 1. Drain every card. Swap cards as each finishes; nothing is wiped yet.
gi-ingest stage --device umi_gripper
gi-ingest stage --watch                 # or let it auto-copy on insert

# 2. Leave this running. One delivery, resumable, safe to nohup.
gi-ingest upload --max-bandwidth 50M

# 3. Next morning, with the cards back in the reader.
gi-ingest reclaim

Why the wipe is a separate step

upload finishes hours after the cards were pulled, so it cannot wipe them — they are back in the rig. stage writes a .staged.json marker and leaves the card intact; reclaim deletes part files only for episodes the server reports as delivered, and writes a tombstone so a re-scan skips them.

Between stage and reclaim, the staging disk holds the only copy. Size it for a full collection round; gi-ingest doctor warns under 50 GiB free.

Choosing where cards are copied to

By default cards land in ~/gilabs-staging. That is rarely where you want a multi-terabyte collection round, so it is settable two ways:

# Persist it — do this once per machine.
gi-ingest config set staging-dir /Volumes/BigDisk/gilabs
gi-ingest config show

# Or override for a single run, without changing the stored setting.
gi-ingest stage --staging /Volumes/OtherDisk/gilabs

config set also accepts ledger-path, api-url and project-id. Commands that read the ledger take --ledger for a one-off override, which is how you keep two independent offload runs from sharing state on one machine.

Other commands

gi-ingest whoami          # vendor, scopes, and where you may deliver
gi-ingest destinations    # the org/project list, with ids for --project
gi-ingest doctor          # token, connectivity, disk — run before a night
gi-ingest queue           # what is staged / in flight / delivered
gi-ingest status          # deliveries and their QC verdicts
gi-ingest status --batch btch_...
gi-ingest retry           # re-queue anything that failed

How a delivery works

1. upload opens a batch and receives AWS credentials scoped to that batch's folder alone — they cannot read or write anything else, including your own other deliveries.
2. Files go straight to S3. Credentials expire hourly and are re-minted automatically, so a multi-hour run and a laptop that sleeps both just work.
3. upload seals the batch with a manifest listing every episode, part, size and SHA-256. GILabs verifies every declared file is present at its declared size before ingesting anything, so a half-finished upload can never be processed as if it were complete.

Interrupted? Re-run upload. It re-uses the same open delivery and skips files already sent — a restart costs a listing, not the bytes.

Choosing a destination

Every delivery is filed against one project. With a single granted destination, or one marked default, upload needs no flag. With several and no default it will refuse to guess:

gi-ingest destinations
gi-ingest upload --project proj_abc123

Duplicates

stage asks GILabs which episodes it already has before copying, so a card a colleague already delivered costs one request instead of its bytes. If you are offline it stages anyway and the duplicate is caught server-side at seal.

Local development

uv venv && uv pip install -e ".[dev]"
uv run pytest
uv run ruff check gi_ingest

Design doc: docs/product-plans/vendor-data-ingestion.md.