gi-ingest

Upload robotics recordings from vendor-managed directories and reclaim only recordings whose delivery has been confirmed by GILabs.

Vendors copy SD cards to SSDs or local storage themselves. gi-ingest uploads those files in place: there is no stage command and no staging disk to configure. Keep each recording's directory, part filenames, and metadata sidecar together.

Install and authenticate

macOS and Linux, Python 3.11 or newer:

uv tool install gi-ingest
gi-ingest login --api-key gik_...
gi-ingest doctor

Get an upload token from the vendor portal at vendor.gilabs.xyz. Select a destination when your vendor has more than one:

gi-ingest destinations --select

Upload from any number of paths

Pass a recording directory, a copied card directory, or a parent containing recordings. Paths may span multiple SSDs and the computer's local disk:

gi-ingest upload /Volumes/SSD1/recordings /Volumes/SSD2/recordings ~/recordings \
  --jobs 4 --max-bandwidth 50M

Discovery is recursive. Overlapping input paths are deduplicated. Symlink paths and recordings containing symlinks are refused. Missing or disconnected input directories are reported rather than silently omitted.

--jobs controls concurrent recording inspection and uploads (default 4, range 1–32). --max-bandwidth is an aggregate limit across the process, in bytes per second; 50M means 50 MiB/s. Stop recorder activity and finish copying before uploading. Keep source disks connected until the command finishes.

gi-ingest upload /Volumes/SSD1/recordings --project <project-id>
gi-ingest upload /Volumes/SSD1/recordings --dry-run

Recordings with bad or unset clocks require an individual collection date. An interactive upload shows each affected recording's full path, device, and reported timestamp, then asks for its date (YYYY-MM-DD). Type skip to leave that recording unresolved. No answer is shared across recordings, even on the same SSD or with the same reported timestamp. Healthy timestamps remain unchanged.

Unattended uploads hold these recordings, save their paths and content identities in the ledger, and return nonzero. Run gi-ingest attribute from a terminal with the SSD mounted to resolve each date, followed by any missing operator/environment labels. Corrections are retained per recording for retries. If files move, run upload with their new paths first. Reclaim keeps unresolved recordings. There is no upload-wide --collection-date option. Recordings with a confirmed missing IMU are held; --allow-missing-imu explicitly permits them. Short clips are uploaded for the platform to evaluate.

Operator and environment attribution is required. Interactive upload shows a numbered inventory with full paths, device, date, and existing labels. It offers:

1. Select recordings and label a group (1,3-5 or all).
2. Label recordings individually.
3. Continue with ready recordings and leave the rest for attribute.

The inventory uses natural full-path order (session_2 before session_10). Numbers stay fixed throughout the labeling session. No grouping is inferred from paths, devices, or dates. A review and confirmation precede every group write. Only missing fields are filled; conflicting existing labels exclude a recording from that group and are never overwritten. Sidecars are updated in place.

To label before uploading, supply paths directly to attribute. It scans recursively and saves source registrations without uploading. With paths it only labels that selection; without paths it uses the saved queue:

gi-ingest attribute /Volumes/SSD1/data /Volumes/SSD2/data
gi-ingest upload /Volumes/SSD1/data /Volumes/SSD2/data

upload --no-prompt holds missing dates and labels for later attribution even in an interactive terminal. Ready recordings still upload, and any held recordings cause a nonzero exit code. Unattended upload uses the same behavior:

gi-ingest upload /Volumes/SSD1/recordings --no-prompt
gi-ingest attribute                   # select groups and resolve held dates
gi-ingest upload /Volumes/SSD1/recordings

Plain MCAP recordings can recover a missing session.json from embedded metadata. The recovered sidecar is created alongside the recording; an existing unreadable sidecar must be repaired. Recording payloads are never rewritten.

A dry run reads and hashes the selected recordings using a temporary copy of ledger state. It does not upload, change source files, or update the saved ledger. Missing labels are checked against recovered metadata in memory too. If a preview finds missing dates or labels, rerun the same upload command without --dry-run from a terminal to answer the prompts. Running attribute immediately after a dry run cannot resolve those findings: the preview has not saved them to the queue.

Interruptions and retries

Run the same upload command again after a crash or network failure. An open delivery is reused. Objects with matching sizes and completed-file checkpoints are skipped; uncertain transfers are sent again. You can also resume the local queue without supplying paths:

gi-ingest upload

If a recording moves to another disk, supply its new path with its original session directory name. Its content identity retains its upload state. To resume an open delivery with explicit paths, include all of that delivery's recordings. The CLI refuses to seal only part of an existing delivery.

A failed transfer leaves the delivery open and returns a nonzero exit code. Recordings held for missing metadata or attribution also make the command return nonzero, even when other recordings were delivered successfully. Read the named failures; do not treat a partial delivery as permission to delete the whole input. Changed files in an open delivery must be restored, or the delivery explicitly abandoned before starting again:

gi-ingest abandon --batch <batch-id>
gi-ingest upload /Volumes/SSD1/recordings

Only one mutating CLI operation may use a ledger at a time. Use --jobs for native concurrency, rather than multiple processes sharing a ledger.

Reclaim only the paths you choose

Keep at least one verified local copy until delivery is confirmed. reclaim is the supported way to enforce that rule. A transfer completing is not enough: the server must have accepted the sealed delivery.

gi-ingest reclaim /Volumes/SSD1/recordings --dry-run
gi-ingest reclaim /Volumes/SSD1/recordings

Reclaim searches only the supplied paths. It compares each recording's part inventory, sizes, SHA-256 hashes, and device metadata against a confirmed entry in the local ledger before deleting its session directory. A second copy, such as the original SD card, can be reclaimed by supplying that copy's path too. Physical card identity or a particular mountpoint is not required.

Unconfirmed recordings, changed contents, missing proof, and directories with unknown files are kept. There is no option to wipe undelivered or unusable recordings through this command. Reclaim never purges another disk or every saved source path as a side effect. There is no default scan of mounted drives.

Confirmation means the server accepted the raw delivery, not that downstream processing or quality review has finished. The current server seal verifies object presence and sizes; the CLI records SHA-256 values and uses them for local content matching. Manually deleting the last local copy before confirmation bypasses these safeguards and can lose data.

Status and local state

gi-ingest status
gi-ingest history
gi-ingest history --batch <batch-id>
gi-ingest retry
gi-ingest config show

Authentication and defaults live in ~/.config/gi-ingest/config.json; upload and delivery records live in ~/.local/state/gi-ingest/ledger.db. Keep the ledger: without a matching confirmation record, reclaim preserves the files. Use --ledger PATH on upload and reclaim to use another ledger, and use the same ledger for both commands.

Existing ledgers and staged files remain usable. Upload their directories directly, or run upload without paths to resume the old queue. staged remains an internal state meaning ready to upload; it no longer implies a separate copy step. The old staging-dir setting is unused by upload and reclaim.

reset and drop preserve files registered by path-based upload. Their legacy cleanup behavior still applies to CLI-owned staging copies from older versions; inspect their dry-run output before changing old queue entries.

Encrypted recordings

VREC-encrypted recordings retain the recorder's ego_*.mcap filenames. Upload preserves ciphertext, including sizes and SHA-256 values in the manifest. Vendors do not need a recording key. The backend handles decryption and IMU inspection; accepting ciphertext does not certify that it contains IMU data.

Keep session.json beside encrypted parts. Without it, the CLI cannot recover metadata from ciphertext and holds the recording until the sidecar is restored.

Development

uv sync --extra dev
uv run pytest -q
uv run ruff check gi_ingest tests