gi-ingest

See CHANGELOG.md for release notes and unreleased changes.

Upload robotics recordings from vendor-managed directories and reclaim only recordings whose delivery has been confirmed by GILabs.

For a native Windows UI with verified SD-card-to-SSD staging, persistent caching, labels, and uploads, see GI Ingest for Windows.

With the CLI, vendors copy SD cards to SSDs or local storage themselves. gi-ingest uploads those files in place: there is no stage command and no staging disk to configure. Keep each recording's directory, part filenames, and metadata sidecar together.

Install and authenticate

macOS and Linux, Python 3.11 or newer:

uv tool install gi-ingest
gi-ingest login --api-key gik_...
gi-ingest doctor

Get an upload token from the vendor portal at vendor.gilabs.xyz. Select a destination when your vendor has more than one:

gi-ingest destinations --select

Upload from any number of paths

Pass a recording directory, a copied card directory, or a parent containing recordings. Paths may span multiple SSDs and the computer's local disk:

gi-ingest upload /Volumes/SSD1/recordings /Volumes/SSD2/recordings ~/recordings \
  --jobs 4 --max-bandwidth 50M

Discovery is recursive. Overlapping input paths are deduplicated. Symlink paths and recordings containing symlinks are refused. Missing or disconnected input directories are reported rather than silently omitted.

--jobs controls concurrent recording inspection and uploads (default 4, range 1–32). --max-bandwidth is an aggregate limit across the process, in bytes per second; 50M means 50 MiB/s. Stop recorder activity and finish copying before uploading. Keep source disks connected until the command finishes.

--accelerate auto (the default) measures standard and accelerated S3 endpoints and uses acceleration when it is at least 1.15× faster. Successful comparisons are cached for three days across deliveries; the endpoint message shows the cached result's age. A changed public egress IP, bucket/region, or bandwidth limit triggers a new comparison. The CLI checks the public IP through AWS's checkip.amazonaws.com; if that lookup fails, it probes without using the cache. Failed comparisons are not cached. Cache storage follows XDG_CACHE_HOME, defaulting to ~/.cache/gi-ingest/acceleration.json.

Use --refresh-acceleration to ignore the cache and measure again, or --accelerate always|never to force an endpoint without probing or reading the cache. Forced modes take precedence over --refresh-acceleration.

gi-ingest upload /Volumes/SSD1/recordings --project <project-id>
gi-ingest upload /Volumes/SSD1/recordings --dry-run

Recordings with bad or unset clocks require an individual collection date. An interactive upload shows each affected recording's full path, device, and reported timestamp, then asks for its date (YYYY-MM-DD). Type skip to leave that recording unresolved. No answer is shared across recordings, even on the same SSD or with the same reported timestamp. Healthy timestamps remain unchanged.

Unattended uploads hold these recordings, save their paths and content identities in the ledger, and return nonzero. Run gi-ingest attribute from a terminal with the SSD mounted to resolve each date, followed by any missing operator/environment labels. Corrections are retained per recording for retries. If files move, run upload with their new paths first. Reclaim keeps unresolved recordings. There is no upload-wide --collection-date option. Recordings with a confirmed missing IMU are held; --allow-missing-imu explicitly permits them. Short clips are uploaded for the platform to evaluate.

Operator and environment attribution is required. Interactive upload shows a numbered inventory with full paths, device, date, and existing labels. It offers:

1. Select recordings and label a group (1,3-5 or all).
2. Label recordings individually.
3. Continue with ready recordings and leave the rest for attribute.

The inventory uses natural full-path order (session_2 before session_10). Numbers stay fixed throughout the labeling session. No grouping is inferred from paths, devices, or dates. A review and confirmation precede every group write. Only missing fields are filled; conflicting existing labels exclude a recording from that group and are never overwritten. Sidecars are updated in place.

To label before uploading, supply paths directly to attribute. It scans recursively and saves source registrations without uploading. With paths it only labels that selection; without paths it uses the saved queue:

gi-ingest attribute /Volumes/SSD1/data /Volumes/SSD2/data
gi-ingest upload /Volumes/SSD1/data /Volumes/SSD2/data

upload --no-prompt holds missing dates and labels for later attribution even in an interactive terminal. Ready recordings still upload, and any held recordings cause a nonzero exit code. Unattended upload uses the same behavior:

gi-ingest upload /Volumes/SSD1/recordings --no-prompt
gi-ingest attribute                   # select groups and resolve held dates
gi-ingest upload /Volumes/SSD1/recordings

Plain MCAP recordings can recover a missing session.json from embedded metadata. The recovered sidecar is created alongside the recording; an existing unreadable sidecar must be repaired. Recording payloads are never rewritten.

A dry run reads and hashes the selected recordings using a temporary copy of ledger state. It does not upload, change source files, or update the saved ledger. Missing labels are checked against recovered metadata in memory too. If a preview finds missing dates or labels, rerun the same upload command without --dry-run from a terminal to answer the prompts. Running attribute immediately after a dry run cannot resolve those findings: the preview has not saved them to the queue.

Interruptions and retries

During a running upload, transient connection failures in file/part uploads and the resume listing are retried with backoff for up to 15 minutes per request, starting at its first connection failure. This includes TLS connections cut off mid-write. Multipart retries keep chunks already sent in that running transfer. The log shows the retry delay and remaining window. Certificate and permission errors do not receive the extended retry window.

On an unstable connection, reduce simultaneous recording uploads with --jobs 1 (or set upload concurrency to 1 in the desktop app). Each recording still sends up to eight chunks concurrently. If retries are exhausted, retry the same delivery with the same cache/ledger; the desktop app preserves delivery membership when you retry its pending queue.

Files of 16 MiB or larger now retain multipart upload checkpoints across failures and app/process restarts. On retry, the CLI reads the source again, verifies the size and SHA-256 of chunks already in S3, and sends only missing or mismatched chunks. Reused chunks count toward progress without inflating transfer speed. Completed files are also reused. Keep the original source files and ledger (the desktop app uses delivery.db in its cache); no new command or option is needed.

This applies to uploads started with a version supporting multipart checkpoints. Older uploads without checkpoints, partial uploads removed by S3 lifecycle cleanup, and uploads whose completion succeeded but could not be confirmed locally restart the affected file conservatively. A partial upload does not confirm a delivery or make its source eligible for reclaim.

Run the same upload command again after a crash or network failure. An open delivery is reused. Objects with matching sizes and completed-file checkpoints are skipped; uncertain transfers are sent again. You can also resume the local queue without supplying paths:

gi-ingest upload

If a recording moves to another disk, supply its new path with its original session directory name. Its content identity retains its upload state. To resume an open delivery with explicit paths, include all of that delivery's recordings. The CLI refuses to seal only part of an existing delivery.

A failed transfer leaves the delivery open and returns a nonzero exit code. Recordings held for missing metadata or attribution also make the command return nonzero, even when other recordings were delivered successfully. Read the named failures; do not treat a partial delivery as permission to delete the whole input. Changed files in an open delivery must be restored, or the delivery explicitly abandoned before starting again:

gi-ingest abandon --batch <batch-id>
gi-ingest upload /Volumes/SSD1/recordings

Only one mutating CLI operation may use a ledger at a time. Use --jobs for native concurrency, rather than multiple processes sharing a ledger.

Reclaim only the paths you choose

Keep at least one verified local copy until delivery is confirmed. reclaim is the supported way to enforce that rule. A transfer completing is not enough: the server must have accepted the sealed delivery.

gi-ingest reclaim /Volumes/SSD1/recordings --dry-run
gi-ingest reclaim /Volumes/SSD1/recordings

Reclaim searches only the supplied paths. It compares each recording's part inventory, sizes, SHA-256 hashes, and device metadata against a confirmed entry in the local ledger before deleting its session directory. A second copy, such as the original SD card, can be reclaimed by supplying that copy's path too. Physical card identity or a particular mountpoint is not required.

Unconfirmed recordings, changed contents, missing proof, and directories with unknown files are kept. There is no option to wipe undelivered or unusable recordings through this command. Reclaim never purges another disk or every saved source path as a side effect. There is no default scan of mounted drives.

Confirmation means the server accepted the raw delivery, not that downstream processing or quality review has finished. The current server seal verifies object presence and sizes; the CLI records SHA-256 values and uses them for local content matching. Manually deleting the last local copy before confirmation bypasses these safeguards and can lose data.

Status and local state

gi-ingest status
gi-ingest history
gi-ingest history --batch <batch-id>
gi-ingest retry
gi-ingest config show

Authentication and defaults live in ~/.config/gi-ingest/config.json; upload and delivery records live in ~/.local/state/gi-ingest/ledger.db. Keep the ledger: without a matching confirmation record, reclaim preserves the files. Use --ledger PATH on upload and reclaim to use another ledger, and use the same ledger for both commands.

Existing ledgers and staged files remain usable. Upload their directories directly, or run upload without paths to resume the old queue. staged remains an internal state meaning ready to upload; it no longer implies a separate copy step. The old staging-dir setting is unused by upload and reclaim.

reset and drop preserve files registered by path-based upload. Their legacy cleanup behavior still applies to CLI-owned staging copies from older versions; inspect their dry-run output before changing old queue entries.

Encrypted recordings

VREC recordings retain their ego_*.mcap names and use the same gi-ingest upload <paths...> workflow. Upload sends ciphertext unchanged and reports that decryption and content checks run on the backend. No recording key is needed locally. The ledger persists source_encrypted; older entries remain unknown until rescanned.

If session.json is missing, the CLI warns and uploads the encrypted parts with metadata recovery pending. DPS decrypts the MCAP, reconstructs the sidecar from embedded capture metadata, and writes both to the episodes bucket. No placeholder sidecar is created locally. An existing unreadable sidecar still requires repair.

Resume, dry-run, and scoped reclaim use the original ciphertext hashes.

Development

uv sync --extra dev
uv run pytest -q
uv run ruff check gi_ingest tests

Upload progress and ETA

gi-ingest upload displays aggregate byte progress, transfer speed, remaining upload time, and completed/total files across all concurrent recording uploads. Resumed objects count toward completion but do not inflate transfer speed. ETA becomes unknown when transfers stall or S3 is still finalizing a file. Redirected output (including nohup) gets a progress line about every 10 seconds and a final summary.

The CLI sends a best-effort snapshot about every 10 seconds to PUT /api/v1/ingest/batches/{batch_id}/progress, plus a final uploaded, failed, or interrupted snapshot. Reporting uses a background thread and short request timeouts; an unavailable or older backend does not stop the upload. A process killed without cleanup leaves its last heartbeat behind.

Batch responses expose upload_progress: total_bytes, uploaded_bytes (includes in-flight and resumed bytes), resumed_bytes, total_files, completed_files, bytes_per_second, eta_seconds, phase, and the server's received_at. Use the timestamp to detect stale reports. These values are client-reported and independent of manifest-verified batch totals and lifecycle status; uploaded means ready for sealing, not delivered. On resume the latest snapshot replaces the previous attempt's progress. Multiple processes uploading the same batch are not aggregated; the latest received snapshot wins.

Apply the add_ingest_upload_progress database migration before deploying the backend. Existing clients continue to work and leave upload_progress null.

Troubleshooting upload connections

Uploads automatically record structured diagnostics in upload-diagnostics/ beside the ledger. Each attempt logs a run ID and records stages, request failure categories, executed retries, recovered requests, progress, and reused multipart chunks. Reports also include standard and accelerated endpoint probe speeds in bytes per second, whether the reading was fresh or cached, and its cache age. Missing readings stay null. Current delivery speed is reported separately in progress.bytes_per_second. The shared Windows engine records the same data.

Reports are sent automatically to the backend for platform engineering analysis; vendors do not need to upload them manually. They are stored in the ingest_batches.upload_diagnostics JSONB column, with up to ten runs per delivery. There is no portal diagnostics UI or read endpoint. See the engineering query guide. Run timestamps come from the vendor's clock; received_at is the server's clock. A stale running report does not prove the client is still running. A forced process kill can lose events since the previous heartbeat (normally 30 seconds).

Reports are saved locally before background delivery. A network outage retains unsent reports, and subsequent heartbeats or upload attempts replay them using the same server and upload-token login. Only 20 runs per login are kept locally; very old reports can be evicted, including unsent ones. Switching tokens/server isolates the outbox. Reports from failures before a batch ID exists stay local. Validation and version checks before an upload attempt are not instrumented.

For optional support investigation while a machine remains offline, export to a new filename:

gi-ingest diagnostics --output upload-diagnostics.json
# Include --ledger /path/to/ledger.db if uploads use a custom ledger.

This JSON contains batch/run IDs and structured statistics, not raw logs, recording contents, paths, API keys, S3 credentials, request URLs/headers, IP addresses, or Wi-Fi names. The file is safe for your support team to inspect; share it through your normal support channel. No third-party telemetry service receives these reports. DNS/connect, timeout, TLS, and server error categories help narrow the investigation but cannot conclusively distinguish bad Wi-Fi from an ISP or upstream network problem.

Server rollout: apply the vendor_upload_diagnostics migration, deploy the backend, then release the updated CLI/desktop app. With an older backend, the CLI keeps local reports and continues uploading normally. Diagnostic writes are advisory, vendor-scoped, and accepted after sealing so delayed reports are not discarded. They never change upload permissions or verified delivery totals.