cue-provider-utility 1.3.0

CLI utility for uploading data to CUE (Cloud Upload Environment)

CUE Provider Utility (cue-upload)

Version: 1.3.0
License: Apache-2.0

Overview

The CUE Provider Utility (cue-upload) is a Python-based command-line interface (CLI) designed to facilitate the upload of files and entire folder structures to the CUE (Cloud Upload Environment) backend system. It is optimized for large files, provides a rich user experience, and ensures data integrity through robust error handling and checksum validation.

This tool is intended for users who need to transfer data to the CUE system from their local machines.

Features

• Upload individual files or entire directories recursively.
• Automatic handling of single-part vs. multipart uploads based on file size.
• Concurrent uploads for multiple files within a folder and for parts of a single large file.
• SHA256 checksum validation for data integrity.
• User-friendly configuration via a config.toml file and interactive configure command.
• Flexible authentication token management (CLI option, config file, environment variable, .netrc).
• Support for different backend environments (prod, uat, sit, local).
• Management of ignored file patterns (system defaults + user-defined).
• Detailed logging to both console (with colors and progress bars via Rich) and a log file.
• Cross-platform compatibility (Windows, macOS, Linux).

Prerequisites

• Python: Version 3.10 or newer.
• Poetry: Version 1.2+ is recommended for dependency management. Installation instructions can be found at python-poetry.org.

Installation

Clone the Repository:

git clone https://github.com/ghrcdaac/CUE-Provider-Utility.git
cd CUE-Provider-Utility

Install Dependencies using Poetry:
This command creates a virtual environment and installs all necessary packages.

poetry install

Configuration

The utility is configured via a file located at ~/.cue-upload/config.toml. This file is created automatically the first time you run the tool.

First-Time Setup

The recommended way to set up the CLI is to use the interactive configure command. This will guide you through setting up your API key storage.

cue-upload configure

This will guide you through each option, showing the current value and prompting for a new one. At the end, it will guide you through the API key setup.

Quickly Updating Your API Key

If you only need to update, change, or remove your stored API key without going through all the other settings, you can use the --key flag. This provides a fast and secure way to manage your credentials.

cue-upload configure --key

The prompt will detect if you already have a key stored and ask if you want to overwrite or delete it.

API Key Management

The CLI retrieves your API key from the following sources, in this order of priority:

1. Command-Line Argument: The -t or --token option. (e.g., cue-upload --token YOUR_KEY ...)
2. Environment Variable: The CUE_UPLOAD_API_TOKEN environment variable.
3. .netrc File: A secure, permissions-controlled file in your home directory.
4. config.toml File: The application's configuration file.

The CLI will print which source it is using when you start an upload.

Basic Usage

Activate the Virtual Environment

Before running any commands, activate the Poetry virtual environment:

poetry shell

General Help

cue-upload -h
cue-upload upload -h

Uploading Files or Folders

# Upload a single file
cue-upload upload -P /path/to/your/file.txt -c your_collection_name

# Upload a folder and specify a target sub-path in the collection
cue-upload upload -P /path/to/your/folder/ -c your_collection_name -tp remote/sub_path/

Managing Configuration

# Run the interactive configuration wizard
cue-upload configure

Retrying Failed Uploads

When upload attempts fail (due to network glitches or server availability), details are saved to ~/.cue-upload/failed_uploads.json. You can retry all failed uploads anytime:

# Retry all pending failed uploads from the default log file
cue-upload retry

# Retry failed uploads from a custom log file
cue-upload retry -f /path/to/custom_failed_uploads.json

Failed Uploads Log Schema (failed_uploads.json)

The failed uploads log uses a simple JSON structure so you can inspect or manually add entries if needed:

{
  "version": "1.0",
  "failed_uploads": [
    {
      "entry_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "session_id": "f81d4fae-7dec-11d0-a765-00a0c91e6bf6",
      "file_path": "C:/data/granule_01.nc",
      "collection": "MY_COLLECTION",
      "target_sub_path": "subfolder/2026",
      "env": "prod",
      "file_size": 104857600,
      "content_type": "application/x-netcdf",
      "checksum": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "file_concurrency": 4,
      "part_concurrency": 4,
      "error_message": "HTTP 503 Service Unavailable",
      "timestamp": "2026-07-21T13:37:19Z"
    }
  ]
}

Note: When creating manual entries, entry_id, session_id, checksum, content_type, and timestamp can be left blank or null.

Viewing Logs

# View the entire log file
cue-upload logs

# View the last 100 lines
cue-upload logs -n 100

Managing Ignored File Patterns

# List all current ignore patterns
cue-upload ignore list

# Add a pattern to ignore all .log files
cue-upload ignore add "*.log"

# Remove a pattern
cue-upload ignore remove "*.log"

# Clear all user-defined patterns
cue-upload ignore reset

Global Options

• -t, --token YOUR_TOKEN: Provide API key directly.
• --env ENV_NAME: Target a specific backend environment (prod, uat, sit, local).
• -v / -vv: Increase console verbosity. -vv includes full network logs.
• -q: Quiet mode, showing only critical errors.
• --config /path/to/config.toml: Use a custom configuration file.

Development

Running Tests

Tests are located in the tests/ directory and can be run with pytest.

# Run all tests
pytest

# Run tests with coverage report
pytest --cov=cue_provider_utility tests/

Linters and Formatters

This project uses black, isort, flake8, and mypy to maintain code quality.

black cue_provider_utility/ tests/
isort cue_provider_utility/ tests/
flake8 cue_provider_utility/ tests/
mypy cue_provider_utility/

License

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for details.