Release Tools

Note: These tools are being used for my personal projects and are not meant for general public use.

This project provides a set of tools to manage publishing a large number of files using GitHub releases, particularly to overcome the 1000 file limit per release by creating supplementary releases. This assumes that gh (the GitHub CLI) is installed and configured on your system, as it is used to interact with GitHub releases.

STAC index preparation and publication are maintained in Indian Open Maps, through iomaps admin stac and iomaps admin make-release.

Features

• Generate file lists: Create a CSV file containing a list of all files in a GitHub release.
• Upload files to a release: Upload files from a local directory to a GitHub release, with an option to overwrite existing files.
• Delete files from a release: Delete specified files from a release and its supplementary releases.

Installation

This project is packaged using Hatch. To install the tools, you can run the following command:

pip install .

This will install the gh_release_tools package and make the following command-line tools available:

• generate-lists
• upload-to-release
• delete-from-release
• download-from-release

Usage

It is recommended to run the tools using uvx to avoid dependency conflicts with other installed packages.

uvx

uvx --from gh_release_tools generate-lists --release <tag> --extension <extension>
uvx --from gh_release_tools upload-to-release --release <tag> --folder <folder> --extension <extension> [--overwrite] [--create-extra-releases] [--batch-size <size>]
uvx --from gh_release_tools download-from-release --release <tag> [--file-list <file>] [--output-dir <dir>] [--skip-existing] [--batch-size <size>] [file1 file2 ...]

generate-lists

This tool generates a CSV file named listing_files.csv containing a list of assets from a specified GitHub release and its associated -extra releases. The CSV file includes the file name, size, and download URL for each asset. The generated CSV is then uploaded to the base release, overwriting any existing listing_files.csv.

Usage:

generate-lists --release <tag> --extension <ext1> [--extension <ext2> ...] [--repo <owner/repo>]

Arguments:

• --release, -r: The base name of the GitHub release (e.g., v1.0.0).
• --extension, -e: The file extension of the assets to include in the list (e.g., .zip). This argument can be specified multiple times for different extensions.
• --repo, -g: (Optional) The GitHub repository in the format owner/repo. If not provided, it will be inferred from the current directory.

Example:

generate-lists --release v1.0.0 --extension .zip --extension .txt

This command will generate a listing_files.csv file with a list of all .zip and .txt files from the v1.0.0 release and any releases named v1.0.0-extra<number>.

upload-to-release

This tool uploads files from a local folder to a GitHub release. It can be configured to skip files that already exist in the release or to overwrite them. It can also create new supplementary releases if the target release is full.

Usage:

upload-to-release --release <tag> --folder <folder> --extension <ext1> [--extension <ext2> ...] [--overwrite] [--create-extra-releases] [--batch-size <size>] [--repo <owner/repo>]

Arguments:

• --release, -r: The git tag of the release to upload to.
• --folder, -d: The local folder containing the files to upload.
• --extension, -e: The file extension of the files to upload (e.g., .zip). This argument can be specified multiple times.
• --overwrite: (Optional) Overwrite existing assets in the release.
• --create-extra-releases, -x: (Optional) Create supplementary releases if the main release is full.
• --batch-size, -b: (Optional) The number of files to upload in a single batch. (default: 1)
• --repo, -g: (Optional) The GitHub repository in the format owner/repo. If not provided, it will be inferred from the current directory.

Example:

upload-to-release --release v1.0.0 --folder /path/to/files --extension .zip

This command will upload all .zip files from the /path/to/files directory to the v1.0.0 release. If a file with the same name already exists in the release, it will be skipped. To overwrite existing files, use the following command:

upload-to-release --release v1.0.0 --folder /path/to/files --extension .zip --overwrite

delete-from-release

This tool deletes specified files from a GitHub release and its associated -extra releases. You can provide a list of files to delete either as command-line arguments or in a separate text file.

Usage:

delete-from-release --release <tag> [--file-list <file>] [--repo <owner/repo>] [file1 file2 ...]

Arguments:

• --release, -r: The base name of the GitHub release (e.g., v1.0.0).
• --file-list, -f: (Optional) Path to a text file with a list of files to delete (one file per line).
• files: (Optional) Space-separated list of file names to delete.
• --repo, -g: (Optional) The GitHub repository in the format owner/repo. If not provided, it will be inferred from the current directory.

Example:

delete-from-release --release v1.0.0 --file-list files_to_delete.txt my_file.zip

This command will delete the files listed in files_to_delete.txt and my_file.zip from the v1.0.0 release and any releases named v1.0.0-extra<number>.

download-from-release

This tool downloads specified files from a GitHub release and its associated -extra releases.

Usage:

download-from-release --release <tag> [--file-list <file>] [--output-dir <dir>] [--skip-existing] [--batch-size <size>] [--repo <owner/repo>] [file1 file2 ...]

Arguments:

• --release, -r: The base name of the GitHub release (e.g., v1.0.0).
• --file-list, -f: (Optional) Path to a text file with a list of files to download (one file per line).
• --output-dir, -d: (Optional) The directory to save the downloaded files. If not provided, files are downloaded to the current directory.
• --skip-existing: (Optional) Skip downloading files that already exist in the output directory.
• --batch-size, -b: (Optional) The number of files to download in a single batch. (default: 1)
• files: (Optional) Space-separated list of file names to download.
• --repo, -g: (Optional) The GitHub repository in the format owner/repo. If not provided, it will be inferred from the current directory.

Example:

download-from-release --release v1.0.0 --file-list files_to_download.txt --output-dir /path/to/downloads

This command will download the files listed in files_to_download.txt from the v1.0.0 release and any releases named v1.0.0-extra<number> into the /path/to/downloads directory.

download-from-release --release v1.0.0 my_file.zip --skip-existing

This command will download my_file.zip to the current directory, but only if it doesn't already exist.