Skip to main content

git-annex support for Google Cloud Storage

This is a git-annex external special remote that supports Google Cloud Storage (GCS).

Features

  • Supports Standard, Durable Reduced Availability, and Nearline storage classes.

  • Accesses Google Cloud Storage via a Google Cloud service account. Does not use the Google Cloud Storage Interoperability API.

  • Should support object sizes up to 5 TB.

License

GNU General Public License, version 3 or later.

Installing

pip install gcsannex

Configuring Google Cloud Storage

  1. Log into the Google Developers Console.

  2. Create a project if you don’t already have one. Remember its project ID.

  3. In the project’s settings, go to APIs & auth > Credentials.

  4. Click “Create new Client ID”, select “Service account”, then click “Create Client ID”.

  5. A credentials file will be downloaded to your computer. You will need this file when configuring gcsannex.

Adding a remote

git annex initremote <remotename> type=external externaltype=gcs encryption={none|shared|pubkey|hybrid} project=<gcs-project-id>

Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path to your credentials file. This is only necessary when running initremote; afterward, git-annex will remember your credentials.

Mandatory settings

encryption

See the git-annex encryption documentation.

project

Your project ID from the Google developer console.

Optional settings

bucket

The bucket name, which must be globally unique within GCS. The default name is based on the remote name and UUID.

chunk

Enable chunking.

embedcreds

Set to yes to commit the login credentials to the Git repository so other clones can read them. When using GPG encryption, the default is yes and the credentials are stored encrypted. Otherwise, the default is no; if set to yes, anyone with access to the repo can also access the GCS bucket.

fileprefix

A string, such as mydata/, to be prepended to the name of each object. This allows a bucket to be shared by multiple remotes.

location

The physical location for the data. Can be any of the location strings supported by GCS, such as ASIA, EU, or US. Defaults to US.

public

If yes, newly-uploaded objects are made publicly readable. Defaults to no.

readonly

If set to true when enabling an existing public remote, files can be retrieved without GCS credentials and without gcsannex installed. Requires development version of git-annex.

storageclass

A storage class supported by GCS, such as STANDARD, DURABLE_REDUCED_AVAILABILITY, or NEARLINE. Defaults to STANDARD.

Release notes

Version 0.2

  • Time out stuck requests

  • Retry after more types of errors

  • Permit encrypted public remotes

  • Support showing public URLs in git annex whereis

  • Improve compatibility with git-annex < 5.20141219

  • Properly accept implicit encryption=hybrid

Version 0.1

  • Initial release

Release files for gcsannex 0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gcsannex 0.2
File Size Uploaded
gcsannex-0.2.tar.gz 21.6 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for gcsannex 0.2
File Interpreter ABI Platform
gcsannex-0.2-py2-none-any.whl Python 2 none any Details

Total release size: 32.1 kB

Release files / gcsannex-0.2.tar.gz

Download URL gcsannex-0.2.tar.gz
Size 21.6 kB
Tags Source
SHA-256 checksum
How to use checksums
f684511d17ed779275c2543af54bd9f9fa720045a18aa3e02f895f1784796319
BLAKE2b-256 checksum
How to use checksums
9c475c4486557aef7c34406a8639f188b34309e597a52a52339962a7860c1f5d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / gcsannex-0.2-py2-none-any.whl

Download URL gcsannex-0.2-py2-none-any.whl
Size 10.5 kB
Tags Python 2
SHA-256 checksum
How to use checksums
ff0a2c2a4eec4a117182e0170c0427ed7bab5f6139b0e8144f855513f3ecc75f
BLAKE2b-256 checksum
How to use checksums
3acd3fed7aaca1d268721bf8009ca4ad4a562b78f7611bd7064f672a3f8c5743
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

This release

0.2 This release

2 release files

0.1

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page