Skip to main content

git-annex special remote for Globus

git-annex-remote-globus adds to git-annex the ability to retrieve files which are available through Globus.

  • Note! code is not yet distributed therefore Usage information are going to change soon

Requirements

  • Datalad
  • Git annex

Setup

  1. git clone this repository: git clone https://github.com/CONP-PCNO/git-annex-remote-globus.git

  2. In a different folder, install the following dataset: datalad install https://github.com/conpdatasets/FRDR-multimodal.gitand cd FRDR-multimodal. From now on we are going to work from the dataset repo location where you cd'ed

  3. Add the path of your git-annex-remote-globus location to your current PATH

  4. Initialize a virtual environment and install requirements.txt. You may need to add the git-globus-annex path manually here not to make the system confuse with the filename of other repos: run pip install -r path/to/git-annex-remote-globus/requirements.txt

  5. In the repository, run git-annex-remote-globus setup and follow the instructions to authenticate. Gmail and ORCHID are supported

  6. Add a remote for globus. Consider the following:

    • Add a git-annex remote called globus
    • Encrypts can be set to none for now
    • The option -d will enable a verbose output
    • The endpoint name corresponds to the name of the dataset in globus so globus can find it
    • The fileprefix corresponds to the location storing your files in globus
git annex initremote globus type=external externaltype=globus encryption=none endpoint=FRDR_Prod_2 fileprefix=/5/published/publication_170/submitted_data/

To debug git-annex initremote --debug.

Now globus is ready to use ! See the Usage section below

Options

Options specific to git-annex-remote-googledrive

  • prefix - The path to the folder that will be used for the remote
  • endpoint - Globus endpoint name where the dataset lives

General git-annex options

  • encryption - One of "none", "hybrid", "shared", or "pubkey". See encryption. (Must be set to none for now)
  • mac - The MAC algorithm. See encryption. (Not supported for now)
  • exporttree - Set to yes to make this special remote usable by git-annex-export. It will not be usable as a general-purpose special remote. (Not supported for now)
  • chunk - Enables chunking when storing large files. (Optional)

Using an existing remote (note on repository layout)

If the globus remote was previously initialized, it can be restarted by running git annex enableremote globus. See additional information here.

Choosing a Chunk Size

Choose your chunk size based on your needs. By using a chunk size below the maximum file size supported by your cloud storage provider for uploads and downloads, you won't need to worry about running into issues with file size. Smaller chunk sizes: leak less information about the size of file size of files in your repository, require less ram, and require less data to be re-transmitted when network connectivity is interrupted. Larger chunks require less round trips to and from your cloud provider and may be faster. Additional discussion about chunk size can be found here

Usage

In order to understand how globus remote works, we can work with one file as an example. The file is currently not available to use as only a symlink generated by annex is pointing at it. Run the following to find the file symlink, which includes the file hash

cd FRDR-multimodal
ll 2015_11_18_cortex/mask/mask.mat

The last command will allow you to visualize the symlink which contains the MD6 hash of the file content. At this point, globus does not know anything about this file and its symlink, as you can see running the following command

git annex whereis 2015_11_18_cortex/mask/mask.mat

Globus is not listed indeed. We now need to tell globus of the existence of the file based on its hash, which we call key. When initialized, globus was given a location ID by annex which distributes one to every remote it communicates with. We can find Globus location ID by running the following command from the dataset root location:

cat .git/config

This file shows the remote "globus" which we just initialized, and the globus remote location ID given by annex. Therefore we need to make this location know about the file we want to retrieve.

To do that, run:

git annex setpresentkey <file_hash> <annex-uuid> 1

For example, for the file we are working with 2015_11_18_cortex/mask/mask.mat we would run:

git annex setpresentkey MD5E-s572--1e5e0b0c5896d16ac14170c8f546d4e1.mat 056ae102-61ce-4417-9180-b45eecc45082 1

The 1 at the end tells globus about the existence of this file with its given key. A 0 would remove knowledge of the file

Now, to make sure globus knows about this file, we can run the command below.

git annex whereis 2015_11_18_cortex/mask/mask.mat

At this point we can go ahead and register a url to be associated with the given file key. In this way, we will connect the points and tell annex where in globus the file is located, so it can reach it. Therefore we will add a globus url which will contain the endpoint name and fileprefix:

git annex registerurl MD5E-s572--1e5e0b0c5896d16ac14170c8f546d4e1.mat globus://frdr_prod_2/5/published/publication_170/submitted_data/2015_11_18_cortex/mask/mask.mat

At this point we can finally obtain the file running the command below

git annex get 2015_11_18_cortex/mask/mask.mat

This is the point where the file becomes available on your machine

You can run whereis again to check that

git annex whereis 2015_11_18_cortex/mask/mask.mat

Moreover you can always run the following command if you want to mack sure the file has not been modified in globus compared to your previously downloaded version by using the annex-uuid again

git annex checkpresentkey MD5E-s572--1e5e0b0c5896d16ac14170c8f546d4e1.mat 056ae102-61ce-4417-9180-b45eecc45082

It will return Success if the file in Globus has not change

Issues, Contributing

If you run into any problems, please check for issues on GitHub. Please submit a pull request or create a new issue for problems or potential improvements.

License

Metadata

Release files for git-annex-remote-globus 1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for git-annex-remote-globus 1.0
File Size Uploaded Provenance Actions
git-annex-remote-globus-1.0.tar.gz 11.5 kB Without provenance Details for git-annex-remote-globus-1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for git-annex-remote-globus 1.0
File Interpreter ABI Platform Actions
git_annex_remote_globus-1.0-py3-none-any.whl Python 3 none any Details for git_annex_remote_globus-1.0-py3-none-any.whl

Total release size: 20.9 kB

Release files / git-annex-remote-globus-1.0.tar.gz

Download URL git-annex-remote-globus-1.0.tar.gz
Size 11.5 kB
Tags Source
SHA-256 checksum
How to use checksums
c769891ec407e1427202f624185ec7e2d0718d24feec25b2a1474ca518d20e20
BLAKE2b-256 checksum
How to use checksums
9255c54b407406d0c71c7eef35bc2d6e4d2639a6133e5c465353e72a80ccd6ba
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.6.8

Release files / git_annex_remote_globus-1.0-py3-none-any.whl

Download URL git_annex_remote_globus-1.0-py3-none-any.whl
Size 9.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3e519c539fc4af314ee63b212111d7be0aa37ea7a9ecf434190cf293c612f44c
BLAKE2b-256 checksum
How to use checksums
42ba1efc4eb38b1460fcbfa790e58d567af763cc4198869a1dd6d98f70e62221
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.6.8

Security for git-annex-remote-globus 1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page