git-annex special remote for Globus
git-annex-remote-globus adds to git-annex the ability to retrieve files which are available through Globus.
- Note! code is not yet distributed therefore Usage information are going to change soon
Requirements
- Datalad
- Git annex
Setup
-
git clone this repository:
git clone https://github.com/CONP-PCNO/git-annex-remote-globus.git -
In a different folder, install the following dataset:
datalad install https://github.com/conpdatasets/FRDR-multimodal.gitandcd FRDR-multimodal. From now on we are going to work from the dataset repo location where you cd'ed -
Add the path of your git-annex-remote-globus location to your current PATH
-
Initialize a virtual environment and install requirements.txt. You may need to add the git-globus-annex path manually here not to make the system confuse with the filename of other repos: run
pip install -r path/to/git-annex-remote-globus/requirements.txt -
In the repository, run
git-annex-remote-globus setupand follow the instructions to authenticate. Gmail and ORCHID are supported -
Add a remote for globus. Consider the following:
- Add a git-annex remote called
globus - Encrypts can be set to none for now
- The option -d will enable a verbose output
- The endpoint name corresponds to the name of the dataset in globus so globus can find it
- The fileprefix corresponds to the location storing your files in globus
- Add a git-annex remote called
git annex initremote globus type=external externaltype=globus encryption=none endpoint=FRDR_Prod_2 fileprefix=/5/published/publication_170/submitted_data/
To debug git-annex initremote --debug.
Now globus is ready to use ! See the Usage section below
Options
Options specific to git-annex-remote-googledrive
prefix- The path to the folder that will be used for the remoteendpoint- Globus endpoint name where the dataset lives
General git-annex options
encryption- One of "none", "hybrid", "shared", or "pubkey". See encryption. (Must be set tononefor now)mac- The MAC algorithm. See encryption. (Not supported for now)exporttree- Set toyesto make this special remote usable by git-annex-export. It will not be usable as a general-purpose special remote. (Not supported for now)chunk- Enables chunking when storing large files. (Optional)
Using an existing remote (note on repository layout)
If the globus remote was previously initialized, it can be restarted by running git annex enableremote globus. See additional information here.
Choosing a Chunk Size
Choose your chunk size based on your needs. By using a chunk size below the maximum file size supported by your cloud storage provider for uploads and downloads, you won't need to worry about running into issues with file size. Smaller chunk sizes: leak less information about the size of file size of files in your repository, require less ram, and require less data to be re-transmitted when network connectivity is interrupted. Larger chunks require less round trips to and from your cloud provider and may be faster. Additional discussion about chunk size can be found here
Usage
In order to understand how globus remote works, we can work with one file as an example. The file is currently not available to use as only a symlink generated by annex is pointing at it. Run the following to find the file symlink, which includes the file hash
cd FRDR-multimodal
ll 2015_11_18_cortex/mask/mask.mat
The last command will allow you to visualize the symlink which contains the MD6 hash of the file content. At this point, globus does not know anything about this file and its symlink, as you can see running the following command
git annex whereis 2015_11_18_cortex/mask/mask.mat
Globus is not listed indeed. We now need to tell globus of the existence of the file based on its hash, which we call key. When initialized, globus was given a location ID by annex which distributes one to every remote it communicates with. We can find Globus location ID by running the following command from the dataset root location:
cat .git/config
This file shows the remote "globus" which we just initialized, and the globus remote location ID given by annex. Therefore we need to make this location know about the file we want to retrieve.
To do that, run:
git annex setpresentkey <file_hash> <annex-uuid> 1
For example, for the file we are working with 2015_11_18_cortex/mask/mask.mat we would run:
git annex setpresentkey MD5E-s572--1e5e0b0c5896d16ac14170c8f546d4e1.mat 056ae102-61ce-4417-9180-b45eecc45082 1
The 1 at the end tells globus about the existence of this file with its given key. A 0 would remove knowledge of the file
Now, to make sure globus knows about this file, we can run the command below.
git annex whereis 2015_11_18_cortex/mask/mask.mat
At this point we can go ahead and register a url to be associated with the given file key. In this way, we will connect the points and tell annex where in globus the file is located, so it can reach it. Therefore we will add a globus url which will contain the endpoint name and fileprefix:
git annex registerurl MD5E-s572--1e5e0b0c5896d16ac14170c8f546d4e1.mat globus://frdr_prod_2/5/published/publication_170/submitted_data/2015_11_18_cortex/mask/mask.mat
At this point we can finally obtain the file running the command below
git annex get 2015_11_18_cortex/mask/mask.mat
This is the point where the file becomes available on your machine
You can run whereis again to check that
git annex whereis 2015_11_18_cortex/mask/mask.mat
Moreover you can always run the following command if you want to mack sure the file has not been modified in globus compared to your previously downloaded version by using the annex-uuid again
git annex checkpresentkey MD5E-s572--1e5e0b0c5896d16ac14170c8f546d4e1.mat 056ae102-61ce-4417-9180-b45eecc45082
It will return Success if the file in Globus has not change
Issues, Contributing
If you run into any problems, please check for issues on GitHub. Please submit a pull request or create a new issue for problems or potential improvements.
License
Metadata
Release files for git-annex-remote-globus 1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| git-annex-remote-globus-1.0.tar.gz | 11.5 kB | Without provenance | Details for git-annex-remote-globus-1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| git_annex_remote_globus-1.0-py3-none-any.whl | Python 3 | none | any | Details for git_annex_remote_globus-1.0-py3-none-any.whl |
Total release size: 20.9 kB
Release files / git-annex-remote-globus-1.0.tar.gz
| Download URL | git-annex-remote-globus-1.0.tar.gz |
|---|---|
| Size | 11.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c769891ec407e1427202f624185ec7e2d0718d24feec25b2a1474ca518d20e20
|
|
BLAKE2b-256 checksum How to use checksums |
9255c54b407406d0c71c7eef35bc2d6e4d2639a6133e5c465353e72a80ccd6ba
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.6.8
|
Release files / git_annex_remote_globus-1.0-py3-none-any.whl
| Download URL | git_annex_remote_globus-1.0-py3-none-any.whl |
|---|---|
| Size | 9.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3e519c539fc4af314ee63b212111d7be0aa37ea7a9ecf434190cf293c612f44c
|
|
BLAKE2b-256 checksum How to use checksums |
42ba1efc4eb38b1460fcbfa790e58d567af763cc4198869a1dd6d98f70e62221
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.6.8
|
Security for git-annex-remote-globus 1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.