A command called globus and foo for the cloudmesh shell
Project description
Cloudmesh Globus
This program is a better globus command line client that allows to transfer millions of files and automatically conduct the transfer through tar files if desired.
We observed this program is needed as in the endpoints that I use and accesses a Globus server the service gets interrupted but does not fail. Hence, the transfer never finishes. With this commandline tool and does not contthe issue could be solved.
Please take a look at our example.
Installation
Mac/Linux
python3.10 -m venv ~/GLOBUS
source ~/GLOBUS/bin/activate
pip install cloudmesh-globus
cms help
cms globus help
Calling the command cms help directly after the pip install is very important as it sets up some directories and files
Windows
This will work on windows, but the creation of the venv is different. let us know how you do it as I do not have a Windows machine available.
Usage
globus login FROM_EP TO_EP
globus transfer FROM_EP TO_EP DIR FROM TO [--mkdir=no] [--tar] [--force] [--dryrun]
globus transfer FROM_EP TO_EP DIR --file=FILE TO [--mkdir=no] [--run=1:100] [--tar] [--force] [--dryrun]
Arguments:
FROM_EP the globus endpoint to transfer the file from
TO_EP the globus endpoint to transfer the file to
DIR the base directory from which to transfer. All transfers
are done relative to this DIR and the new remote directory
will not have the prefix DIR.
FROM a relative directory to transfer files from
TO the directory to transfer the relative directory FROM
FILE a file that contains files or directories to transfer
Options:
--file=FILE specify the file that contains the transfers
--tar transfer the data in a tar file
--mkdir=no assume the remote directory to place the file in is
already created This can speed up the transfer in
case you have many transfers to execute. So create
the file tree first. It also helps in case a transfer
failed and you need to redo it, but the directories
are already created.
--force in case force is not specified and the tar file already
exists on the remote, the transfer will be not conducted.
With force the transfer will be done.
--run=1:10 Only execute the transfers in the lines 1 to 10
Description:
globus transfer FROM_EP TO_EP DIR FROM TO [--mkdir=no] [--tar]
This command is a convenience wrapper for the globus command
allowing the transfer of relative directories towards a source
directory. The program creates on the destination the same
directory tree as on the source starting from the base directory
specified with DIR. The FROM directory locates at DIR/FROM is
then copied to TO without the prefix DIR.
If you specify the --mkdir=no the destination directory is assumed to
exist and is not explicitly created. If it does not exist the program
will be terminated.
bookmarks
bookmarks can be stored in a file called
~/.cloudmesh/globus.yaml
let us assume the file contains
bookmark:
uva: c4d80096-7612-11e7-8b5e-22000b9923ef
laptoa: 1a1a1a1--7612-11e7-8b5e-22000b9923ef
then you can use the keyword uva for an endpoint in the command
Examples:
Transfer via multiple globus tasks
cd mydir
ls -1 > file.txt
cat -n file.txt
cms globus transfer mars uva `pwd` --file=files.txt /remotedir --mkdir=no --run=13:18 --tar
This command will from the file transfer the files in the lines 13 to 18 contained in files.txt.
Each line will be handled as a separate transfer and the data will be placed in a tar file. that
tar file will then be transferred. On the remote machine the tar file will not be untarred. This
you do have to do by loggin into the machine such as with ssh abd untar the file. T
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cloudmesh_globus-4.3.3-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f233be3ffccbf6daa2f8cfe35a95d999ddbf08bc22d70eba6cab42f642a724ef |
|
MD5 | 003c9bb5d7ebf69f7004a3e1bd3a93fc |
|
BLAKE2b-256 | fadffe9361fa9b4b499903b553fcb8a6e9b68915cdbf9996852fa3f3e71ee51b |