stratocumulus 0.3.0

File backend component of Cumulus

Stratocumulus is the backend component of Cumulus, which aims to providing a unified backend interface for different cloud environments.

Installation

Stratocumulus does not include any Cloud SDK. You need to install the corresponding SDK separately, depending on which Cloud platform you use:

• Google Cloud: Install Google Cloud SDK.

• Amazon AWS: Install AWS CLI.

• Local machine: Works for Linux and macOS. Please make sure rsync is installed, as some commands will use it.

After that, install Stratocumulus in your Python environment by command:

pip install stratocumulus

Usage

Stratocumulus has 4 commands:

• cp: Copy file(s)/folder(s).

• sync: Synchronize two folders, usually used for copying one folder.

• rm: Remove file(s)/folder(s).

• exist: Check if a (remote) file/folder exists.

These commands have options specific to backend. For now, Stratocumulus supports the following backends:

• aws: Amazon AWS.

• gcp: Google Cloud.

• local: Local machine. In specific, macOS or Linux.

cp

This command copies file(s) or folder(s). See examples below:

# AWS upload
strato cp file1 folder2 s3://my-bucket/target_folder/
# AWS download
strato cp s3://my-bucket/source_folder/*.zip /target_folder/

# GCP upload
strato cp -r --ionice file1 folder2 gs://my-bucket/target_folder/
# GCP download
mkdir /target_folder
strato cp gs://my-bucket/source_folder/*.zip /target_folder/

# Local Machine
strato cp -r file1 folder2 /target_folder/

Notice that:

• For AWS backend, you must explicitly attach a trailing slash for source folder.

• For Google Cloud download, you’ll have to explicitly create the target folder, and then run strato cp command.

• Wildcards are acceptable.

• -r option is not needed for AWS, and copying folders is always recursive.

• --ionice option only work for Google Cloud.

For details on the options, try command strato cp -h.

sync

This command synchronizes two folders. Notice that this synchronization will delete content in the target folder not existing in the source folder.

See examples below:

# AWS
strato sync source_folder s3://my-bucket/target_folder
# GCP
strato sync --ionice source_folder gs://my-bucket/target_folder
# Local Machine
strato sync source_folder target_folder

Notice that:

• --ionice option only work for Google Cloud.

For details on the options, try command strato sync -h.

rm

This command deletes file(s) or folder(s). See examples below:

# AWS
strato rm s3://my-bucket/file1 s3://my-bucket/folder2/
# GCP
strato rm gs://my-bucket/file1 gs://my-bucket/folder2 gs://my-bucket/folder3/*.zip
# Local Machine
strato rm file1 folder2

Notice that:

• For AWS backend, wildcards are not yet accepted.

For details on the options, try command strato rm -h.

exists

This command checks if a (remote) file or folder exists. If it exists, the command terminates without any output. Otherwise, the command will break with Python subprocess.CalledProcessError exception.

See examples below:

# AWS
strato exists s3://my-bucket/file1
# GCP
strato exists gs://my-bucket/folder2/
# Local Machine
strato exists folder2/

Notice that this command works for both file and folder, regardless of backend.

For details on the options, try command strato exists -h.

help

Type strato -h or strato --help to check available Stratocumulus commands.

Check version

Type strato -v or strato --version to check the version of Stratocumulus currently installed on your machine.