File backend component of Cumulus
Project description
Stratocumulus is the backend component of Cumulus, which aims to providing a unified backend interface for different cloud environments.
Installation
Stratocumulus does not include any Cloud SDK. You need to install the corresponding SDK separately, depending on which Cloud platform you use:
Google Cloud: Install Google Cloud SDK.
Amazon AWS: Install AWS CLI.
Local machine: Works for Linux and macOS. Please make sure rsync is installed, as some commands will use it.
After that, install Stratocumulus in your Python environment by command:
pip install stratocumulus
Usage
Stratocumulus has 4 commands:
cp: Copy file(s)/folder(s).
sync: Synchronize two folders, usually used for copying one folder.
rm: Remove file(s)/folder(s).
exist: Check if a (remote) file/folder exists.
These commands have options specific to backend. For now, Stratocumulus supports the following backends:
aws: Amazon AWS.
gcp: Google Cloud.
local: Local machine. In specific, macOS or Linux.
cp
This command copies file(s) or folder(s). See examples below:
# AWS upload strato cp file1 folder2 s3://my-bucket/target_folder/ # AWS download strato cp s3://my-bucket/source_folder/*.zip /target_folder/ # GCP upload strato cp -m -r --ionice file1 folder2 gs://my-bucket/target_folder/ # GCP download mkdir /target_folder strato cp -m gs://my-bucket/source_folder/*.zip /target_folder/ # Local Machine strato cp -r file1 folder2 /target_folder/
Notice that:
For AWS backend, you must explicitly attach a trailing slash for source folder.
For Google Cloud download, you’ll have to explicitly create the target folder, and then run strato cp command.
Wildcards are acceptable.
-r option is not needed for AWS, and copying folders is always recursive.
-m and --ionice options only work for Google Cloud.
For details on the options, try command strato cp -h.
sync
This command synchronizes two folders. Notice that this synchronization will delete content in the target folder not existing in the source folder.
See examples below:
# AWS strato sync source_folder s3://my-bucket/target_folder # GCP strato sync -m --ionice source_folder gs://my-bucket/target_folder # Local Machine strato sync source_folder target_folder
Notice that:
-m and --ionice options only work for Google Cloud.
For details on the options, try command strato sync -h.
rm
This command deletes file(s) or folder(s). See examples below:
# AWS strato rm s3://my-bucket/file1 s3://my-bucket/folder2/ # GCP strato rm -m gs://my-bucket/file1 gs://my-bucket/folder2 gs://my-bucket/folder3/*.zip # Local Machine strato rm file1 folder2
Notice that:
-m option only works for Google Cloud.
For AWS backend, wildcards are not yet accepted.
For details on the options, try command strato rm -h.
exists
This command checks if a (remote) file or folder exists. If it exists, the command terminates without any output. Otherwise, the command will break with Python subprocess.CalledProcessError exception.
See examples below:
# AWS strato exists s3://my-bucket/file1 # GCP strato exists gs://my-bucket/folder2/ # Local Machine strato exists folder2/
Notice that this command works for both file and folder, regardless of backend.
For details on the options, try command strato exists -h.
help
Type strato -h or strato --help to check available Stratocumulus commands.
Check version
Type strato -v or strato --version to check the version of Stratocumulus currently installed on your machine.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for stratocumulus-0.2.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab8e3bcd8ef3a05b2c4c3b928f0bce9e563c160b2d7f6e96fe7d9e3b5c8c9ab2 |
|
MD5 | 1b6d489e8d8cd7f411abe2457174fa76 |
|
BLAKE2b-256 | 71123d3ed26d44487d43425568c49500ff4d314a0f9b007e5788d08dd95b31ca |