Skyplane efficiently transports data between cloud regions and providers.
Project description
Skyplane: A Unified Data Layer for the Multi-Cloud
Skyplane is lifting cloud object stores to the Sky.
Instructions to build and run demo
Skyplane is composed of two components: A ReplicatorClient that runs locally on your machine that is responsible for provisioning instances and coordinating replication jobs and a GatewayDaemon that runs on each provisioned instance to actually copy data through the overlay.
This package represents both components as a single binary. Docker builds a single container with the GatewayDaemon and pushes it to the Github Container Registry (ghcr.io). After provisioning an instance, a GatewayDaemon is started by launching that container. Therefore, it's simple and fast to launch a new Gateway.
Requirements
- Python 3.7 or greater
- Docker
- Ensure you have authenticated your Github account with Docker: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry
- TLDR:
- (1) Install docker with
curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
- (2) Create a Personal Access Token at https://github.com/settings/tokens/new with "write:packages" permissions
- (3) Run
echo <PERSONAL_ACCESS_TOKEN> | sudo docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
- (1) Install docker with
- AWS:
- (1) Install AWS CLI with
sudo apt install awscli
- (2) Configure AWS by running
aws configure
and input the necessary information. See https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html. - (3) Ensure that you have a sufficient AWS vCPU limit in any regions you intend to use
- (4) Install netcat with
sudo apt install netcat
- (1) Install AWS CLI with
Building and deploying the gateway
First, clone and enter the skyplane directory:
$ git clone https://github.com/skyplane-project/skyplane
$ cd skyplane
Then, configure cloud credentials as needed:
- AWS:
aws configure
- GCS:
gcloud auth application-default login
- Azure:
az login
Finally, install and initalize Skyplane:
$ pip install -e ".[all]"
$ skyplane init
To run a sample transfer, first build a new version of the GatewayDaemon Docker image and push it to ghcr.io (ensure you are authenticated as above):
$ source scripts/pack_docker.sh
pack_docker result
$ pip install -e ".[all]"
$ source scripts/pack_docker.sh
Building docker image
[+] Building 0.0s (2/2) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount683951637/Dockerfile: no such file or directory
Uploading docker image to ghcr.io/skyplane-project/skyplane:local-PotRzrFT
The push refers to repository [ghcr.io/skyplane-project/skyplane]
20d2ed8618ca: Layer already exists
1c4146875228: Layer already exists
1f4f7ac2f199: Layer already exists
d1e36ec88afa: Layer already exists
824bf068fd3d: Layer already exists
local-PotRzrFT: digest: sha256:f412e376290d5a7bad28aca57ce9ffcf579e8dd7db3f4d6fb68ceae829d0a6b2 size: 1371
Deleted build cache objects:
tltkismwtov5n8zokghil1py9
u0e2ymhmv64oriiq66ibepn63
Total reclaimed space: 0B
SKYPLANE_DOCKER_IMAGE=ghcr.io/skyplane-project/skyplane:local-PotRzrFT
The script will export the new image (ghcr.io/skyplane-project/skyplane:local-PotRzrFT in this example) to an environment variable (SKYPLANE_DOCKER_IMAGE
). Ensure you use source
so the environment variable is published to your shell.
Running a basic cloud to cloud transfer job
We then run the ReplicatorClient with that new Docker image (stored in $SKYPLANE_DOCKER_IMAGE
):
$ skyplane cp [s3/gs/azure]://[source-bucket-name]/[source-key-prefix] [s3/gs/azure]://[dest-bucket-name]/[dest-key-prefix]
skyplane cp result
$ skyplane cp s3://skyplane-example-us-east-1/ s3://skyplane-example-us-west-1/
11:34:48 [DEBUG] Cloud SSH key initialization: 3.23s
11:35:20 [DEBUG] Provisioning instances and waiting to boot: 31.87s
11:35:24 [DEBUG] Install docker: 2.79s
11:35:24 [DEBUG] Install docker: 2.50s
11:35:37 [DEBUG] Starting gateway aws:us-west-1:i-09dda9567bcf9ecad, host: 52.53.229.126: Docker pull: 13.10s
11:35:37 [DEBUG] Starting gateway aws:us-west-1:i-09dda9567bcf9ecad, host: 52.53.229.126: Starting gateway container
11:35:39 [DEBUG] Starting gateway aws:us-west-1:i-09dda9567bcf9ecad, host: 52.53.229.126: Gateway started fabfc1cd5aefa24c0cb5d5572501b19ff33e483cadfcccddc9bd0d90368c5329
11:36:05 [DEBUG] Starting gateway aws:us-east-1:i-08a9b4f70ee2caca3, host: 54.158.252.172: Docker pull: 39.93s
11:36:05 [DEBUG] Starting gateway aws:us-east-1:i-08a9b4f70ee2caca3, host: 54.158.252.172: Starting gateway container
11:36:14 [DEBUG] Starting gateway aws:us-east-1:i-08a9b4f70ee2caca3, host: 54.158.252.172: Gateway started 18ebc8a3a04b632375a71ae88e18286d402364e467b30b00ad3168391a914eaf
11:36:15 [DEBUG] Install gateway package on instances: 55.05s
11:36:15 [INFO] Provisioned ReplicationTopologyGateway(region='aws:us-east-1', instance=0): http://54.158.252.172:8888/container/18ebc8a3a04b
11:36:15 [INFO] Provisioned ReplicationTopologyGateway(region='aws:us-west-1', instance=0): http://52.53.229.126:8888/container/fabfc1cd5aef
11:36:15 [INFO] Batch 0 size: 4387690 with 3 chunks
11:36:15 [DEBUG] Building chunk requests: 0.00s
11:36:15 [DEBUG] Sending 3 chunk requests to 54.158.252.172
11:36:15 [DEBUG] Dispatch chunk requests: 0.27s
11:36:15 [INFO] 0.00GByte replication job launched
0/3 chunks done (0.00 / 0.00GB, 0.00Gbit/s, ETA=unknown)
Replication: average 0.02Gbit/s: 100%|███████████████████████████████████████████████████████████████████████████████| 33.5M/33.5M [00:02<00:00, 17.0Mbit/s]
11:36:17 [INFO] Copying gateway logs from aws:us-east-1:i-08a9b4f70ee2caca3
11:36:17 [INFO] Copying gateway logs from aws:us-west-1:i-09dda9567bcf9ecad
11:36:21 [DEBUG] Wrote profile to /tmp/skyplane/transfer_2022-03-29T11:36:17.755039/traceevent_5cb6dfbf-fac6-4ce6-a61b-1b813e83723d.json, visualize using `about://tracing` in Chrome
11:36:22 [WARN] Deprovisioning 2 instances
11:36:23 [WARN] Deprovisioned aws:us-west-1:i-09dda9567bcf9ecad
11:36:24 [WARN] Deprovisioned aws:us-east-1:i-08a9b4f70ee2caca3
{"total_runtime_s": 1.692787, "throughput_gbits": 0.019311843710588934, "monitor_status": "completed", "success": true}
When done, stop all instances started by Skyplane by running:
skyklark deprovision
skyplane deprovision result
$ skyplane deprovision
No GCP project ID given, so will only deprovision AWS instances
Deprovisioning 3 instances
Deprovisioning (aws:ap-northeast-1): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00, 2.33it/s]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for skyplane-nightly-0.0.1.dev20220526.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77c7822eb1cb26d49537ef0b47348ea93336324c6f4d28c242d661a1203da0e9 |
|
MD5 | c1779a028fad8518cfb72cd55c3d10d1 |
|
BLAKE2b-256 | 808a63e9aff50a38fb8d534ea5d8451f40691ed699bfb8f5816614ed7bbaa738 |
Hashes for skyplane_nightly-0.0.1.dev20220526-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b7fd67868f06f0965ac6ad8dcce19a10cbdbebf7e8188f085eebb5f3dfd7384d |
|
MD5 | 2254af090b3e21cdbed7d3e729a94ea2 |
|
BLAKE2b-256 | fc13d0c68eede143b0f1672f89e3e4042676fae00bd1bfd968e2950f4cacd124 |