This release is a pre-release and may not be stable for production use.
Skyplane: A Unified Data Layer for the Multi-Cloud
Skyplane is lifting cloud object stores to the Sky.
Instructions to build and run demo
Skyplane is composed of two components: A ReplicatorClient that runs locally on your machine that is responsible for provisioning instances and coordinating replication jobs and a GatewayDaemon that runs on each provisioned instance to actually copy data through the overlay.
This package represents both components as a single binary. Docker builds a single container with the GatewayDaemon and pushes it to the Github Container Registry (ghcr.io). After provisioning an instance, a GatewayDaemon is started by launching that container. Therefore, it's simple and fast to launch a new Gateway.
Requirements
- Python 3.7 or greater
- Docker
- Ensure you have authenticated your Github account with Docker: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry
- TLDR:
- (1) Install docker with
curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh - (2) Create a Personal Access Token at https://github.com/settings/tokens/new with "write:packages" permissions
- (3) Run
echo <PERSONAL_ACCESS_TOKEN> | sudo docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
- (1) Install docker with
- AWS:
- (1) Install AWS CLI with
sudo apt install awscli - (2) Configure AWS by running
aws configureand input the necessary information. See https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html. - (3) Ensure that you have a sufficient AWS vCPU limit in any regions you intend to use
- (4) Install netcat with
sudo apt install netcat
- (1) Install AWS CLI with
Building and deploying the gateway
First, clone and enter the skyplane directory:
$ git clone https://github.com/skyplane-project/skyplane
$ cd skyplane
Then, configure cloud credentials as needed:
- AWS:
aws configure - GCS:
gcloud auth application-default login - Azure:
az login
Finally, install and initalize Skyplane:
$ pip install -e ".[all]"
$ skyplane init
To run a sample transfer, first build a new version of the GatewayDaemon Docker image and push it to ghcr.io (ensure you are authenticated as above):
$ source scripts/pack_docker.sh
pack_docker result
$ pip install -e ".[all]"
$ source scripts/pack_docker.sh
Building docker image
[+] Building 0.0s (2/2) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount683951637/Dockerfile: no such file or directory
Uploading docker image to ghcr.io/skyplane-project/skyplane:local-PotRzrFT
The push refers to repository [ghcr.io/skyplane-project/skyplane]
20d2ed8618ca: Layer already exists
1c4146875228: Layer already exists
1f4f7ac2f199: Layer already exists
d1e36ec88afa: Layer already exists
824bf068fd3d: Layer already exists
local-PotRzrFT: digest: sha256:f412e376290d5a7bad28aca57ce9ffcf579e8dd7db3f4d6fb68ceae829d0a6b2 size: 1371
Deleted build cache objects:
tltkismwtov5n8zokghil1py9
u0e2ymhmv64oriiq66ibepn63
Total reclaimed space: 0B
SKYPLANE_DOCKER_IMAGE=ghcr.io/skyplane-project/skyplane:local-PotRzrFT
The script will export the new image (ghcr.io/skyplane-project/skyplane:local-PotRzrFT in this example) to an environment variable (SKYPLANE_DOCKER_IMAGE). Ensure you use source so the environment variable is published to your shell.
Running a basic cloud to cloud transfer job
We then run the ReplicatorClient with that new Docker image (stored in $SKYPLANE_DOCKER_IMAGE):
$ skyplane cp [s3/gs/azure]://[source-bucket-name]/[source-key-prefix] [s3/gs/azure]://[dest-bucket-name]/[dest-key-prefix]
skyplane cp result
$ skyplane cp s3://skyplane-example-us-east-1/ s3://skyplane-example-us-west-1/
11:34:48 [DEBUG] Cloud SSH key initialization: 3.23s
11:35:20 [DEBUG] Provisioning instances and waiting to boot: 31.87s
11:35:24 [DEBUG] Install docker: 2.79s
11:35:24 [DEBUG] Install docker: 2.50s
11:35:37 [DEBUG] Starting gateway aws:us-west-1:i-09dda9567bcf9ecad, host: 52.53.229.126: Docker pull: 13.10s
11:35:37 [DEBUG] Starting gateway aws:us-west-1:i-09dda9567bcf9ecad, host: 52.53.229.126: Starting gateway container
11:35:39 [DEBUG] Starting gateway aws:us-west-1:i-09dda9567bcf9ecad, host: 52.53.229.126: Gateway started fabfc1cd5aefa24c0cb5d5572501b19ff33e483cadfcccddc9bd0d90368c5329
11:36:05 [DEBUG] Starting gateway aws:us-east-1:i-08a9b4f70ee2caca3, host: 54.158.252.172: Docker pull: 39.93s
11:36:05 [DEBUG] Starting gateway aws:us-east-1:i-08a9b4f70ee2caca3, host: 54.158.252.172: Starting gateway container
11:36:14 [DEBUG] Starting gateway aws:us-east-1:i-08a9b4f70ee2caca3, host: 54.158.252.172: Gateway started 18ebc8a3a04b632375a71ae88e18286d402364e467b30b00ad3168391a914eaf
11:36:15 [DEBUG] Install gateway package on instances: 55.05s
11:36:15 [INFO] Provisioned ReplicationTopologyGateway(region='aws:us-east-1', instance=0): http://54.158.252.172:8888/container/18ebc8a3a04b
11:36:15 [INFO] Provisioned ReplicationTopologyGateway(region='aws:us-west-1', instance=0): http://52.53.229.126:8888/container/fabfc1cd5aef
11:36:15 [INFO] Batch 0 size: 4387690 with 3 chunks
11:36:15 [DEBUG] Building chunk requests: 0.00s
11:36:15 [DEBUG] Sending 3 chunk requests to 54.158.252.172
11:36:15 [DEBUG] Dispatch chunk requests: 0.27s
11:36:15 [INFO] 0.00GByte replication job launched
0/3 chunks done (0.00 / 0.00GB, 0.00Gbit/s, ETA=unknown)
Replication: average 0.02Gbit/s: 100%|███████████████████████████████████████████████████████████████████████████████| 33.5M/33.5M [00:02<00:00, 17.0Mbit/s]
11:36:17 [INFO] Copying gateway logs from aws:us-east-1:i-08a9b4f70ee2caca3
11:36:17 [INFO] Copying gateway logs from aws:us-west-1:i-09dda9567bcf9ecad
11:36:21 [DEBUG] Wrote profile to /tmp/skyplane/transfer_2022-03-29T11:36:17.755039/traceevent_5cb6dfbf-fac6-4ce6-a61b-1b813e83723d.json, visualize using `about://tracing` in Chrome
11:36:22 [WARN] Deprovisioning 2 instances
11:36:23 [WARN] Deprovisioned aws:us-west-1:i-09dda9567bcf9ecad
11:36:24 [WARN] Deprovisioned aws:us-east-1:i-08a9b4f70ee2caca3
{"total_runtime_s": 1.692787, "throughput_gbits": 0.019311843710588934, "monitor_status": "completed", "success": true}
When done, stop all instances started by Skyplane by running:
skyklark deprovision
skyplane deprovision result
$ skyplane deprovision
No GCP project ID given, so will only deprovision AWS instances
Deprovisioning 3 instances
Deprovisioning (aws:ap-northeast-1): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00, 2.33it/s]
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file skyplane-nightly-0.0.1.dev20220525.tar.gz.
File metadata
- Download URL: skyplane-nightly-0.0.1.dev20220525.tar.gz
- Upload date:
- Size: 93.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.2.0b1 CPython/3.8.12 Linux/5.13.0-1023-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
469057f7be94e975ba68c375bb4f98892f878b1b7f7dde40c845575502e48d10
|
|
| MD5 |
c0610bbd0bdd0393be281e2905c3af9e
|
|
| BLAKE2b-256 |
e0372cd015aff63a233bf895cf7ce995224ab04bc68b6c2acbcbe4938db50bef
|
File details
Details for the file skyplane_nightly-0.0.1.dev20220525-py3-none-any.whl.
File metadata
- Download URL: skyplane_nightly-0.0.1.dev20220525-py3-none-any.whl
- Upload date:
- Size: 112.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.2.0b1 CPython/3.8.12 Linux/5.13.0-1023-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6f38f8e9e41fb79b34e6768b0f689e2c8e303216bfba24ff705445d7ad176393
|
|
| MD5 |
8aa79d08149b754514fe7337dde12af7
|
|
| BLAKE2b-256 |
204b8c654c11c57216ed22399d0da56a71f82b36cf4c4422dd46d4f78796ba3c
|
