michaelvll-skypilot 0.0.2

An intercloud broker above the cloud

Sky

Sky is a framework to run any workload seamlessly across different cloud providers through a unified interface. No knowledge of cloud offerings is required or expected – you simply define the workload and its resource requirements, and Sky will automatically execute it on AWS, Google Cloud Platform or Microsoft Azure.

Getting Started

Please refer to our documentation.

• Installation
• Quickstart
• Sky CLI

Developer Guide

Setup

Use editable mode (-e) when installing:

# Sky requires python >= 3.6 and < 3.10.
# You can just install the dependencies for
# certain clouds, e.g., ".[aws,azure,gcp]"
pip install -e ".[all]"
pip install -r requirements-dev.txt

Submitting pull requests

• After you commit, format your code with format.sh.
• In the PR description, write a Tested: section to describe relevant tests performed.
• For changes that touch the core system, run the smoke tests and ensure they pass.
• Follow the Google style guide.

Environment Variable Options

• export SKY_DEV=1 to show debugging logs (logging.DEBUG).
• export SKY_DISABLE_USAGE_COLLECTION=1 to disable usage logging.
• export SKY_MINIMIZE_LOGGING=1 to minimize the sky outputs for demo purpose.

Dump timeline

Timeline is useful for performance analysis and debugging in Sky.

Here are the APIs:

from utils import timeline


# record a function in the timeline with the function path name
@timeline.Event.decorator
def f(): ...


# record a function in the timeline using name='my_name'
@timeline.Event.decorator(name='event_name')
def f(): ...


# record an event over a code block in the timeline:
with timeline.Event(name='event_name'):
  ...

# use a file lock with event:
with timeline.FileLockEvent(lockpath):
  pass

To dump the timeline, set environment variable SKY_TIMELINE_FILE_PATH to a file path.

View the dumped timeline file using Chrome (chrome://tracing) or Perfetto.

Updating the sky docker image

1. Authenticate with sky ECR repository. Contact [e-mail] for access:

   aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/a9w6z7w5
   

2. Build and tag the docker image:

   docker build -t public.ecr.aws/a9w6z7w5/sky:latest .
   

3. Push the image to ECR:

   docker push public.ecr.aws/a9w6z7w5/sky:latest
   

Some general engineering practice suggestions

These are suggestions, not strict rules to follow. When in doubt, follow the style guide.

• Use TODO(author_name)/FIXME(author_name) instead of blank TODO/FIXME. This is critical for tracking down issues. You can write TODOs with your name and assign it to others (on github) if it is someone else's issue.
• Delete your branch after merging it. This keeps the repo clean and faster to sync.
• Use an exception if this is an error. Only use assert for debugging or proof-checking purpose. This is because exception messages usually contain more information.
• Use modern python features and styles that increases code quality.
  • Use f-string instead of .format() for short expressions to increase readability.
  • Use class MyClass: instead of class MyClass(object):. The later one was a workaround for python2.x.
  • Use abc module for abstract classes to ensure all abstract methods are implemented.
  • Use python typing. But you should not import external objects just for typing. Instead, import typing-only external objects under if typing.TYPE_CHECKING:.

Testing

To run smoke tests:

bash tests/run_smoke_tests.sh

# Run one of the smoke tests
bash tests/run_smoke_tests.sh test_minimal

For profiling code, use:

pip install tuna # Tuna for viz
python3 -m cProfile -o sky.prof -m sky.cli status # Or some other command
tuna sky.prof