Skip to main content

An AWS spot orchestrator for SPITZ enabled programs

Project description

CloudPITS / Instance Selection

This is the initial idea behind Cloud-PITS, which involves selecting a group of instances that provides a price per cost ratio that will execute a given SPITS code in less than an user-defined input amount. Since this is a prototype it still has a lot of limitations and is not ready for general usage. However the general idea behind the algorithm is here.

Disclaimer

This is still a proof of concept. The SPITS program used for validation computes the Zero Offset Non-hyperbolic Common Reflection Surface (DOI:10.1111/j.1365-2478.2012.01055.x) parameters, therefore there are some parameters that are currenlty tailored for that specific software. This is not yet generic for any SPITS program!

How it works

The instance selection algorithm works with a stored database containing previous executions performance measurements. By using previous executions information for the same experiment it is possible to infer how the current experiment will perform, therefore allows us to create a good initial instance poll. Additionally, by verifying periodically how the instances are performing and how expensive they are at a given moment, it is possible to select the types that offer better performance for their money.

The three main files are the following (with their input/output described in their own headers):

launch/create_ondemand_jm.py
populate/simulation.py
populate/to_execute.py

The first one (launch/create_ondemand_zocrsomp_jm.py) is used to launch an On-Demand instance that should work as the master instance (Job Manager and instance selector). Therefore it is necessary that in some part of the user_data variable inserting the code for calling the "to_execute.py" code to call this instance selection algorithm and start the Job Manager process.

The second one simulates an execution, given previous the information stored in the database and the price log stored in a file (populate/log_prices.csv). This allows the user to simulate how much would be spent for each input price.

The last one is the actual Python script to be executed, in which the script will start a new poll of instances and then verify from time to time for instances that are performing below the desired cost vs performance threshold. Replacing bad performing types with better ones.

Database

The algorithm extracts information from a database as configured in the "rds_config.py" file. The database should be called "experimentos" and have the tables as displayed in the picture below. There is a SQL script to generate it in the folder "databases/".

Performance measurement report

The worker instances must report their performance to the Job Manager via CloudWatch. To do so, the user needs to create a new metric with the following characteristics:

Namespace='Performance',
MetricName='perf_sec',
Dimensions=[{'Name': 'Instance Id', 'Value': instance_id},
            {'Name': 'Type', 'Value': instance_type}]

Namespace='Performance',
MetricName='perf_sec_stdev',
Dimensions=[{'Name': 'Instance Id', 'Value': instance_id},
            {'Name': 'Type', 'Value': instance_type}]

Namespace='Performance',
MetricName='tasks_completed',
Dimensions=[{'Name': 'Instance Id', 'Value': instance_id},
            {'Name': 'Type', 'Value': instance_type}]

This report also goes to the database aforementioned so that future executions use them. Also, they are mandatory so that the instance selection Python script can select the initial poll of instances for the SPITS program being optimized.

Dependencies

The program have a few dependencies. They can be installed using the python pip command:

python3 -m pip install --user -r requirements.txt

Most Python packages already come with the some dependencies installed, however, two of them are not included, namely:

boto3: To access the AWS instances
pymysql: To enable Python to access a MySQL database and perform queries

Further information about all files are in their own documentations.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cloudpits-0.0.1.tar.gz (19.1 kB view details)

Uploaded Source

Built Distributions

cloudpits-0.0.1-py3-none-any.whl (29.9 kB view details)

Uploaded Python 3

cloudpits-0.0.1-py2-none-any.whl (29.9 kB view details)

Uploaded Python 2

File details

Details for the file cloudpits-0.0.1.tar.gz.

File metadata

  • Download URL: cloudpits-0.0.1.tar.gz
  • Upload date:
  • Size: 19.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.4.3

File hashes

Hashes for cloudpits-0.0.1.tar.gz
Algorithm Hash digest
SHA256 67dd1f4a714d14110a82e30702603b3fce28310fa056da20e79e5a3b15415e2c
MD5 76b5ef2f84b5789731dd94ecc82a1ddf
BLAKE2b-256 7c7a16eed351c5aea2f63b9058539f4f27c738bde3acd804e80f054fca96e16f

See more details on using hashes here.

File details

Details for the file cloudpits-0.0.1-py3-none-any.whl.

File metadata

  • Download URL: cloudpits-0.0.1-py3-none-any.whl
  • Upload date:
  • Size: 29.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.4.3

File hashes

Hashes for cloudpits-0.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 59b4c9538964cc06a3ab507cd912125e04a84acf92d280a04ff3dd0fc7871795
MD5 580168d85a3bbb9d30edd4e741a6267b
BLAKE2b-256 6df8fb6213f53e0e1cb605ae7fb074914acae128700235f0272d3ae5f6eaff55

See more details on using hashes here.

File details

Details for the file cloudpits-0.0.1-py2-none-any.whl.

File metadata

  • Download URL: cloudpits-0.0.1-py2-none-any.whl
  • Upload date:
  • Size: 29.9 kB
  • Tags: Python 2
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.4.3

File hashes

Hashes for cloudpits-0.0.1-py2-none-any.whl
Algorithm Hash digest
SHA256 7f41a6174c2b6dc1d667e42b7fbe70a36c4914b480833a8f3c430dbc0fd41b01
MD5 75b6b4746249998fbaccde1d28f25638
BLAKE2b-256 13c259ffa21cc182833a770f7feb7cd37b4297c6c4f9821948609301603a6c4a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page