Skip to main content

Use linear programming to allocate applications to cloud infrastructure

Project description

Malloovia

Documentation Status  Build status

Use linear programming to allocate applications to cloud infrastructure.

Introduction

Malloovia is a Python package to solve virtual machine (VM) allocation problems in Infrastructure as a Service (IaaS) clouds from the point of view of the cloud customer. It was first presented in the paper "Cost Minimization of Virtual Machine Allocation in Public Clouds Considering Multiple Applications" presented at GECON 2017.

The problem to solve is: given a cloud infrastructure composed of different virtual machine types, each one with its own hardware characteristics, and prices, some of them with different pricing schemas, such as discounts for reservation over long periods, and given a set of applications to run on that infrastructure, each one with a different performance on each different VM type, and with a different workload over time, find the number of VMs of each type to activate at each timeslot for each application, so that the expected workload is fulfilled for all applications, the cloud provider limits are not exceeded and the total cost is minimized.

It works in two phases: first, it computes the number of reserved VMs using a Long Term Workload Prediction (LTWP) and then, it computes the number of on-demand for each time slot using a Short Term Workload Prediction (STWP).

Malloovia can be directly used in Python or by a CLI interface. The problems and the solutions can be saved using a YAML format.

This is an example that assumes that the problem definition is in problems.yaml, with problem1 describing the LTWP and problem2 describing the STWP:

$ malloovia solve problems.yaml --phase-i-id=problem1 --phase-ii-id=problem2
Reading problems.yaml...(0.004s)
Solving phase I...(0.020s)
Solving Phase II |███████████████████████████████████| 100.0% - ETA: 0:00:00
(0.101s)
Writing solutions in problems-sol.yaml...(0.006s)

This is an example in Python (explained in more detail in the Usage section of the documentation):

from malloovia import *

# Infrastructure definition
region1 = LimitingSet("r1", name="us.east", max_vms=20)
zone1 =  LimitingSet("r1_z1", name="us.east_a", max_vms=20)
m3large_z1 = InstanceClass(
    "m3large_r1_z1", name="reserved m3.large in us.east_a",
    limiting_sets=(zone1,), is_reserved=True,
    price=7, max_vms=20)
m4xlarge_r1 = InstanceClass(
    "m4xlarge_r1", name="ondemand m4.xlarge in us.east",
    limiting_sets=(region1,), is_reserved=False,
    price=10, max_vms=10)

# Performances
app0 = App("a0", "Web server")
app1 = App("a1", "Database")
performances = PerformanceSet(
    id="example_perfs",
    values=PerformanceValues({
        m3large_z1: {app0: 12, app1: 500},
        m4xlarge_r1: {app0: 44, app1: 1800}
        })
)

# Workload

# Long term workload prediction of each app, for Phase I
ltwp_app0 = Workload(
    "ltwp0", description="rph to the web server", app=app0,
    values=(201, 203, 180, 220, 190, 211, 199, 204, 500, 200)
)
ltwp_app1 = Workload(
    "ltwp1", description="rph to the database", app=app1,
    values=(2010, 2035, 1807, 2202, 1910, 2110, 1985, 2033, 5050, 1992)
)

# Building the problem for phase I and solving
problem = Problem(
    id="example1",
    name="Example problem",
    workloads=(ltwp_app0, ltwp_app1),
    instance_classes=(m3large_z1, m4xlarge_r1),
    performances=performances
)

phase_i_solution = PhaseI(problem).solve()

# Building the problem for a timeslot in phase II and solving
phase_ii = PhaseII(problem, phase_i_solution)
timeslot_solution = phase_ii.solve_timeslot(
    workloads=(Workload("stwp0", app=app0, description=None, values=(315,)),
               Workload("stwp1", app=app1, description=None, values=(1950,))
               )
    )

# Showing the cost and the allocation
print("Cost:", timeslot_solution.solving_stats.optimal_cost)
print(timeslot_solution.allocation._inspect())

You can find example problems and solutions in YAML format in the test data directory and in the GECON 2017 data repository, where you can find a notebook that shows how to compute the solutions from the problems.

Please, refer to the documentation and the he paper "Cost Minimization of Virtual Machine Allocation in Public Clouds Considering Multiple Applications" for more details.

History

0.1.0 (2017-07-24)

  • First release on PyPI.

0.2.0 (2017-07-27)

  • from malloovia import * imports all relevant classes and methods.
  • read_problems_from_github() added.
  • Integration with Travis-CI and ReadTheDocs.
  • Working on the documentation.
  • Modified YAML schema of the Solutions.

0.3.0 (2017-07-31)

  • Much improved documentation. Windows installation covered.
  • Command-line interface
  • Changed from PyYAML to ruamel.yaml, much faster
  • Read from YAML now accepts gzipped files too
  • Some bugs fixed in the schema

1.0.0 (2017-11-01)

  • Incompatible API change: it is required to specify time_unit in InstanceClass, PerformanceSet and Workload classes, in order to clarify the time unit for price, performance and workload timeslots.
  • Added utility function to read solutions from yaml files.
  • Revised documentation and code quality. Improved README for github.
  • Minor bugfixes.

1.0.1 (2018-01-12)

  • Bugfix to make all malloovia classes pickable, allowing for multiprocessing.

1.1.0 (2018-03-16)

  • New class PhaseIIGuided which allows to solve a single timeslot using a given allocation which specifies the minimum number of VMs to keep running.

2.1.1 (2019-06-20)

  • Internal refactorization of Malloovia Model's classes, which are now based on typing.NamedTuple instead of collections.namedtuple, which allows for proper type checking and documentation of the fields.
  • Several typing bugs related to YAML export and import fixed.
  • This version introduces backwards incompatibility, since it requires python 3.6+ to run. However the API and usage is the same.

2.2.0 (2020-03-04)

  • Updated to work with PuLP 2.0 (and fix that version in setup.py)
  • Fixed problem with LP variable names too long wen the number of apps in the problem is large.

2.3.0 (2020-09-25)

  • Modified to support hybrid clouds via is_private attribute in instance classes

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

malloovia-2.3.0.tar.gz (128.9 kB view hashes)

Uploaded Source

Built Distribution

malloovia-2.3.0-py3-none-any.whl (42.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page