Introduction

VmShepherd is an application that helps you maintain groups (clusters) of virtual machines. It keeps defined parameters (like count, image, etc.) by checking state in an IaaS layer, and tests underlying services with a specified health check.

Architecture

VmShepherd is designed to be easily extensible with plugins. Empowered by python3 and its asyncio module to facilitate scaling. The diagram below shows the base components of the app.

• Preset Manager is responsible for fetching cluster spec/definition (preset). Built-in presets: DirectoryDriver and GitRepoDriver

• Runtime Manger exposes functionality of locking preset, and holds intermediate states. Currently available: InMemoryDriver, PostgresDriver, ZookeeperDriver.

• IaaS is a glue (api wrapper) to IaaS provider, OpenStackDriver is the first implemented.

• Healthcheck allows to check service’s state, not only for existence of a virtual machine. HttpHealthcheckDriver is built-in.

For more information, please take look at the documentation.

Installation

Application requires python 3.6 or later. The latest stable version is available on Pypi.

pip install vmshepherd

Another way of installation for a VmShepherd is a docker. You can easily download latest version of our application from a docker hub.

docker pull dreamlabcloud/vmshepherd

Before you run a application, you need to prepare configuration files according to this rules.

When you create a configuration file, you can deploy a VmShepherd like that:

docker run -v $PATH_TO_CONFIG_DIRECTORY/:/home/shepherd -p 8888:8888 -it vmshepherd -c config/settings.example.yaml

• Where PATH_TO_CONFIG_DIRECTORY is a localisation of a configuration files on your host

• -c config/settings.example.yaml is a list of arguments passed to a VmShepherd in container

Example:

➜  VmShepherd/docker ✗ sudo docker run -v $(realpath ../)/:/home/shepherd -p   8888:8888 -it vmshepherd -c config/settings.example.yaml
INFO:root:Starting server, listening on 8888.
INFO:root:VmShepherd start cycle...
INFO:root:VMs Status: 1 expected, 0 in iaas, 0 running, 0 nearby shutdown, 0 pending, 0   after time shutdown, 0 terminated, 0 error, 0 unknown, 1 missing
INFO:root:VMs Status update: 0 terminated, 0 terminated by healthcheck, 1 created, 0 failed healthcheck

We also provide a Dockerfile which can be used during a development:

docker build -t vmshepherd .
docker run -it  -p 8888:8888 -p 8000:8000 vmshepherd run

Usage

After installation you need to create a main configuration file (check examples in config/ directory in this repo).

Run:

vmshepherd -c CONFIGFILE

Contributing to VmShepherd

Thank you for your interest in contributing to VmShepherd. Like always there are many ways to contribute, and we appreciate all of them.

Pull requests and issues are the primary mechanism we use to change VmShepherd. Github itself has a great documentation about using Pull Requests. We use the “fork and pull” model, where contributors push changes to their personal fork and create pull requests to bring those changes into the source repository.

If you want to find something to work on, please check issues in our roadmap.

Check out the documetation http://doc.dreamlab.pl/VmShepherd/development/index.html.

TL;DR

Pull requests will need:

• Tests

• Documentation

• A logical series of well written commits

License

Apache License 2.0

1.6.4 (2021-05-18)

• bugfix: preset logging

1.6.3 (2021-05-15)

• bugfix: aiohttp lib version

1.6.2 (2021-05-13)

• feature: log info how long virtual machine fails healthcheck

1.6.1 (2020-08-13)

• feature: new vm object state - unhealthy

1.6.0 (2019-12-18)

• feature: new rpc methods - list_presets_with_tags, get_preset_tags

1.5.2 (2019-09-03)

• bugfix: terminate via iaas on healthcheck

1.5.1 (2019-08-26)

• bugfix: vm class - remove terminate method

1.5.0 (2019-08-23)

• feature: the vms list is keeping in runtime driver (latest possible)

  ans is refreshed by the git worker on every cycle

1.4.2 (2019-08-14)

• bugfix: remove next _initialize_master_working_set line

1.4.1 (2019-08-14)

• bugfix: remove _initialize_master_working_set on each driver get

1.4.0 (2019-08-12)

• feature: rpc - list_vms - created at field

1.3.5 (2019-07-26)

• bugfix: fix problem with sys.path in drivers

• bugfix: reload entry_points

1.3.3 (2019-07-26)

• bugfix: add asyncio.lock for git clone/pull commands

1.3.2 (2019-07-26)

• bugfix: Add unique name for a tmp preset directory in a Git Driver

1.3.1 (2019-06-03)

• bugfix: change the return type for get_vm_ip

• feature: custom exception class for errors

1.3.0 (2019-06-03)

• feature: configure unmanaged state for presets

• feature: rpc api - get_vm_ip(preset_name, vm_id)

1.2.3 (2019-05-10)

• bugfix: panel - fail to load due jinja error

1.2.1 (2019-05-09)

• bugfix: update runtime data after manage even if unlocked

• bugfix: panel - deterministic preset order

1.2.0 (2018-08-07)

• feature: api - added list_presets

1.1.0 (2018-06-13)

• feature: adjustable timeout for http requests to iaas

1.0.1 (2018-06-08)

• bugfix: API should use last cycle data

• bugfix: OpenStackDriver refresh token

1.0.0 (2018-06-06)

• preserve preset state (dashboard shows data from last cycle)

• bump AsyncOpenStackClient (0.6.2)

• docs

0.7.7 (2018-05-10)

• bump AsyncOpenStackClient (0.5.2)

0.7.6 (2018-05-10)

• bump AsyncOpenStackClient (0.5.1)

0.7.5 (2018-04-09)

• runtime driver abstract fix

0.7.4 (2018-03-22)

• cleanup, verify

0.7.3 (2018-03-22)

• bump AsyncOpenStackClient (+ compat)

0.7.2 (2018-03-22)

• user-data base64

0.7.1 (2018-03-21)

• multiple network interfaces

0.7.0 (2018-03-20)

• AsyncOpenStackClient

0.6.0 (2018-02-01)

• web panel

• rpc api

0.0.3 (2018-02-01)

• initial