Skip to main content

Cluster manager

Project description


image0 image1


VmShepherd is an application that helps you maintain groups (clusters) of virtual machines. It keeps defined parameters (like count, image, etc.) by checking state in an IaaS layer, and tests underlying services with a specified health check.


VmShepherd is designed to be easily extensible with plugins. Empowered by python3 and its asyncio module to facilitate scaling. The diagram below shows the base components of the app.
  • Preset Manager is responsible for fetching cluster spec/definition (preset). Built-in presets: DirectoryDriver and GitRepoDriver

  • Runtime Manger exposes functionality of locking preset, and holds intermediate states. Currently available: InMemoryDriver, PostgresDriver, ZookeeperDriver.

  • IaaS is a glue (api wrapper) to IaaS provider, OpenStackDriver is the first implemented.

  • Healthcheck allows to check service’s state, not only for existence of a virtual machine. HttpHealthcheckDriver is built-in.

For more information, please take look at the documentation.


Application requires python 3.6 or later. The latest stable version is available on Pypi.

pip install vmshepherd

Another way of installation for a VmShepherd is a docker. You can easily download latest version of our application from a docker hub.

docker pull dreamlabcloud/vmshepherd

Before you run a application, you need to prepare configuration files according to this rules.

When you create a configuration file, you can deploy a VmShepherd like that:

docker run -v $PATH_TO_CONFIG_DIRECTORY/:/home/shepherd -p 8888:8888 -it vmshepherd -c config/settings.example.yaml
  • Where PATH_TO_CONFIG_DIRECTORY is a localisation of a configuration files on your host

  • -c config/settings.example.yaml is a list of arguments passed to a VmShepherd in container


➜  VmShepherd/docker ✗ sudo docker run -v $(realpath ../)/:/home/shepherd -p   8888:8888 -it vmshepherd -c config/settings.example.yaml
INFO:root:Starting server, listening on 8888.
INFO:root:VmShepherd start cycle...
INFO:root:VMs Status: 1 expected, 0 in iaas, 0 running, 0 nearby shutdown, 0 pending, 0   after time shutdown, 0 terminated, 0 error, 0 unknown, 1 missing
INFO:root:VMs Status update: 0 terminated, 0 terminated by healthcheck, 1 created, 0 failed healthcheck

We also provide a Dockerfile which can be used during a development:

docker build -t vmshepherd .
docker run -it  -p 8888:8888 -p 8000:8000 vmshepherd run


After installation you need to create a main configuration file (check examples in config/ directory in this repo).


vmshepherd -c CONFIGFILE

Contributing to VmShepherd

Thank you for your interest in contributing to VmShepherd. Like always there are many ways to contribute, and we appreciate all of them.

Pull requests and issues are the primary mechanism we use to change VmShepherd. Github itself has a great documentation about using Pull Requests. We use the “fork and pull” model, where contributors push changes to their personal fork and create pull requests to bring those changes into the source repository.

If you want to find something to work on, please check issues in our roadmap.

Check out the documetation


Pull requests will need:


Apache License 2.0


1.6.4 (2021-05-18)

  • bugfix: preset logging

1.6.3 (2021-05-15)

  • bugfix: aiohttp lib version

1.6.2 (2021-05-13)

  • feature: log info how long virtual machine fails healthcheck

1.6.1 (2020-08-13)

  • feature: new vm object state - unhealthy

1.6.0 (2019-12-18)

  • feature: new rpc methods - list_presets_with_tags, get_preset_tags

1.5.2 (2019-09-03)

  • bugfix: terminate via iaas on healthcheck

1.5.1 (2019-08-26)

  • bugfix: vm class - remove terminate method

1.5.0 (2019-08-23)

  • feature: the vms list is keeping in runtime driver (latest possible)

    ans is refreshed by the git worker on every cycle

1.4.2 (2019-08-14)

  • bugfix: remove next _initialize_master_working_set line

1.4.1 (2019-08-14)

  • bugfix: remove _initialize_master_working_set on each driver get

1.4.0 (2019-08-12)

  • feature: rpc - list_vms - created at field

1.3.5 (2019-07-26)

  • bugfix: fix problem with sys.path in drivers

  • bugfix: reload entry_points

1.3.3 (2019-07-26)

  • bugfix: add asyncio.lock for git clone/pull commands

1.3.2 (2019-07-26)

  • bugfix: Add unique name for a tmp preset directory in a Git Driver

1.3.1 (2019-06-03)

  • bugfix: change the return type for get_vm_ip

  • feature: custom exception class for errors

1.3.0 (2019-06-03)

  • feature: configure unmanaged state for presets

  • feature: rpc api - get_vm_ip(preset_name, vm_id)

1.2.3 (2019-05-10)

  • bugfix: panel - fail to load due jinja error

1.2.1 (2019-05-09)

  • bugfix: update runtime data after manage even if unlocked

  • bugfix: panel - deterministic preset order

1.2.0 (2018-08-07)

  • feature: api - added list_presets

1.1.0 (2018-06-13)

  • feature: adjustable timeout for http requests to iaas

1.0.1 (2018-06-08)

  • bugfix: API should use last cycle data

  • bugfix: OpenStackDriver refresh token

1.0.0 (2018-06-06)

  • preserve preset state (dashboard shows data from last cycle)

  • bump AsyncOpenStackClient (0.6.2)

  • docs

0.7.7 (2018-05-10)

  • bump AsyncOpenStackClient (0.5.2)

0.7.6 (2018-05-10)

  • bump AsyncOpenStackClient (0.5.1)

0.7.5 (2018-04-09)

  • runtime driver abstract fix

0.7.4 (2018-03-22)

  • cleanup, verify

0.7.3 (2018-03-22)

  • bump AsyncOpenStackClient (+ compat)

0.7.2 (2018-03-22)

  • user-data base64

0.7.1 (2018-03-21)

  • multiple network interfaces

0.7.0 (2018-03-20)

  • AsyncOpenStackClient

0.6.0 (2018-02-01)

  • web panel

  • rpc api

0.0.3 (2018-02-01)

  • initial

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vmshepherd-1.6.4.tar.gz (375.4 kB view hashes)

Uploaded Source

Built Distribution

vmshepherd-1.6.4-py2.py3-none-any.whl (274.6 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page