Skip to main content

Our KVM solution, clulstered and self hosted

Project description

ourkvm

Our KVM solution. Cluster, API and local tools all in one.

  • Flake8 Checks
  • Bandit Security Checks
  • mypy Type Checks
  • PyPi Release Upload
  • pytest Checks

What is ourkvm?

There are four overall components:

Product Description
API A FastAPI backend that browsers can talk to
Cluster The API optionally supports enabling cluster services
CLI Tool A CLI tool that can produce virtual machines, report health and run cluster agents
Library A Python library which the above uses to perform their tasks

Demo of Usage

Creating a machine, then starting it, attaching to the --serial device, snapshotting the machine with --snapshot and stopping it:

!video

API

The API is a rest API enabled by running python -m ourkvm --api
The API is built using FastAPI.

The API requires authentication and uses OpenID connect and JWT for SSO.
This is done using fastapi_resource_server. It's tested against Keycloak using a custom realm, users and roles/groups to isolate permissions.

Cluster

The cluster communicates on port 8050 using JWT and standard sockets. For documentation on the protocol, see docs/cluster for more information. To register a cluster agent, simply run the CLI tool with the parameter --cluster-agent.

CLI Tool

The library ships with a Python module that can produce qemu strings that you can use to launch a machine.
It can create local resources such as Qemu disk images, Virtual Machine templates, configuration and .service files.

These .service files that the module generates, can be started with systemctl --user start machineX.service which is described below.

Creating a local virtual machine

$ python -m ourkvm \
    --machine-name testmachine \
    --namespace testmachine \
    --memory 4096 \
    --harddrives ./testimg.qcow2:20G,./testlarge.qcow2:40G \
    --cdroms ~/archiso/out/*.iso \
    --service /etc/systemd/system/ \
    --config /etc/qemu.d/

The following will create a minimal virtual machine using NAT for networking, headless operation meant to be started with systemctl start testmachine.service.

Stopping a machine

$ sudo systemctl stop testmachine.service

Using the above example service of testmachine.service, the service will trigger python -m ourkvm --machine-name testmachine --stop which will attach to the Qemu QMP socket at /tmp/testmachine.qmp and execute a poweroff (followed by qemu-quit after a grace period if the machine has not yet powered off).

Adding custom networking

$ python -m ourkvm \
    --machine-name testmachine \
    --namespace testmachine \
    --memory 4096 \
    --harddrives ./testimg.qcow2:20G,./testlarge.qcow2:40G \
    --cdroms ~/archiso/out/*.iso \
    --service /etc/systemd/system/ \
    --config /etc/qemu.d/ \
    --network '[ {  "type": "tap",  "name": "tap0",  "bridge": "ns_br0",  "namespace": {"from": null, "to": true},  "attach": true}, {  "type": "veth",  "name": "vens0",  "bridge": "test_bridge",  "namespace": {"from": null, "to": true},  "veth_pair": "vens0_ns" }, {  "type": "veth",  "name": "vens0_ns",  "bridge": "ns_br0",  "namespace": {"from": null, "to": true},  "veth_pair": "vens0",  "mac": "fe:00:00:00:00:01" }]'

Adding to the previous example, this will add networking according to the following JSON layout:

[
    {
        "type": "tap",
        "name": "tap0",
        "bridge": "ns_br0",
        "namespace": {"from": null, "to": true},
        "attach": true
    },
    {
        "type": "veth",
        "name": "vens0",
        "bridge": "test_bridge",
        "namespace": {"from": null, "to": null},
        "veth_pair": "vens0_ns"
    },
    {
        "type": "veth",
        "name": "vens0_ns",
        "bridge": "ns_br0",
        "namespace": {"from": null, "to": true},
        "veth_pair": "vens0",
        "mac": "fe:00:00:00:00:01"
    }
]

This creates several network components. Beginning from the top of the JSON file (but backwards logically):

  1. A tap0 interface attached to the virtual machine
  2. A bridge ns_br0 connecting tap0 to it
  3. Moving the above two interfaces into a namespace called testmachine
  4. Creating a veth-pair of vens0<-->vens0_ns
  5. Creating a bridge called test_bridge
  6. Adding vens0 to the test_bridge
  7. Moving vens0_ns into the namespace testmachine
  8. Sets MAC address FE:00:00:00:00:01 to interface vens0_ns upon VM startup

Creating a network chain that looks like the following:
[host] test_bridge <--> vens0--|--vens0_ns <--> ns_br0 <--> tap0 [vm].
The API will take care of creating the elaborate network infrastructure, but the CLI does the work for now.

Note: the namespace declaration in the network struct is True and will automatically get converted to the --namespace definition. Specific namespace names can be supplied here instead if the VM are to connect between multiple namespaces using bridges or veth interfaces.

note: MAC addresses will be auto-generated for tap0 and vens0 in the above example upon creation

Contributing

We use tabs over spaces. We follow pep8 to some extent using Flake8 (see .flake8 for exceptions). We follow strict typing using mypy with the --strict parameter and we require every function to have a associated pytest function under /tests/.

We welcome PR's on any addition/change. They might not all make it, but we develop straight against main which is our master branch. Occational vX.y.z-dev branch might appear to fix an older version while a major release is being on the way. PR's will not be merged until the three GitHub workflows (flake8, mypy and pytest) have completed successfully.

Help

Feel free to open a issue if you think it's a bug or you want to suggest an improvement.

Discussions

Open a discussion on a topic you believe is relevant to discuss or talk about surrounding ourkvm, if it doesn't fit the #help section.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

python-ourkvm-0.0.24.tar.gz (46.2 kB view details)

Uploaded Source

Built Distribution

python_ourkvm-0.0.24-py2.py3-none-any.whl (56.0 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file python-ourkvm-0.0.24.tar.gz.

File metadata

  • Download URL: python-ourkvm-0.0.24.tar.gz
  • Upload date:
  • Size: 46.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.10

File hashes

Hashes for python-ourkvm-0.0.24.tar.gz
Algorithm Hash digest
SHA256 f059a2a6675203f7c64872f2b5a97345acf8110456bdac4eed3990ec5199855a
MD5 8e9af693fcf7f4dde5e2f411af84e9ee
BLAKE2b-256 c4d634ed45686fa079299de64bd7555359658727ba0905525de19e45a43078f4

See more details on using hashes here.

File details

Details for the file python_ourkvm-0.0.24-py2.py3-none-any.whl.

File metadata

  • Download URL: python_ourkvm-0.0.24-py2.py3-none-any.whl
  • Upload date:
  • Size: 56.0 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.10

File hashes

Hashes for python_ourkvm-0.0.24-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 3c357fcdd6a9c9098b56d1eca41001e77d199cd4cea3f5d8df8a147d211835d0
MD5 a7c8757da471401089c32c810c4d5995
BLAKE2b-256 b1b6152f6deb149f220cbebf12492469c5959fe56d74833d0b311c854d227b74

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page