sardine cake - managing libvirt via OCI registries

this is a framework inspired by tart for sharing libvirt VMs via OCI-compatible container registries.

the user-facing program shall be called sarde (or sard?), as it sounds similar to tart.

the "sardine" as inspired by the libvirt logo, the "sardine cake" (or fishcake) is some kind of gross-ish (i think, but idk) tarte.

(quiche would have been a nice bow to QEMU, but that is already taken. sardine itself is taken as well)

Motivation

The main motivation for sarde comes from running isolated Continuous Integration (CI) jobs within Virtual Machines (VMs).

When setting up such a system (e.g. using GitLab-CI) you quickly notice that you need a way to distribute VM images to the various runners, so that they all use the same VMs. Ideally the same mechanism can be used for the CI-users to specify the image to run their jobs on.

Most CI jobs run in isolated Linux containers via docker, that solves this issue very nicely.

Unfortunately, this doesn't work so well for non-Linux containers. AFAIK, even when running docker on macOS or Windows, inside the containers you are still on Linux.

For macOS (on Apple Silicon), Cirrus Labs have released their fantastic tart toolset, which works very similar to docker but instead uses full-fledged VMs. I especially like, that they adopted the Open Container Initiative (OCI) approach for distributing VM images (OCI evolved from Docker, so Docker registries are a subset of OCI registries. The differences are so small that many registry providers to be used with Docker can be used as arbitrary OCI compliant registries).

For Windows, I'm not aware of a similar solution. This is where sarde comes into play. It tries to be as general as possible, building on top of libvirt. (So it should be usable with all the virtualization solutions supported by libvirt. In practice we only test qemu/kvm, though).

As such, sarde is not concerned with running VMs: this is already handled well enough by libvirt.

Usage

Using VMs

1. (login to a registry for read permissions with sarde login (once))
2. fetch the VM with sarde pull
3. import the VM with sarde import
4. start the VM

export LIBVIRT_DEFAULT_URI=qemu:///system
sarde login registry.example.org
sarde fetch registry.example.org/sardines/debian12:latest
name=$(sarde import registry.example.org/sardines/debian12:latest)
virsh start "$(name)"

As can be seen sarde import returns the name of the created VM, so you easily start the VM afterwards.

For convenience, the pull command combines fetch and import. So mostly, you should be able to simply do:

name=$(sarde pull registry.example.org/sardines/debian12:latest)
virsh start "$(name)"

notes

By default, sarde import (and thus sarde pull) generates a unique VM name, based entirely on the VM data (disks and configuration). This is nice in automated environments (where you cannot manually intervene to guarantee uniqueness of the name), but comes at the expense of names that are not human friendly.

Creating new VMs

1. Prepare your VM in libvirt.
2. export the VM with sarde export
3. (login to a registry for write permissions with sarde login (once))
4. publish the VM with sarde push

export LIBVIRT_DEFAULT_URI=qemu:///system
sarde login registry.example.org
sarde export --tag registry.example.org/sardines/debian12:latest debian12
sarde push registry.example.org/sardines/debian12:latest

restrictions

You probably should only export VMs that are shut down.

When using copy-on-write thin copies for disk images, make sure that

• they live in the same directory as their backing files (all of them)
• the backing images are references via relative paths (all of them)

You can check whether these requirements are met by using

disk="mydisk.qcow2

! qemu-img info --backing-chain "${disk}" \
	| egrep "^backing file:" \
	| sed -e 's|\s*(actual path: .*)||' \
	| grep "/" \
|| (echo "disk image must use relative paths!"; false)

(LATER this restriction might eventually be resolved by sarde automatically converting the backing-chain).

Configuration

You can provide a default configuration in ~/.config/sardinecake/sarde.ini and /etc/sardinecake/sarde.ini. You can provide different configurations for each sub-command (by using a section named like the subcommand), though in practice this is probably of little use, as all sub-commands inherit the configuration from the [DEFAULT] section.

They keys in the configuration files are the same as the long option names without the leading double-dash. So sarde list --connect qemu:///system is equivalent to using

[list]
connect = qemu:///system

You can provide an alternative configuration file by setting the SARDE_CONF_FILES environment variable to a list of colon-separated filenames:

export SARDE_CONF_FILES=mysarde.ini
sarde list

See examples/sarde.ini

gitlab-runner integration

The gitlab-sardinecake-executor can be used for running GitLab-CI jobs within VMs managed by sardinecake.

Configuration

gitlab-runner

Tell gitlab-runner to use sardinecake as the custom executor:

[[runners]]
  # ...
  builds_dir = "/home/vagrant/build"
  cache_dir = "/home/vagrant/cache"
  executor = "custom"

  [runners.custom]
    config_exec = "gitlab-sardinecake-executor"
    config_args = [ "config" ]
    prepare_exec = "gitlab-sardinecake-executor"
    prepare_args = [ "prepare" ]
    run_exec = "gitlab-sardinecake-executor"
    run_args = [ "run" ]
    cleanup_exec = "gitlab-sardinecake-executor"
    cleanup_args = [ "cleanup" ]

gitlab-sardinecake-executor

gitlab-sardinecake-executor looks for INI-style configuration files in

• /etc/gitlab-runner/gitlab-sardinecake-executor.ini
• ~/.config/gitlab-runner/gitlab-sardinecake-executor.ini (in the home directory of the user running gitlab-sardinecake-executor)

[DEFAULT]
# how to connect to the sardinecake daemon
connect = qemu:///system
# where the downloaded OCI containers are stored
ocidir = /var/lib/sardinecake/OCI/
# where the extracted VMs are stored
vmdir = /var/lib/sardinecake/VMs/

See examples/gitlab-sardinecake-executor.ini

gitlab-ci.yml

Now you can use libvirt VMs in your .gitlab-ci.yml:

# You can use any libvirt VM the executor knows about
image: registry.example.org/sardines/debian12:latest

test:
  # In case you tagged runners that have
  # GitLab Libvirt Executor configured on them
  tags:
    - sardinecake

  script:
    - uname -a

Caveats

sardinecake VMs can be rather large. E.g. a Windows VM with various MSYS2/MinGW flavours installed can easily require a disk-image that takes 60GB. In the sardinecake OCI containers, these disk images are stored compressed (about ½ the size).

Nevertheless, when cloning a VM for the first time, the data has to be downloaded and uncompressed, which may take a while.

Both the OCI containers and VMs created from the containers are kept on disk, so future uses should bring up the VM very fast (within seconds).