gitlab-runner-tart-driver 0.1.3

No project description provided

Gitlab Runner Custom Tart Driver

About

Tart is a virtualization toolset to build, run and manage macOS and Linux virtual machines on Apple Silicon. Tart is using Apple’s native Virtualization.Framework that was developed along with architecting the first M1 chip. This seamless integration between hardware and software ensures smooth performance without any drawbacks.

For storing virtual machine images Tart integrates with OCI-compatible container registries. This allows you to work with virtual machines as you used to with Docker containers.

This Custom GitLab Runner Extension integrates Tart semlessly into the GitLabCI ecosystem with similar functionality as the well docker-runner

Executor	Tart	Docker
Clean build environment for every build	✓	✓
Reuse previous clone if it exists	✓	✓
Runner file system access protected	✓	✓
Migrate runner machine	?	✓
Zero-configuration support for concurrent builds	✓	✓
Complicated build environments	✗	✓
Debugging build problems	easy	medium

Setup

ATTENTION This setup will only work on an Apple Silicon based machine

To setup the gitlab-runner-tart-driver you will need to go through the following steps

1. Install Homebrew
2. Install python3
3. Install tart
4. Install gitlab-runner
5. Install gitlab-runner-tart-driver
6. Configure and register a custom gitlab-runner with your GitLab Instance
7. Update config.toml

To install Homebrew simply execute the command

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Next we need to install tart by executing

brew install cirruslabs/cli/tart

We can verify to virtualization functionality by pulling an image for the official tart managed images

tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base
tart run ventura-base

After the image has been downloaded you will see a window come up with a virtualized OSX Ventura running

To run the gitlab-runner-tart-driver we will need a valid python3 installation as well as the gitlab-runner

brew install python3 gitlab-runner

Finally we will need to install gitlab-runner-tart-driver.

pip install gitlab-runner-tart-driver

GitLab Runner Configuration

Follow the guide Install GitLab Runner on macOS.

1. Register your GitLab Runner and use the custom executor

gitlab-runner register

2. Install the service with gitlab-runner install

ATTENTION The GitLab Runner currently fails to properly setup the LaunchAgent. The workaround described can be omitted as soon the the Bug has been resolved

# Fix missing log path issue
sudo mkdir -p /usr/local/var/log
sudo chmod a+rw /usr/local/var/log

gitlab-runner uninstall
gitlab-runner install
gitlab-runner start

3. Configure the config.toml for the tart custom driver

The config.toml should be located at ~/.gitlab-runner/config.toml. Open it and add the following content

config.toml

[runners.custom]
    config_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
    config_args = [ "config", '-x', '/opt/homebrew/bin/tart' ]
    config_exec_timeout = 200

    prepare_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
    prepare_args = [ "prepare", '-x', '/opt/homebrew/bin/tart' ]
    prepare_exec_timeout = 200

    run_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
    run_args = [ "run", '-x', '/opt/homebrew/bin/tart' ]

    cleanup_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
    cleanup_args = [ "cleanup", '-x', '/opt/homebrew/bin/tart' ]
    cleanup_exec_timeout = 200

Now restart the gitlab-runner with gitlab-runner restart

GitLab CI

One of the great advantages in using Tart is that it gives almost all functionality to we are used from the standard docker executors allowing us to write GitLabCI piplines that do not need any or minimal adaptions for OSX.

ATTENTION: The driver will always automatically login to your GitLab Registry using CI_REGISTRY_USER, CI_REGISTRY_PASSWORD and CI_REGISTRY

stages:
 - build

run-on-tart:
  image: ghcr.io/cirruslabs/macos-ventura-base:latest
  stage:
    - build
  tags:
    - tart
  before_script:
    - brew install gitlab-runner # you will need to make sure `gitlab-runner` executable is available to be able to upload artifacts
  script:
    - echo "This is brought to you by tart" >> artifact.txt
  artifacts:
    paths:
      - artifact.txt

Private OCI Registries

Oftentimes you might want to provide your own OCI-compliant images created with Packer and the official Tart Builder. If you push your images to a private registry you will need to provide the credentials for the gitlab-runner-tart-driver to login there first.

You can provide a default login registry using the CLI parameters (see Command prepare) but also provide the credentials from with each job variable definition

stages:
 - build

variables:
    TART_REGISTRY_USERNAME: [e-mail]
    TART_REGISTRY_PASSWORD: <some_secret>
    TART_REGISTRY: private.registry.io

job1:
  image: private.registry.io/tart/xcode14:latest
  stage:
    - build
  tags:
    - tart
  before_script:
    - brew install gitlab-runner # you will need to make sure `gitlab-runner` executable is available to be able to upload artifacts
  script:
    - echo "This is brought to you by tart" >> artifact.txt
  artifacts:
    paths:
      - artifact.txt

job2:
  image: private.other.io/tart/xcode14:latest
  variables:
    # override the OCI login information for this job only
    TART_REGISTRY_USERNAME: [e-mail]
    TART_REGISTRY_PASSWORD: <some_secret>
    TART_REGISTRY: private.other.io
  stage:
    - build
  tags:
    - tart
  before_script:
    - brew install gitlab-runner # you will need to make sure `gitlab-runner` executable is available to be able to upload artifacts
  script:
    - echo "This is brought to you by tart" >> artifact.txt
  artifacts:
    paths:
      - artifact.txt

Configuring the SSH Credentials

The Tart managed images come with a default user admin and password admin which will be used to establish an ssh connection. If you decide to provide your own images, you might opt for a different user and password. You can provide the default user and password using the config.toml within the run section using --default-ssh-user and --default-ssh-password or provide it from within your job

job2:
  image: private.other.io/tart/xcode14:latest
  variables:
    # override the OCI login information for this job only
    TART_REGISTRY_USERNAME: [e-mail]
    TART_REGISTRY_PASSWORD: <some_secret>
    TART_REGISTRY: private.other.io
    TART_SSH_USERNAME: gitlab
    TART_SSH_PASSWORD: gitlab
  stage:
    - build
  tags:
    - tart
  before_script:
    - brew install gitlab-runner # you will need to make sure `gitlab-runner` executable is available to be able to upload artifacts
  script:
    - echo "This is brought to you by tart" >> artifact.txt
  artifacts:
    paths:
      - artifact.txt

Advanced Configuration

Concurrency

Currently tart only supports two executions of VMs at the same time. This limits the scalability of the solution on a single host. Please ensure you are setting the concurrent setting to a maximum of 2 in your config.toml.

concurrent = 2 # <-- ATTENTION: ensure not sure go higher than '2'
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "macos-tart-driver1"
  url = "https://gitlab.com/"
  id = 11111111
  token = "XXXXXXXXXXXX"
  token_obtained_at = 2023-04-03T07:21:42Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "custom"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.custom]
    config_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
    config_args = [ "config" ]

    prepare_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
    prepare_args = [ "prepare", '-x', '/opt/homebrew/bin/tart', '--concurrency', '2', '--auto-resources' ]

    run_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
    run_args = [ "run", '-x', '/opt/homebrew/bin/tart' ]

    cleanup_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
    cleanup_args = [ "cleanup", '-x', '/opt/homebrew/bin/tart' ]

Host Cache and Builds directories

With tart it is possible to mount local directories into the VM. Mounting local directories brings a number of benefits like up tp 30% higher IO performance as well as a correct caching mechanism. To enable Host-local caching and builds, simply pass --builds-dir and --cache-dir to the prepare command

prepare_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
prepare_args = [ "prepare", '-x', '/opt/homebrew/bin/tart', '--concurrency', '2', '--auto-resources', '--cache-dir', '/Users/gitlab/gitlab-runner/cache', '--builds-dir', '/Users/gitlab/gitlab-runner/builds']

Volume Mounts

Just like with docker the gitlab-runner-tart-driver allows you to mount any arbitrary host path into your VM. This can be especially useful if you have to mount runner-local resources like shared test-data or additional caches into the executor. You can follow the standard docker syntax for volume mounts --volume /my/local/dir:/opt/remote/dir. Additionally you can also pass the ro flag to make the mount read-only i.e. --volume /my/local/dir:/opt/remote/dir:ro.

[runners.custom]
  config_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
  config_args = [ "config" ]

  prepare_exec = "/opt/homebrew/bin/gitlab-runner-tart-driver"
  prepare_args = [ "prepare",
    '-x', '/opt/homebrew/bin/tart',
    '--concurrency', '2',
    '--auto-resources',
    '--volume /data/models/:/opt/data/models:ro']

Auto Host Resource Distribution

tart images come with a pre-defined number of CPUs and Memory allocation. Typical numbers for default images are cpu_count=4 and memory=8192. With the concurrency limitation of two VMs/images running at the same time this might not utilize your host system completely. Per default, --auto-resources is enable for the gitlab-runner-tart-driver which will split the host resources equally to the VMs defined by --concurrency. The default concurrency is 1 and therefore will assign all host resources to the VM.

ATTENTION make sure your gitlab-runner concurrency setting is the same as the --concurrency parameter you are passing to the prepare command

Custom shell

You can use a custom shell to execute your commands. The default ist /bin/zsh

see Command run

Custom pull_policy

You can use a custom pull_policy. The default policy is if-not-present

see Command prepare

CLI Parameters for config.toml

The gitlab-runner-tart-driver gives a number of advanced configuration options. Use gitlab-runner-tart-driver [stage] --help to get a full list of options that you can pass to the exectuable using your config.toml file.

Usage: gitlab-runner-tart-driver [OPTIONS] COMMAND [ARGS]...

Options:
  -v, --version  Show the version and exit.
  --help         Show this message and exit.

Commands:
  cleanup  Command to greet a user.
  config   Implementation of the CONFIG stage of the Custom Executor.
  prepare  Prepare the environment and start the tart VM.
  run      Run commands.

Command config

Usage: gitlab-runner-tart-driver config [OPTIONS]

  Implementation of the CONFIG stage of the Custom Executor. Details on how to
  use this command can be found at
  https://docs.gitlab.com/runner/executors/custom.html#config.

Options:
  --help                      Show this message and exit.

Command prepare

Usage: gitlab-runner-tart-driver prepare [OPTIONS]

  Prepare the environment and start the tart VM.

Options:
  --default-ssh-username TEXT     username to login to a tart vm
  --default-ssh-password TEXT     password to login to a tart vm
  --pull-policy [always|if-not-present|never]
                                  define how runners pull tart images from
                                  registries
  --registry-username TEXT        username to login to a oci registry
  --registry-password TEXT        password to login to a oci registry
  --registry TEXT                 username to login to a oci registry
  --cpu INTEGER                   Number of CPUs associated to VM
  --memory INTEGER                VM memory size in megabytes associated to VM
  --display TEXT                  VM display resolution in a format of
                                  <width>x<height>. For example, 1200x800
  --auto-resources / --no-auto-resources
                                  If enabled, the driver will divide system
                                  resources equally to the concurrent VMs.
  --concurrency INTEGER           Number of concurrent processes that are
                                  supported. ATTENTION tart currently only
                                  support two concurrent VMs
  --cache-dir TEXT                Caching dir to be used.
  --builds-dir TEXT               Path to the builds directory.
  --timeout INTEGER               Timeout in seconds for the VM to be
                                  reachable via SSH.
  --volume TEXT                   Volume mount definition with docker syntax.
                                  <host_dir>:<vm_dir>[:ro]
  -x, --tart-executable TEXT      Path to the tart executable.
  --help                          Show this message and exit.                      Show this message and exit.

Command run

Usage: gitlab-runner-tart-driver run [OPTIONS] SCRIPT STAGE

  Run commands.

Options:
  --default-ssh-username TEXT  username to login to a tart vm
  --default-ssh-password TEXT  password to login to a tart vm
  -x, --tart-executable TEXT   Path to the tart executable.
  --shell TEXT                 Path to the shell to be used for commands over
                               ssh.
  --help                       Show this message and exit.

Command cleanup

Usage: gitlab-runner-tart-driver cleanup [OPTIONS]

  Command to greet a user.

Options:
  -x, --tart-executable TEXT  Path to the tart executable.
  --help                      Show this message and exit.