Skip to main content

🧸 smol-k8s-lab

PyPI - Version PyPI - Downloads

A terminal based tool to install slimmer k8s distros on metal, with batteries included!

Screenshot of the smol-k8s-lab help menu

Features

  • Deploys Argo CD by default, so you can manage your entire lab using files in open source git repos
    • Argo CD ships with a dashboard with a custom theme 💙
    • Argo CD is default secured with a self hosted OIDC provider, Zitadel
  • Uses a local password manager of your choice. Currently supported:
    • Bitwarden
    • KeyPass
    • macOS keychain
    • Freedesktop Secret Service supports many DE including GNOME (requires secretstorage)
    • KDE4 & KDE5 KWallet (requires dbus)
  • External Secerts management for your sensitive Kubernetes Secret values. Currently supported:
  • Manages all your authentication needs centrally using Zitadel, a selfed hosted OIDC provider
  • Supports initialization on a range of common self-hosted apps 📱
  • Lots o' docs
  • commitment to never monetizing this app. It will always be AGPL licensed and free 💙
  • No AI or LLMs "features" and No AI contribution policy

Installation

🐝 sure to check out our full installation guide, but the gist of it is smol-k8s-lab can be installed via pipx (or brew coming soon).

pipx

smol-k8s-lab requires Python 3.12+ (and pipx). If you've already got both and other pre-reqs, you should be able to:

# install the CLI
pipx install smol-k8s-lab

# Check the help menu before proceeding
smol-k8s-lab --help

brew (still unstable)

brew is for macOS/Debian/Ubuntu, as this will also install any non-python prerequisites you need, so you don't need to worry about them. This method is new, so please let us know if anything isn't working for you.

# tap the special homebrew repo for our formula and install it
brew install open-engineering/tap/smol-k8s-lab

Then you should be able to check the version and cli options with:

smol-k8s-lab --help

Output of smol-k8s-lab --help after cloning the directory and installing the prerequisites.

Checkout our TUI docs for more info on how to get started playing with smol-k8s-lab :-)

Usage

Initialization

After you've followed the installation instructions, if you're new to smol-k8s-lab, initialize a new config file:

# we'll walk you through any configuration needed before
# saving the config and deploying it for you
smol-k8s-lab
Upgrading to 11.0.0

v11.0.0 introduces the concept of environments and also drops support for the now deprecated Ingress Nginx and starts using Traefik with Gateway API. This is important for people running smol-k8s-lab for their home labs or running it as a production system for small orgs.

A major difference is that your config directory $XDG_CONFIG_HOME/smol-k8s-lab/ is now split up into a couple different directories. For simplicity's sake, we'll refer to $XDG_CONFIG_HOME/smol-k8s-lab/ as your config directory, but note that this defaults to ~/.config/smol-k8s-lab unless you have set the $XDG_CONFIG_HOME env var to be something else.

Your name config directory $XDG_CONFIG_HOME/smol-k8s-lab/apps file no longer contains your apps. Instead, all of your apps now live in $XDG_CONFIG_HOME/smol-k8s-lab/apps/$ENV with files for each app in that environment.

The following apps are considered core apps and should live in $XDG_CONFIG_HOME/smol-k8s-lab/apps/core as files of that name:

  • alloy
  • argocd
  • autoscaler
  • certmanager
  • cilium
  • cnpg_operator
  • external_secrets_operator
  • k8tz
  • k8up
  • kyverno
  • longhorn
  • openbao
  • pxc_operator
  • snapshot_controller
  • traefik
  • velero

These files are optionally included in core and can also live in other envs as well:

  • grafana_stack
  • zitadel

Zitadel is a special app, in that where ever it is deployed, it will be used for OIDC apps in that environment only. Everything in the core apps, should behave as they did before with no major changes. If you were only using smol-k8s-lab to boostrap a cluster and nothing else, then from here, all you need to do is remove ingress nginx, and rerun smol-k8s-lab with traefik enabled (and set apps_global_config.ingressClass and apps_global_config.gatewayClassName to traefik in your config.yaml file).

Everything else should live in a directory of that environment name, so for instance, if you'd like an environment called dev-my-org, you'd create a directory called $XDG_CONFIG_HOME/smol-k8s-lab/apps/dev-my-org.

To create a new environment, you can run smol-k8s-lab -n and we'll walk you through setting up a new environment. You can also launch the TUI and on the apps config screen, check the Add New Env check box and we'll walk you through that as well.

This release also introduces OpenBao, a fork of Vault, as the default External Secrets provider, rather than using the Argo CD ApplicationSet Secrets Plugin or Bitwarden. The ApplicationSet Secrets plugin was never encrypted and not intended to be used for sensitive values, while Bitwarden's company may not remain open source forever, so we're making the decision to quickly adopt OpenBao instead which can do both sensitive and insensitive values as well as much much more when it comes to security.

Because of this, while the smol-k8s-lab out of the box experience will be streamlined and straight forward, upgrading to 11.x from 10.x is going to be a bit of a chore. If you don't care about anything currently managed by smol-k8s-lab, you can just delete your cluster and wipe your $XDG_CONFIG_HOME/smol-k8s-lab to start fresh. Then you can use the TUI to setup your new values, or you can just run smol-k8s-lab and we'll install all our default apps.

If you do care about your current deployed apps:

  1. We recommend backing up EVERYTHING, including both all your config files, but also, take restic backups using k8up of all your current PVCs and verify they are working and available in your remote S3 buckets.

  2. You'll need to also schedule down time so that any users know that you'll be offline during the migration as you'll need to take down all your non-core apps. When you're good to take them all down AFTER you've verified your backups are working, you can proceed to the next step.

  3. Create a directory for your new environment, so let's say that's called prod, you'd create a directory called $XDG_CONFIG_HOME/smol-k8s-lab/apps/prod and move all your non-core apps into it (e.g. home assistant, mastodon, peertube, etc). At this time, you can take down your existing apps (completely delete them from Kubernetes).

  4. Set apps_global_config.external_secrets to openbao and apps_global_config.migrate_to_openbao to true in your config.yaml file, and then set the following in each of your apps: restore.enabled: true. Then, you should be able to run smol-k8s-lab as normal and all should be well!

Upgrading from 9.x to 10.x

This update officially introduces the beginning of our smol-schema, meaning we stop using using the appset secret plugin officially, and start instead using targeted helm charts of Argo CD Applications. We also stop overloading the init.values of every app, and instead use targeted parameters. This means instead of e.g. for zitadel's admin and smtp section:

init:
  values:
    # admin info
    admin_user: ""
    admin_email: ""
    # smtp info
    smtp_host: ""
    smtp_user: ""
    smtp_port: ""
    smtp_protocol: "SMTP"
    # smtp from email (verified sender)
    smtp_from_address: example.com
    smtp_password:
      value_from:
        env: GHOST_SMTP_PASSWORD

You would provide the following in your $XDG_CONFIG_HOME/smol-k8s-lab/apps/zitadel.yaml:

# note that this is at the top level of the yaml
admin:
  username: ""
  email: ""

# note that this is at the top level of the yaml
smtp:
  # smtp info
  host: ""
  username: ""
  port: ""
  protocol: "SMTP"
  # smtp from email (verified sender)
  from_address: example.com
  password:
    value_from:
      env: GHOST_SMTP_PASSWORD

Likewise, for databases, you'd provide:

# note that this is at the top level of the yaml
db:
  # postgresql is the only supported app for most apps, but sometimes we support mysql like in the case of ghost and writefreely
  type: postgresql
  # this is your database pvc capacity
  capacity: 5Gi

For affinity and tolerations:

# note that these are at the top level of the yaml
affinity: {}
tolerations: []

PVCs now also get their own parameters like this:

# note this is a top level parameter
pvc:
  # this is for an example PVC named ghost
  ghost:
    # this enables it
    enabled: true
    # this is the ghost pvc capacity
    capacity: 10Gi

Valkey can be enabled or disabled with:

# note this is a top level parameter
valkey:
  # enable valkey
  enabled: true

for hostnames, those are also declared at the top level:

hostnames:
  zitadel: "iam.example.com"
Upgrading from 8.5.1 to 9.x

Changes how we provide prometheus CRDs and alloy:

  • alloy is now its own app and prometheus_crds are not their own app, instead you can set alloy's syncWaves to 1 to accomplish this
  • alloy is no longer part of the grafana stack, so you can deploy it separately if needed
Upgrading from 8.5.0 to 8.5.1

fix default config to call pvc_storage_class storage_class instead: 5c4f31eb3f

Your config should now read something like this:

# anything here gets set for all apps if you're using our default repos
apps_global_config:
  # setting this changes all the below domains to use the following cluster_issuer
  # change to letsencrypt-prod when you're ready to go live with your infra
  cluster_issuer: "letsencrypt-staging"
  # change to your tz: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
  time_zone: "Europe/Amsterdam"
  # always deploy extnernal secrets. *Must be a string of either:
  # ""           - don't use external secrets
  # "bitwarden"  - use bitwarden for external secrets
  # "openbao"    - (coming soon!) use openbao (a fork of vault) for external secrets
  external_secrets: "bitwarden"
  # the default storage class you want to use for the creation of all Persistent Volume claims
  # it defaults to local-path, but you can set it to longhorn or whatever you'd like :)
  # for storage classes see:
  storage_class: "local-path"
Upgrading from 8.4.x to 8.5.0
  • changed how we determine if we should install the appset secret plugin: 04514b1066

You now need to pass the following into your argocd app's config:

  argocd:
    # set this to true if you want to use the appset secret plugin (defaults to false)
    enable_appset_secret_plugin: true
Upgrading from 7.x to 8.4.x
  • This update begins removing ALL references to our previous GitHub repos. Please instead point to our new codeberg.org/open-engineering org for all Argo CD Applications and any helm charts of ours you used.
  • We switched to the Grafana community helm chart repo. You'll want this for any future community grafana updates
  • bumped Kubernetes and Ruamel lib versions major versions to latest.
Upgrading from 6.x to 7.x
  • This is where we officially switch from Thanos to Mimir and start supporting Tempo for self hosted tracing.
    • we don't offer a migration path from thanos unfortunately. We recommend re-deploying the grafana_stack
  • We also start supporting sensitive values for jellyfin.
Upgrading from 5.x to 6.x

A major difference is that your config directory $XDG_CONFIG_HOME/smol-k8s-lab/ is now split up into a couple different directories. For simplicity's sake, we'll refer to $XDG_CONFIG_HOME/smol-k8s-lab/ as your config directory, but note that this defaults to ~/.config/smol-k8s-lab unless you have set the $XDG_CONFIG_HOME env var to be something else.

Your name config file $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml file no longer contains your apps at all. Instead, all of your apps now live in $XDG_CONFIG_HOME/smol-k8s-lab/apps and within that directory, you'll files for each app in that environment. The apps parameter in your $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml will no longer be used. Instead, take each app's config parameters and move them to a file by that name in your apps directory, so for example, this is what your $XDG_CONFIG_HOME/smol-k8s-lab/apps/zitadel.yaml should look like, in order to deploy zitadel:

zitadel.yaml contents
```yaml
enabled: true
description: |
  [link=https://zitadel.com/opensource]ZITADEL[/link] is an open source self hosted IAM platform for the cloud era

  smol-k8s-lab supports initialization of:
    - an admin service account
    - a human admin user (including an autogenerated password)
    - a project with a name of your chosing
    - 2 OIDC applications for Argo CD and Vouch
    - 2 Argo CD groups (admins and users), 1 vouch groups
    - groupsClaim action to enforce group roles on authentication
    - updates your appset_secret_plugin secret and refreshes the pod

  The default app will also deploy SeaweedFS to backup your database which in turn is backed up to a remote s3 provider of your choice.

  To provide sensitive values via environment variables to smol-k8s-lab use:
    - ZITADEL_S3_BACKUP_ACCESS_ID
    - ZITADEL_S3_BACKUP_SECRET_KEY
    - ZITADEL_RESTIC_REPO_PASSWORD
init:
  # Switch to false if you don't want to create intial secrets or use the
  # API via a service acocunt to create the above described resources
  enabled: true
  restore:
    # set to true to run a restic restore via a k8up job for seaweedfs PVCs,
    # and restoring your postgresql database
    enabled: false
    # for Cloudnative postgres operator cluster restores only
    cnpg_restore: true
    # the restic snapshot IDs of the PVC snapshots you want to restore
    restic_snapshot_ids:
      # seaweedFS volume pvc snapshot id. if not given, we use the latest
      seaweedfs_volume: ""
      # seaweedFS filer pvc snapshot id. if not given, we use the latest
      seaweedfs_filer: ""
  values:
    # initial human user's usename
    admin_user: ""
    # initial human user's email
    email: ""
    # initial human's first name
    first_name: ""
    # initial human's last name
    last_name: ""
    # options: GENDER_UNSPECIFIED, GENDER_MALE, GENDER_FEMALE, GENDER_DIVERSE
    # more coming soon, see: https://github.com/zitadel/zitadel/issues/6355
    gender: GENDER_UNSPECIFIED
    # name of the default project to create OIDC applications in
    project: core
    # mail server, must include port! e.g. mymailserver.com:587
    smtp_host:
      value_from:
        env: ZITADEL_SMTP_HOST
    # mail user
    smtp_user:
      value_from:
        env: ZITADEL_SMTP_USER
    # mail password
    smtp_password:
      value_from:
        env: ZITADEL_SMTP_PASSWORD
    # mail from address
    smtp_from_address:
      value_from:
        env: ZITADEL_SMTP_FROM_ADDRESS
    # mail from name
    smtp_from_name:
      value_from:
        env: ZITADEL_SMTP_FROM_NAME
    # mail reply to address
    smtp_reply_to_address:
      value_from:
        env: ZITADEL_SMTP_REPLY_TO_ADDRESS
backups:
  # cronjob syntax schedule to run zitadel seaweedfs pvc backups
  pvc_schedule: 10 0 * * *
  # cronjob syntax (with SECONDS field) for zitadel postgres backups
  # must happen at least 10 minutes before pvc backups, to avoid corruption
  # due to missing files. This is because the cnpg backup shows as completed
  # before it actually is, due to the wal archive it lists as it's end not
  # being in the backup yet
  postgres_schedule: 0 0 0 * * *
  s3:
    # these are for pushing remote backups of your local s3 storage, for speed and cost optimization
    endpoint: ""
    bucket: ""
    region: ""
    secret_access_key:
      value_from:
        env: ZITADEL_S3_BACKUP_SECRET_KEY
    access_key_id:
      value_from:
        env: ZITADEL_S3_BACKUP_ACCESS_ID
  restic_repo_password:
    value_from:
      env: ZITADEL_RESTIC_REPO_PASSWORD
argo:
  # secrets keys to make available to Argo CD ApplicationSets
  secret_keys:
    # FQDN to use for zitadel
    hostname: ""
    # set the local s3 provider for zitadel's database backups. can be minio or seaweedfs
    s3_provider: seaweedfs
    # local s3 endpoint for postgresql backups, backed up constantly
    s3_endpoint: ""
    # capacity for the PVC backing your local s3 instance
    s3_pvc_capacity: 2Gi
  # repo to install the Argo CD app from
  # git repo to install the Argo CD app from
  repo: https://codeberg.org/small-hack/argocd-apps
  # path in the argo repo to point to.
  path: "zitadel/app_of_apps/"
  # either the branch or tag to point at in the argo repo above
  revision: main
  # kubernetes cluster to install the k8s app into, defaults to Argo CD default
  cluster: https://kubernetes.default.svc
  # namespace to install the k8s app in
  namespace: "zitadel"
  # recurse directories in the provided git repo
  directory_recursion: false
  # source repos for Argo CD App Project (in addition to argo.repo)
  project:
    name: zitadel
    source_repos:
      - https://charts.zitadel.com
      - https://zitadel.github.io/zitadel-charts
      - https://small-hack.github.io/cloudnative-pg-cluster-chart
      - https://operator.min.io/
      - https://seaweedfs.github.io/seaweedfs/helm
      - https://git.smallhack.org/small-hack/argocd-apps
      - https://codeberg.org/small-hack/argocd-apps
    destination:
      # automatically includes the app's namespace and argocd's namespace
      namespaces: []
```

There are no other major changes except that:

  • we upgrade Textual to a new major version. This is not something you need to worry about unless you forked this repo.
  • we drop support for collabora as a stand alone app (it's now part of nextcloud)
  • we add gotosocial as a default app
Upgrading config from v4.x to v5.x

If you've installed smol-k8s-lab prior to v5.0.0, please backup your old configuration, and then remove the ~/.config/smol-k8s-lab/config.yaml (or $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml) file entirely, then run the following with either pip or pipx:

if using pip:

# this uninstalls the old smol-k8s-lab for python 3.11
pip3.11 uninstall smol-k8s-lab

# this installs smol-k8s-lab for python 3.12
pip3.12 install --upgrade smol-k8s-lab

# this initializes a new configuration
smol-k8s-lab

or if using pipx:

# this upgrades smol-k8s-lab
pipx upgrade smol-k8s-lab

# this initializes a new configuration
smol-k8s-lab

We have done a masive upgrade of the config file. You'll need to update your configs based on the details in https://github.com/open-engineering-nl/smol-k8s-lab/pull/210 . The main changes are to the following (check each doc link for details):

Upgrading config from v3.7.1 to v4.x

If you've installed smol-k8s-lab prior to v4.0.0, please backup your old configuration, and then remove the ~/.config/smol-k8s-lab/config.yaml (or $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml) file entirely, then run the following with either pip or pipx:

if using pip:

# this upgrades smol-k8s-lab
pip3.11 install --upgrade smol-k8s-lab

# this initializes a new configuration
smol-k8s-lab

or if using pipx:

# this upgrades smol-k8s-lab
pipx upgrade smol-k8s-lab

# this initializes a new configuration
smol-k8s-lab

The main breaking changes between v3.7.1 and v4.0.0 are that we now default enable metrics on most apps. Because of this, you need to have the Prometheus ServiceMonitor CRD installed ahead of time. Luckily, we now provide that as an app as well :) If you deleted your config and created a new one, it will already be there, but if you want to reuse your old config, you can add the app like this:

apps:
  prometheus_crds:
    description: |
      [link=https://prometheus.io/docs/introduction/overview/]Prometheus[/link] CRDs to start with.
      You can optionally disable this if you don't want to deploy apps with metrics.

    enabled: true
    argo:
      # secrets keys to make available to Argo CD ApplicationSets
      secret_keys: {}
      # git repo to install the Argo CD app from
      repo: https://codeberrg.org/open-engineering/argocd-apps
      # path in the argo repo to point to. Trailing slash very important!
      path: prometheus/crds/
      # either the branch or tag to point at in the argo repo above
      revision: main
      # namespace to install the k8s app in
      namespace: prometheus
      # recurse directories in the provided git repo
      directory_recursion: false
      # source repos for Argo CD App Project (in addition to argo.repo)
      project:
        name: prometheus
        source_repos:
        - https://github.com/prometheus-community/helm-charts.git
        destination:
          # automatically includes the app's namespace and argocd's namespace
          namespaces:
          - kube-system
          - prometheus

If using the default repos, please also disable directory directory_recursion for:

  • your prometheus stack app
  • zitadel

For all changes, please check out PR #206.

Upgrading config from v2.2.4 to v3.x

If you've installed smol-k8s-lab prior to v3.0.0, please backup your old configuration, and then remove the ~/.config/smol-k8s-lab/config.yaml (or $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml) file entirely, then run the following with either pip or pipx:

if using pip:

# this upgrades smol-k8s-lab
pip3.11 install --upgrade smol-k8s-lab

# this initializes a new configuration
smol-k8s-lab

or if using pipx:

# this upgrades smol-k8s-lab
pipx upgrade smol-k8s-lab

# this initializes a new configuration
smol-k8s-lab

The main breaking changes between v2.2.4 and v3.0 are as follows:

  • home assistant has graduated from demo app to live app

You'll need to change apps.home_assistant.argo.path to either home-assistant/toleration_and_affinity/ if you're using node labels and taints, or home-assistant/ if you're deploying to a single node cluster. Here's an example with no tolerations or node affinity:

apps:
  home_assistant:
    enabled: false
    description: |
      [link=https://home-assistant.io]Home Assistant[/link] is a home IOT management solution.

      By default, we assume you want to use node affinity and tolerations to keep home assistant pods on certain nodes and keep other pods off said nodes. If you don't want to use either of these features but still want to use the argocd-apps repo, first change the argo path to /home-assistant/ and then remove the 'toleration_' and 'affinity' secret_keys from the yaml file under apps.home_assistant.description.
    argo:
      secret_keys:
        hostname: "home-assistant.coolestdogintheworld.dog"
      repo: https://github.com/open-engineering/argocd-apps
      path: home-assistant/
      revision: main
      namespace: home-assistant
      directory_recursion: false
      project:
        source_repos:
        - http://jessebot.github.io/home-assistant-helm
        destination:
          namespaces:
          - argocd

And here's an example for labeled and tainted nodes, where your pod can use tolerations and node affinity:

apps:
  home_assistant:
    enabled: false
    description: |
      [link=https://home-assistant.io]Home Assistant[/link] is a home IOT management solution.

      By default, we assume you want to use node affinity and tolerations to keep home assistant pods on certain nodes and keep other pods off said nodes. If you don't want to use either of these features but still want to use the argocd-apps repo, first change the argo path to /home-assistant/ and then remove the 'toleration_' and 'affinity' secret_keys from the yaml file under apps.home_assistant.description.
    argo:
      secret_keys:
        hostname: "home-assistant.coolestdogintheworld.dog"
        toleration_key: "blutooth"
        toleration_operator: "Equals"
        toleration_value: "True"
        toleration_effect: "NoSchedule"
        affinity_key: "blutooth"
        affinity_value: "True"
      repo: https://github.com/open-engineering/argocd-apps
      path: home-assistant/toleration_and_affinity/
      revision: main
      namespace: home-assistant
      directory_recursion: false
      project:
        source_repos:
        - http://jessebot.github.io/home-assistant-helm
        destination:
          namespaces:
          - argocd
  • new k3s feature for adding additional nodes

This feature changes k8s_distros.k3s.nodes to be a dictionary so that you can include additional nodes for us to join to the cluster after we create it, but before we install apps. Here's an example of how you can add a new node to k3s on installation:

k8s_distros:
  k3s:
    enabled: false
    k3s_yaml:
      # if you enable MetalLB, we automatically add servicelb to the disable list
      # enables encryption at rest for Kubernetes secrets
      secrets-encryption: true
      # disables traefik so we can enable ingress-nginx, remove if you're using traefik
      disable:
      - "traefik"
      node-label:
      - "ingress-ready=true"
      kubelet-arg:
      - "max-pods=150"
    # nodes to SSH to and join to cluster. example:
    nodes:
      # name can be a hostname or ip address
      serverfriend1.lan:
        # change ssh_key to the name of a local private key to use
        ssh_key: id_rsa
        # must be node type of "worker" or "control_plane"
        node_type: worker
        # labels are optional, but may be useful for pod node affinity
        node_labels:
          - iot=true
        # taints are optional, but may be useful for pod tolerations
        node_taints:
          - iot=true:NoSchedule

if you don't want to add any nodes, this is what you should change your nodes section to be:

k8s_distros:
  k3s:
    enabled: false
    k3s_yaml:
      # if you enable MetalLB, we automatically add servicelb to the disable list
      # enables encryption at rest for Kubernetes secrets
      secrets-encryption: true
      # disables traefik so we can enable ingress-nginx, remove if you're using traefik
      disable:
      - "traefik"
      node-label:
      - "ingress-ready=true"
      kubelet-arg:
      - "max-pods=150"
    # nodes to SSH to and join to cluster. example:
    nodes: {}
  • cert-manager now supports DNS01 challenge solver using the Cloudflare provider

This feature reworks the apps.cert_manager.init and apps.cert_manager.argo.secret_keys sections.

Here's an example of using the HTTP01 challenge solver, which would be the only previously supported challenge solver, so if you want everything to just work how it did before your config file should look like this:

apps:
  cert_manager:
    enabled: true
    description: |
      [link=https://cert-manager.io/]cert-manager[/link] let's you use LetsEncrypt to generate TLS certs for all your apps with ingress.

      smol-k8s-lab supports optional initialization by creating [link=https://cert-manager.io/docs/configuration/acme/]ACME Issuer type[/link] [link=https://cert-manager.io/docs/concepts/issuer/]ClusterIssuers[/link] using either the HTTP01 or DNS01 challenge solvers. We create two ClusterIssuers: letsencrypt-staging and letsencrypt-staging.

      For the DNS01 challange solver, you will need to either export $CLOUDFLARE_API_TOKEN as an env var, or fill in the sensitive value for it each time you run smol-k8s-lab.

      Currently, Cloudflare is the only supported DNS provider for the DNS01 challenge solver. If you'd like to use a different DNS provider or use a different Issuer type all together, please either set one up outside of smol-k8s-lab. We also welcome [link=https://codeberg.org/open-engineering/smol-k8s-lab/pulls]PRs[/link] to add these features :)

    # Initialize of the app through smol-k8s-lab
    init:
      # Deploys staging and prod ClusterIssuers and prompts you for
      # values if they were not set. Switch to false if you don't want
      # to deploy any ClusterIssuers
      enabled: true
      values:
        # Used for to generate certs and alert you if they're going to expire
        email: "you@emailsforfriends.com"
        # choose between "http01" or "dns01"
        cluster_issuer_acme_challenge_solver: http01
        # only needed if cluster_issuer_challenge_solver set to dns01,
        # currently only cloudflare is supported
        cluster_issuer_acme_dns01_provider: cloudflare
      sensitive_values: []
    argo:
      secret_keys: {}
      # git repo to install the Argo CD app from
      repo: "https://codeberg.org/open-engineering/argocd-apps"
      # path in the argo repo to point to. Trailing slash very important!
      path: "cert-manager/"
      # either the branch or tag to point at in the argo repo above
      revision: main
      # namespace to install the k8s app in
      namespace: "cert-manager"
      # recurse directories in the provided git repo
      directory_recursion: false
      # source repos for cert-manager CD App Project (in addition to argo.repo)
      project:
        source_repos:
          - https://charts.jetstack.io
        destination:
          # automatically includes the app's namespace and argocd's namespace
          namespaces:
            - kube-system

And here's how you'd use the new DNS01 feature (keep in mind you need to either provide a sensitive value each time you run smol-k8s-lab, OR you need to export $CLOUDFLARE_API_TOKEN as an env var prior to running smol-k8s-lab):

apps:
  cert_manager:
    enabled: true
    description: |
      [link=https://cert-manager.io/]cert-manager[/link] let's you use LetsEncrypt to generate TLS certs for all your apps with ingress.

      smol-k8s-lab supports optional initialization by creating [link=https://cert-manager.io/docs/configuration/acme/]ACME Issuer type[/link] [link=https://cert-manager.io/docs/concepts/issuer/]ClusterIssuers[/link] using either the HTTP01 or DNS01 challenge solvers. We create two ClusterIssuers: letsencrypt-staging and letsencrypt-staging.

      For the DNS01 challange solver, you will need to either export $CLOUDFLARE_API_TOKEN as an env var, or fill in the sensitive value for it each time you run smol-k8s-lab.

      Currently, Cloudflare is the only supported DNS provider for the DNS01 challenge solver. If you'd like to use a different DNS provider or use a different Issuer type all together, please either set one up outside of smol-k8s-lab. We also welcome [link=https://codeberg.org/open-engineering/smol-k8s-lab/pulls]PRs[/link] to add these features :)

    # Initialize of the app through smol-k8s-lab
    init:
      # Deploys staging and prod ClusterIssuers and prompts you for
      # values if they were not set. Switch to false if you don't want
      # to deploy any ClusterIssuers
      enabled: true
      values:
        # Used for to generate certs and alert you if they're going to expire
        email: "you@emailsforfriends.com"
        # choose between "http01" or "dns01"
        cluster_issuer_acme_challenge_solver: dns01
        # only needed if cluster_issuer_challenge_solver set to dns01
        # currently only cloudflare is supported
        cluster_issuer_acme_dns01_provider: cloudflare
      sensitive_values:
        # can be passed in as env vars if you pre-pend CERT_MANAGER_
        # e.g. CERT_MANAGER_CLOUDFLARE_API_TOKEN
      - CLOUDFLARE_API_TOKEN
    argo:
      secret_keys: {}
      # git repo to install the Argo CD app from
      repo: "https://codeberg.org/open-engineering/argocd-apps"
      # path in the argo repo to point to. Trailing slash very important!
      path: "cert-manager/"
      # either the branch or tag to point at in the argo repo above
      revision: main
      # namespace to install the k8s app in
      namespace: "cert-manager"
      # recurse directories in the provided git repo
      directory_recursion: false
      # source repos for cert-manager CD App Project (in addition to argo.repo)
      project:
        source_repos:
          - https://charts.jetstack.io
        destination:
          # automatically includes the app's namespace and argocd's namespace
          namespaces:
            - kube-system
Upgrading config from v1.x to v2.x

If you've installed smol-k8s-lab prior to v2.0.0, please backup your old configuration, and then remove the ~/.config/smol-k8s-lab/config.yaml (or $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml) file entirely, then run the following:

# this upgrades smol-k8s-lab
pip3.11 install --upgrade smol-k8s-lab

# this initializes a new configuration
smol-k8s-lab

The main difference between the old and new config files are for apps, we've added:

  • apps.APPNAME.description - for adding a custom description, set it to whatever you like
  • apps.APPNAME.argo.directory_recursion - so you can have bigger nested apps :)
  • apps.APPNAME.argo.project.destination.namespaces - control what namespaces are allowed for a project

And we've changed:

  • apps.APPNAME.argo.ref to apps.APPNAME.argo.revision
  • apps.APPNAME.argo.project_source_repos to apps.APPNAME.argo.project.source_repos

And we've REMOVED:

  • apps.APPNAME.argo.part_of_app_of_apps - this was mostly used internally, we think

Here's an example of an updated cert-manager app with the new config:

apps:
  cert_manager:
    # ! NOTE: you currently can't set this to false. It is necessary to deploy
    # most of our supported Argo CD apps since they often have TLS enabled either
    # for pod connectivity or ingress
    enabled: true
    description: |
      [link=https://cert-manager.io/]cert-manager[/link] let's you use LetsEncrypt to generate TLS certs for all your apps with ingress.

      smol-k8s-lab supports initialization by creating two [link=https://cert-manager.io/docs/concepts/issuer/]ClusterIssuers[/link] for both staging and production using a provided email address as the account ID for acme.

    # Initialize of the app through smol-k8s-lab
    init:
      # Deploys staging and prod ClusterIssuers and prompts you for
      # cert-manager.argo.secret_keys if they were not set. Switch to false if
      # you don't want to deploy any ClusterIssuers
      enabled: true
    argo:
      secret_keys:
        # Used for letsencrypt-staging, to generate certs
        email: ""
      # git repo to install the Argo CD app from
      repo: "https://codeberg.org/open-engineering/argocd-apps"
      # path in the argo repo to point to. Trailing slash very important!
      path: "cert-manager/"
      # either the branch or tag to point at in the argo repo above
      revision: main
      # namespace to install the k8s app in
      namespace: "cert-manager"
      # recurse directories in the provided git repo
      directory_recursion: false
      # source repos for cert-manager CD App Project (in addition to argo.repo)
      project:
        source_repos:
          - https://charts.jetstack.io
        destination:
          # automatically includes the app's namespace and argocd's namespace
          namespaces:
            - kube-system

Under the hood

Note: this project is not officially affiliated with any of the below tooling or applications.

Supported k8s distributions

We always install the latest version of Kubernetes that is available from the distro's startup script.

Distro Description

k3s
The certified Kubernetes distribution built for IoT & Edge computing

k3d
TESTING PHASE k3s in docker 🐳

KinD
kind is a tool for running local Kubernetes clusters using Docker container “nodes”. kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

We tend to test first on k3s first, then the other distros. k3d support coming soon.

Default Installed Applications

All of these can be disabled with the exception of Argo CD, which is optional, but if not installed, smol-k8s-lab will only install: MetalLB, nginx-ingress, and cert-manager.

Application Description Initialization Supported
metallb logo, blue arrow pointing up, with small line on one leg of arrow to show balance
metallb
Loadbalancer and IP Address pool manager for metal
cilium logo
cilium
Gateway API (a bit clunky right now, but getting better!)
traefik logo, blue gopher wearing a plane director outfit
Traefik
Traefik is both an Ingress and Gateway API controller and we're currently switching to supporting it in addition to cilium since ingress-nginx is deprecated.
cert manager logo
cert-manager
For SSL/TLS certificates
argo CD logo, an organer squid wearing a fishbowl helmet
Argo CD
Gitops - Continuous Deployment
argo CD logo, an organer squid wearing a fishbowl helmet
Argo CD Appset Secret Plugin
Gitops - Continuous Deployment
ESO logo, outline of robot with astricks in a screen in it's belly
ESO
external-secrets-operator integrates external secret management systems like Bitwarden
ESO logo, again
Bitwarden ESO Provider
Bitwarden external-secrets-operator provider
OpenBao logo, drawing of 3 small bao with simple eyes personifying them
OpenBao
OpenBao (Hashicorp Vault fork) - it's a Secrets Store.
Zitadel logo, an orange arrow pointing left
ZITADEL
An identity provider and OIDC provider to provide SSO
Prometheus logo, a torch
Prometheus Stack
Prometheus monitoring and logging stack using loki/alloy, alert manager, and grafana

For a complete list of installable applications, checkout the default apps docs. To install your own custom apps, you can check out an example via the config file or learn how to do it via the tui.

Phases

Below we define each of the phases your app can be installed in.

Bootstrap

no metrics.

  • metallb.yaml
  • prometheus_crds.yaml
  • cilium.yaml or traefik.yaml
  • traefik.yaml or traefik.yaml
  • ingress_nginx.yaml (deprecated 😞)
  • certmanager.yaml
  • k8tz.yaml

operator

yes metrics.

  • external_secrets_operator.yaml
  • k8up.yaml
  • cnpg_operator.yaml

cd

needs both bootstrap and operators...

  • argocd.yaml

auth

required for any sort of thing that uses oidc

  • zitadel.yaml

default

requires bootstrap, operators, cd, and auth

  • forgejo.yaml

Status

Somewhat stable and being actively supported, so if you'd like to contribute or just found a :bug:, feel free to open an issue (and/or pull request), and we'll try to take a look ASAP!

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

smol_k8s_lab-11.0.0.tar.gz (224.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

smol_k8s_lab-11.0.0-py3-none-any.whl (329.0 kB view details)

Uploaded Python 3

File details

Details for the file smol_k8s_lab-11.0.0.tar.gz.

File metadata

  • Download URL: smol_k8s_lab-11.0.0.tar.gz
  • Upload date:
  • Size: 224.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/2.4.1 CPython/3.11.2 Linux/6.1.0-51-amd64

File hashes

Hashes for smol_k8s_lab-11.0.0.tar.gz
Algorithm Hash digest
SHA256 617e5f887cc5ec1cd9074b7e5e72e071ad9fc12726c4b6b22b511a02b34707e6
MD5 bf6d6f5ed173ead43fc521684aec49b9
BLAKE2b-256 3791d7272ea7b5629b456e8238b869d99953d0139a2c937eabdf0d5719043cca

See more details on using hashes here.

File details

Details for the file smol_k8s_lab-11.0.0-py3-none-any.whl.

File metadata

  • Download URL: smol_k8s_lab-11.0.0-py3-none-any.whl
  • Upload date:
  • Size: 329.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/2.4.1 CPython/3.11.2 Linux/6.1.0-51-amd64

File hashes

Hashes for smol_k8s_lab-11.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 b693af06da799495511657db868ec35a113c281d51575f133c8f4efb13672611
MD5 4087bb672a0cc8216915cb528e7d3f53
BLAKE2b-256 475e5c38d94c35d92c19682d6e815d6bd10a70a3066bfca39c1fa2059d8facc1

See more details on using hashes here.

Release history Release notifications | RSS feed

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page