🧸 smol-k8s-lab
A terminal based tool to install slimmer k8s distros on metal, with batteries included!
Features
- Deploys Argo CD by default, so you can manage your entire lab using files in open source git repos
- Argo CD ships with a dashboard with a custom theme 💙
- Argo CD is default secured with a self hosted OIDC provider, Zitadel
- Uses a local password manager of your choice. Currently supported:
- Bitwarden
- KeyPass
- macOS keychain
- Freedesktop Secret Service supports many DE including GNOME (requires secretstorage)
- KDE4 & KDE5 KWallet (requires dbus)
- External Secerts management for your sensitive Kubernetes Secret values. Currently supported:
- Bitwarden
- OpenBao
- Manages all your authentication needs centrally using Zitadel, a selfed hosted OIDC provider
- Supports initialization on a range of common self-hosted apps 📱
- featured initialized apps such as Zitadel, Nextcloud, Matrix, and Home Assistant include [b]backups and restores[/b]!
- Lots o' docs
- commitment to never monetizing this app. It will always be AGPL licensed and free 💙
- No AI or LLMs "features" and No AI contribution policy
Installation
🐝 sure to check out our full installation guide, but the gist of it is smol-k8s-lab can be installed via pipx (or brew coming soon).
pipx
smol-k8s-lab requires Python 3.12+ (and pipx). If you've already got both and other pre-reqs, you should be able to:
# install the CLI
pipx install smol-k8s-lab
# Check the help menu before proceeding
smol-k8s-lab --help
brew (still unstable)
brew is for macOS/Debian/Ubuntu, as this will also install any non-python prerequisites you need, so you don't need to worry about them. This method is new, so please let us know if anything isn't working for you.
# tap the special homebrew repo for our formula and install it
brew install open-engineering/tap/smol-k8s-lab
Then you should be able to check the version and cli options with:
smol-k8s-lab --help
Checkout our TUI docs for more info on how to get started playing with smol-k8s-lab :-)
Usage
Initialization
After you've followed the installation instructions, if you're new to smol-k8s-lab, initialize a new config file:
# we'll walk you through any configuration needed before
# saving the config and deploying it for you
smol-k8s-lab
Upgrading to 11.0.0
v11.0.0 introduces the concept of environments and also drops support for the now deprecated Ingress Nginx and starts using Traefik with Gateway API. This is important for people running smol-k8s-lab for their home labs or running it as a production system for small orgs.
A major difference is that your config directory $XDG_CONFIG_HOME/smol-k8s-lab/ is now split up into a couple different directories. For simplicity's sake, we'll refer to $XDG_CONFIG_HOME/smol-k8s-lab/ as your config directory, but note that this defaults to ~/.config/smol-k8s-lab unless you have set the $XDG_CONFIG_HOME env var to be something else.
Your name config directory $XDG_CONFIG_HOME/smol-k8s-lab/apps file no longer contains your apps. Instead, all of your apps now live in $XDG_CONFIG_HOME/smol-k8s-lab/apps/$ENV with files for each app in that environment.
The following apps are considered core apps and should live in $XDG_CONFIG_HOME/smol-k8s-lab/apps/core as files of that name:
- alloy
- argocd
- autoscaler
- certmanager
- cilium
- cnpg_operator
- external_secrets_operator
- k8tz
- k8up
- kyverno
- longhorn
- openbao
- pxc_operator
- snapshot_controller
- traefik
- velero
These files are optionally included in core and can also live in other envs as well:
- grafana_stack
- zitadel
Zitadel is a special app, in that where ever it is deployed, it will be used for OIDC apps in that environment only. Everything in the core apps, should behave as they did before with no major changes. If you were only using smol-k8s-lab to boostrap a cluster and nothing else, then from here, all you need to do is remove ingress nginx, and rerun smol-k8s-lab with traefik enabled (and set apps_global_config.ingressClass and apps_global_config.gatewayClassName to traefik in your config.yaml file).
Everything else should live in a directory of that environment name, so for instance, if you'd like an environment called dev-my-org, you'd create a directory called $XDG_CONFIG_HOME/smol-k8s-lab/apps/dev-my-org.
To create a new environment, you can run smol-k8s-lab -n and we'll walk you through setting up a new environment. You can also launch the TUI and on the apps config screen, check the Add New Env check box and we'll walk you through that as well.
This release also introduces OpenBao, a fork of Vault, as the default External Secrets provider, rather than using the Argo CD ApplicationSet Secrets Plugin or Bitwarden. The ApplicationSet Secrets plugin was never encrypted and not intended to be used for sensitive values, while Bitwarden's company may not remain open source forever, so we're making the decision to quickly adopt OpenBao instead which can do both sensitive and insensitive values as well as much much more when it comes to security.
Because of this, while the smol-k8s-lab out of the box experience will be streamlined and straight forward, upgrading to 11.x from 10.x is going to be a bit of a chore. If you don't care about anything currently managed by smol-k8s-lab, you can just delete your cluster and wipe your $XDG_CONFIG_HOME/smol-k8s-lab to start fresh. Then you can use the TUI to setup your new values, or you can just run smol-k8s-lab and we'll install all our default apps.
If you do care about your current deployed apps:
-
We recommend backing up EVERYTHING, including both all your config files, but also, take restic backups using k8up of all your current PVCs and verify they are working and available in your remote S3 buckets.
-
You'll need to also schedule down time so that any users know that you'll be offline during the migration as you'll need to take down all your non-core apps. When you're good to take them all down AFTER you've verified your backups are working, you can proceed to the next step.
-
Create a directory for your new environment, so let's say that's called
prod, you'd create a directory called$XDG_CONFIG_HOME/smol-k8s-lab/apps/prodand move all your non-core apps into it (e.g. home assistant, mastodon, peertube, etc). At this time, you can take down your existing apps (completely delete them from Kubernetes). -
Set
apps_global_config.external_secretstoopenbaoandapps_global_config.migrate_to_openbaototruein your config.yaml file, and then set the following in each of your apps:restore.enabled: true. Then, you should be able to runsmol-k8s-labas normal and all should be well!
Upgrading from 9.x to 10.x
This update officially introduces the beginning of our smol-schema, meaning we stop using using the appset secret plugin officially, and start instead using targeted helm charts of Argo CD Applications. We also stop overloading the init.values of every app, and instead use targeted parameters. This means instead of e.g. for zitadel's admin and smtp section:
init:
values:
# admin info
admin_user: ""
admin_email: ""
# smtp info
smtp_host: ""
smtp_user: ""
smtp_port: ""
smtp_protocol: "SMTP"
# smtp from email (verified sender)
smtp_from_address: example.com
smtp_password:
value_from:
env: GHOST_SMTP_PASSWORD
You would provide the following in your $XDG_CONFIG_HOME/smol-k8s-lab/apps/zitadel.yaml:
# note that this is at the top level of the yaml
admin:
username: ""
email: ""
# note that this is at the top level of the yaml
smtp:
# smtp info
host: ""
username: ""
port: ""
protocol: "SMTP"
# smtp from email (verified sender)
from_address: example.com
password:
value_from:
env: GHOST_SMTP_PASSWORD
Likewise, for databases, you'd provide:
# note that this is at the top level of the yaml
db:
# postgresql is the only supported app for most apps, but sometimes we support mysql like in the case of ghost and writefreely
type: postgresql
# this is your database pvc capacity
capacity: 5Gi
For affinity and tolerations:
# note that these are at the top level of the yaml
affinity: {}
tolerations: []
PVCs now also get their own parameters like this:
# note this is a top level parameter
pvc:
# this is for an example PVC named ghost
ghost:
# this enables it
enabled: true
# this is the ghost pvc capacity
capacity: 10Gi
Valkey can be enabled or disabled with:
# note this is a top level parameter
valkey:
# enable valkey
enabled: true
for hostnames, those are also declared at the top level:
hostnames:
zitadel: "iam.example.com"
Upgrading from 8.5.1 to 9.x
Changes how we provide prometheus CRDs and alloy:
- alloy is now its own app and prometheus_crds are not their own app, instead you can set alloy's syncWaves to 1 to accomplish this
- alloy is no longer part of the grafana stack, so you can deploy it separately if needed
Upgrading from 8.5.0 to 8.5.1
fix default config to call pvc_storage_class storage_class instead: 5c4f31eb3f
Your config should now read something like this:
# anything here gets set for all apps if you're using our default repos
apps_global_config:
# setting this changes all the below domains to use the following cluster_issuer
# change to letsencrypt-prod when you're ready to go live with your infra
cluster_issuer: "letsencrypt-staging"
# change to your tz: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
time_zone: "Europe/Amsterdam"
# always deploy extnernal secrets. *Must be a string of either:
# "" - don't use external secrets
# "bitwarden" - use bitwarden for external secrets
# "openbao" - (coming soon!) use openbao (a fork of vault) for external secrets
external_secrets: "bitwarden"
# the default storage class you want to use for the creation of all Persistent Volume claims
# it defaults to local-path, but you can set it to longhorn or whatever you'd like :)
# for storage classes see:
storage_class: "local-path"
Upgrading from 8.4.x to 8.5.0
- changed how we determine if we should install the appset secret plugin: 04514b1066
You now need to pass the following into your argocd app's config:
argocd:
# set this to true if you want to use the appset secret plugin (defaults to false)
enable_appset_secret_plugin: true
Upgrading from 7.x to 8.4.x
- This update begins removing ALL references to our previous GitHub repos. Please instead point to our new codeberg.org/open-engineering org for all Argo CD Applications and any helm charts of ours you used.
- We switched to the Grafana community helm chart repo. You'll want this for any future community grafana updates
- bumped Kubernetes and Ruamel lib versions major versions to latest.
Upgrading from 6.x to 7.x
- This is where we officially switch from Thanos to Mimir and start supporting Tempo for self hosted tracing.
- we don't offer a migration path from thanos unfortunately. We recommend re-deploying the grafana_stack
- We also start supporting sensitive values for jellyfin.
Upgrading from 5.x to 6.x
A major difference is that your config directory $XDG_CONFIG_HOME/smol-k8s-lab/ is now split up into a couple different directories. For simplicity's sake, we'll refer to $XDG_CONFIG_HOME/smol-k8s-lab/ as your config directory, but note that this defaults to ~/.config/smol-k8s-lab unless you have set the $XDG_CONFIG_HOME env var to be something else.
Your name config file $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml file no longer contains your apps at all. Instead, all of your apps now live in $XDG_CONFIG_HOME/smol-k8s-lab/apps and within that directory, you'll files for each app in that environment. The apps parameter in your $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml will no longer be used. Instead, take each app's config parameters and move them to a file by that name in your apps directory, so for example, this is what your $XDG_CONFIG_HOME/smol-k8s-lab/apps/zitadel.yaml should look like, in order to deploy zitadel:
zitadel.yaml contents
```yaml
enabled: true
description: |
[link=https://zitadel.com/opensource]ZITADEL[/link] is an open source self hosted IAM platform for the cloud era
smol-k8s-lab supports initialization of:
- an admin service account
- a human admin user (including an autogenerated password)
- a project with a name of your chosing
- 2 OIDC applications for Argo CD and Vouch
- 2 Argo CD groups (admins and users), 1 vouch groups
- groupsClaim action to enforce group roles on authentication
- updates your appset_secret_plugin secret and refreshes the pod
The default app will also deploy SeaweedFS to backup your database which in turn is backed up to a remote s3 provider of your choice.
To provide sensitive values via environment variables to smol-k8s-lab use:
- ZITADEL_S3_BACKUP_ACCESS_ID
- ZITADEL_S3_BACKUP_SECRET_KEY
- ZITADEL_RESTIC_REPO_PASSWORD
init:
# Switch to false if you don't want to create intial secrets or use the
# API via a service acocunt to create the above described resources
enabled: true
restore:
# set to true to run a restic restore via a k8up job for seaweedfs PVCs,
# and restoring your postgresql database
enabled: false
# for Cloudnative postgres operator cluster restores only
cnpg_restore: true
# the restic snapshot IDs of the PVC snapshots you want to restore
restic_snapshot_ids:
# seaweedFS volume pvc snapshot id. if not given, we use the latest
seaweedfs_volume: ""
# seaweedFS filer pvc snapshot id. if not given, we use the latest
seaweedfs_filer: ""
values:
# initial human user's usename
admin_user: ""
# initial human user's email
email: ""
# initial human's first name
first_name: ""
# initial human's last name
last_name: ""
# options: GENDER_UNSPECIFIED, GENDER_MALE, GENDER_FEMALE, GENDER_DIVERSE
# more coming soon, see: https://github.com/zitadel/zitadel/issues/6355
gender: GENDER_UNSPECIFIED
# name of the default project to create OIDC applications in
project: core
# mail server, must include port! e.g. mymailserver.com:587
smtp_host:
value_from:
env: ZITADEL_SMTP_HOST
# mail user
smtp_user:
value_from:
env: ZITADEL_SMTP_USER
# mail password
smtp_password:
value_from:
env: ZITADEL_SMTP_PASSWORD
# mail from address
smtp_from_address:
value_from:
env: ZITADEL_SMTP_FROM_ADDRESS
# mail from name
smtp_from_name:
value_from:
env: ZITADEL_SMTP_FROM_NAME
# mail reply to address
smtp_reply_to_address:
value_from:
env: ZITADEL_SMTP_REPLY_TO_ADDRESS
backups:
# cronjob syntax schedule to run zitadel seaweedfs pvc backups
pvc_schedule: 10 0 * * *
# cronjob syntax (with SECONDS field) for zitadel postgres backups
# must happen at least 10 minutes before pvc backups, to avoid corruption
# due to missing files. This is because the cnpg backup shows as completed
# before it actually is, due to the wal archive it lists as it's end not
# being in the backup yet
postgres_schedule: 0 0 0 * * *
s3:
# these are for pushing remote backups of your local s3 storage, for speed and cost optimization
endpoint: ""
bucket: ""
region: ""
secret_access_key:
value_from:
env: ZITADEL_S3_BACKUP_SECRET_KEY
access_key_id:
value_from:
env: ZITADEL_S3_BACKUP_ACCESS_ID
restic_repo_password:
value_from:
env: ZITADEL_RESTIC_REPO_PASSWORD
argo:
# secrets keys to make available to Argo CD ApplicationSets
secret_keys:
# FQDN to use for zitadel
hostname: ""
# set the local s3 provider for zitadel's database backups. can be minio or seaweedfs
s3_provider: seaweedfs
# local s3 endpoint for postgresql backups, backed up constantly
s3_endpoint: ""
# capacity for the PVC backing your local s3 instance
s3_pvc_capacity: 2Gi
# repo to install the Argo CD app from
# git repo to install the Argo CD app from
repo: https://codeberg.org/small-hack/argocd-apps
# path in the argo repo to point to.
path: "zitadel/app_of_apps/"
# either the branch or tag to point at in the argo repo above
revision: main
# kubernetes cluster to install the k8s app into, defaults to Argo CD default
cluster: https://kubernetes.default.svc
# namespace to install the k8s app in
namespace: "zitadel"
# recurse directories in the provided git repo
directory_recursion: false
# source repos for Argo CD App Project (in addition to argo.repo)
project:
name: zitadel
source_repos:
- https://charts.zitadel.com
- https://zitadel.github.io/zitadel-charts
- https://small-hack.github.io/cloudnative-pg-cluster-chart
- https://operator.min.io/
- https://seaweedfs.github.io/seaweedfs/helm
- https://git.smallhack.org/small-hack/argocd-apps
- https://codeberg.org/small-hack/argocd-apps
destination:
# automatically includes the app's namespace and argocd's namespace
namespaces: []
```
There are no other major changes except that:
- we upgrade Textual to a new major version. This is not something you need to worry about unless you forked this repo.
- we drop support for collabora as a stand alone app (it's now part of nextcloud)
- we add gotosocial as a default app
Upgrading config from v4.x to v5.x
If you've installed smol-k8s-lab prior to v5.0.0, please backup your old configuration, and then remove the ~/.config/smol-k8s-lab/config.yaml (or $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml) file entirely, then run the following with either pip or pipx:
if using pip:
# this uninstalls the old smol-k8s-lab for python 3.11
pip3.11 uninstall smol-k8s-lab
# this installs smol-k8s-lab for python 3.12
pip3.12 install --upgrade smol-k8s-lab
# this initializes a new configuration
smol-k8s-lab
or if using pipx:
# this upgrades smol-k8s-lab
pipx upgrade smol-k8s-lab
# this initializes a new configuration
smol-k8s-lab
We have done a masive upgrade of the config file. You'll need to update your configs based on the details in https://github.com/open-engineering-nl/smol-k8s-lab/pull/210 . The main changes are to the following (check each doc link for details):
- accessibility features
- k3s nodes section
- backups and restores
- sensitive values
- k9s has been removed in favor of run command (hint: you can still use k9s via run command)
Upgrading config from v3.7.1 to v4.x
If you've installed smol-k8s-lab prior to v4.0.0, please backup your old configuration, and then remove the ~/.config/smol-k8s-lab/config.yaml (or $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml) file entirely, then run the following with either pip or pipx:
if using pip:
# this upgrades smol-k8s-lab
pip3.11 install --upgrade smol-k8s-lab
# this initializes a new configuration
smol-k8s-lab
or if using pipx:
# this upgrades smol-k8s-lab
pipx upgrade smol-k8s-lab
# this initializes a new configuration
smol-k8s-lab
The main breaking changes between v3.7.1 and v4.0.0 are that we now default enable metrics on most apps. Because of this, you need to have the Prometheus ServiceMonitor CRD installed ahead of time. Luckily, we now provide that as an app as well :) If you deleted your config and created a new one, it will already be there, but if you want to reuse your old config, you can add the app like this:
apps:
prometheus_crds:
description: |
[link=https://prometheus.io/docs/introduction/overview/]Prometheus[/link] CRDs to start with.
You can optionally disable this if you don't want to deploy apps with metrics.
enabled: true
argo:
# secrets keys to make available to Argo CD ApplicationSets
secret_keys: {}
# git repo to install the Argo CD app from
repo: https://codeberrg.org/open-engineering/argocd-apps
# path in the argo repo to point to. Trailing slash very important!
path: prometheus/crds/
# either the branch or tag to point at in the argo repo above
revision: main
# namespace to install the k8s app in
namespace: prometheus
# recurse directories in the provided git repo
directory_recursion: false
# source repos for Argo CD App Project (in addition to argo.repo)
project:
name: prometheus
source_repos:
- https://github.com/prometheus-community/helm-charts.git
destination:
# automatically includes the app's namespace and argocd's namespace
namespaces:
- kube-system
- prometheus
If using the default repos, please also disable directory directory_recursion for:
- your prometheus stack app
- zitadel
For all changes, please check out PR #206.
Upgrading config from v2.2.4 to v3.x
If you've installed smol-k8s-lab prior to v3.0.0, please backup your old configuration, and then remove the ~/.config/smol-k8s-lab/config.yaml (or $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml) file entirely, then run the following with either pip or pipx:
if using pip:
# this upgrades smol-k8s-lab
pip3.11 install --upgrade smol-k8s-lab
# this initializes a new configuration
smol-k8s-lab
or if using pipx:
# this upgrades smol-k8s-lab
pipx upgrade smol-k8s-lab
# this initializes a new configuration
smol-k8s-lab
The main breaking changes between v2.2.4 and v3.0 are as follows:
- home assistant has graduated from demo app to live app
You'll need to change apps.home_assistant.argo.path to either home-assistant/toleration_and_affinity/ if you're using node labels and taints, or home-assistant/ if you're deploying to a single node cluster. Here's an example with no tolerations or node affinity:
apps:
home_assistant:
enabled: false
description: |
[link=https://home-assistant.io]Home Assistant[/link] is a home IOT management solution.
By default, we assume you want to use node affinity and tolerations to keep home assistant pods on certain nodes and keep other pods off said nodes. If you don't want to use either of these features but still want to use the argocd-apps repo, first change the argo path to /home-assistant/ and then remove the 'toleration_' and 'affinity' secret_keys from the yaml file under apps.home_assistant.description.
argo:
secret_keys:
hostname: "home-assistant.coolestdogintheworld.dog"
repo: https://github.com/open-engineering/argocd-apps
path: home-assistant/
revision: main
namespace: home-assistant
directory_recursion: false
project:
source_repos:
- http://jessebot.github.io/home-assistant-helm
destination:
namespaces:
- argocd
And here's an example for labeled and tainted nodes, where your pod can use tolerations and node affinity:
apps:
home_assistant:
enabled: false
description: |
[link=https://home-assistant.io]Home Assistant[/link] is a home IOT management solution.
By default, we assume you want to use node affinity and tolerations to keep home assistant pods on certain nodes and keep other pods off said nodes. If you don't want to use either of these features but still want to use the argocd-apps repo, first change the argo path to /home-assistant/ and then remove the 'toleration_' and 'affinity' secret_keys from the yaml file under apps.home_assistant.description.
argo:
secret_keys:
hostname: "home-assistant.coolestdogintheworld.dog"
toleration_key: "blutooth"
toleration_operator: "Equals"
toleration_value: "True"
toleration_effect: "NoSchedule"
affinity_key: "blutooth"
affinity_value: "True"
repo: https://github.com/open-engineering/argocd-apps
path: home-assistant/toleration_and_affinity/
revision: main
namespace: home-assistant
directory_recursion: false
project:
source_repos:
- http://jessebot.github.io/home-assistant-helm
destination:
namespaces:
- argocd
- new k3s feature for adding additional nodes
This feature changes k8s_distros.k3s.nodes to be a dictionary so that you can include additional nodes for us to join to the cluster after we create it, but before we install apps. Here's an example of how you can add a new node to k3s on installation:
k8s_distros:
k3s:
enabled: false
k3s_yaml:
# if you enable MetalLB, we automatically add servicelb to the disable list
# enables encryption at rest for Kubernetes secrets
secrets-encryption: true
# disables traefik so we can enable ingress-nginx, remove if you're using traefik
disable:
- "traefik"
node-label:
- "ingress-ready=true"
kubelet-arg:
- "max-pods=150"
# nodes to SSH to and join to cluster. example:
nodes:
# name can be a hostname or ip address
serverfriend1.lan:
# change ssh_key to the name of a local private key to use
ssh_key: id_rsa
# must be node type of "worker" or "control_plane"
node_type: worker
# labels are optional, but may be useful for pod node affinity
node_labels:
- iot=true
# taints are optional, but may be useful for pod tolerations
node_taints:
- iot=true:NoSchedule
if you don't want to add any nodes, this is what you should change your nodes section to be:
k8s_distros:
k3s:
enabled: false
k3s_yaml:
# if you enable MetalLB, we automatically add servicelb to the disable list
# enables encryption at rest for Kubernetes secrets
secrets-encryption: true
# disables traefik so we can enable ingress-nginx, remove if you're using traefik
disable:
- "traefik"
node-label:
- "ingress-ready=true"
kubelet-arg:
- "max-pods=150"
# nodes to SSH to and join to cluster. example:
nodes: {}
- cert-manager now supports DNS01 challenge solver using the Cloudflare provider
This feature reworks the apps.cert_manager.init and apps.cert_manager.argo.secret_keys sections.
Here's an example of using the HTTP01 challenge solver, which would be the only previously supported challenge solver, so if you want everything to just work how it did before your config file should look like this:
apps:
cert_manager:
enabled: true
description: |
[link=https://cert-manager.io/]cert-manager[/link] let's you use LetsEncrypt to generate TLS certs for all your apps with ingress.
smol-k8s-lab supports optional initialization by creating [link=https://cert-manager.io/docs/configuration/acme/]ACME Issuer type[/link] [link=https://cert-manager.io/docs/concepts/issuer/]ClusterIssuers[/link] using either the HTTP01 or DNS01 challenge solvers. We create two ClusterIssuers: letsencrypt-staging and letsencrypt-staging.
For the DNS01 challange solver, you will need to either export $CLOUDFLARE_API_TOKEN as an env var, or fill in the sensitive value for it each time you run smol-k8s-lab.
Currently, Cloudflare is the only supported DNS provider for the DNS01 challenge solver. If you'd like to use a different DNS provider or use a different Issuer type all together, please either set one up outside of smol-k8s-lab. We also welcome [link=https://codeberg.org/open-engineering/smol-k8s-lab/pulls]PRs[/link] to add these features :)
# Initialize of the app through smol-k8s-lab
init:
# Deploys staging and prod ClusterIssuers and prompts you for
# values if they were not set. Switch to false if you don't want
# to deploy any ClusterIssuers
enabled: true
values:
# Used for to generate certs and alert you if they're going to expire
email: "you@emailsforfriends.com"
# choose between "http01" or "dns01"
cluster_issuer_acme_challenge_solver: http01
# only needed if cluster_issuer_challenge_solver set to dns01,
# currently only cloudflare is supported
cluster_issuer_acme_dns01_provider: cloudflare
sensitive_values: []
argo:
secret_keys: {}
# git repo to install the Argo CD app from
repo: "https://codeberg.org/open-engineering/argocd-apps"
# path in the argo repo to point to. Trailing slash very important!
path: "cert-manager/"
# either the branch or tag to point at in the argo repo above
revision: main
# namespace to install the k8s app in
namespace: "cert-manager"
# recurse directories in the provided git repo
directory_recursion: false
# source repos for cert-manager CD App Project (in addition to argo.repo)
project:
source_repos:
- https://charts.jetstack.io
destination:
# automatically includes the app's namespace and argocd's namespace
namespaces:
- kube-system
And here's how you'd use the new DNS01 feature (keep in mind you need to either provide a sensitive value each time you run smol-k8s-lab, OR you need to export $CLOUDFLARE_API_TOKEN as an env var prior to running smol-k8s-lab):
apps:
cert_manager:
enabled: true
description: |
[link=https://cert-manager.io/]cert-manager[/link] let's you use LetsEncrypt to generate TLS certs for all your apps with ingress.
smol-k8s-lab supports optional initialization by creating [link=https://cert-manager.io/docs/configuration/acme/]ACME Issuer type[/link] [link=https://cert-manager.io/docs/concepts/issuer/]ClusterIssuers[/link] using either the HTTP01 or DNS01 challenge solvers. We create two ClusterIssuers: letsencrypt-staging and letsencrypt-staging.
For the DNS01 challange solver, you will need to either export $CLOUDFLARE_API_TOKEN as an env var, or fill in the sensitive value for it each time you run smol-k8s-lab.
Currently, Cloudflare is the only supported DNS provider for the DNS01 challenge solver. If you'd like to use a different DNS provider or use a different Issuer type all together, please either set one up outside of smol-k8s-lab. We also welcome [link=https://codeberg.org/open-engineering/smol-k8s-lab/pulls]PRs[/link] to add these features :)
# Initialize of the app through smol-k8s-lab
init:
# Deploys staging and prod ClusterIssuers and prompts you for
# values if they were not set. Switch to false if you don't want
# to deploy any ClusterIssuers
enabled: true
values:
# Used for to generate certs and alert you if they're going to expire
email: "you@emailsforfriends.com"
# choose between "http01" or "dns01"
cluster_issuer_acme_challenge_solver: dns01
# only needed if cluster_issuer_challenge_solver set to dns01
# currently only cloudflare is supported
cluster_issuer_acme_dns01_provider: cloudflare
sensitive_values:
# can be passed in as env vars if you pre-pend CERT_MANAGER_
# e.g. CERT_MANAGER_CLOUDFLARE_API_TOKEN
- CLOUDFLARE_API_TOKEN
argo:
secret_keys: {}
# git repo to install the Argo CD app from
repo: "https://codeberg.org/open-engineering/argocd-apps"
# path in the argo repo to point to. Trailing slash very important!
path: "cert-manager/"
# either the branch or tag to point at in the argo repo above
revision: main
# namespace to install the k8s app in
namespace: "cert-manager"
# recurse directories in the provided git repo
directory_recursion: false
# source repos for cert-manager CD App Project (in addition to argo.repo)
project:
source_repos:
- https://charts.jetstack.io
destination:
# automatically includes the app's namespace and argocd's namespace
namespaces:
- kube-system
Upgrading config from v1.x to v2.x
If you've installed smol-k8s-lab prior to v2.0.0, please backup your old configuration, and then remove the ~/.config/smol-k8s-lab/config.yaml (or $XDG_CONFIG_HOME/smol-k8s-lab/config.yaml) file entirely, then run the following:
# this upgrades smol-k8s-lab
pip3.11 install --upgrade smol-k8s-lab
# this initializes a new configuration
smol-k8s-lab
The main difference between the old and new config files are for apps, we've added:
apps.APPNAME.description- for adding a custom description, set it to whatever you likeapps.APPNAME.argo.directory_recursion- so you can have bigger nested apps :)apps.APPNAME.argo.project.destination.namespaces- control what namespaces are allowed for a project
And we've changed:
apps.APPNAME.argo.reftoapps.APPNAME.argo.revisionapps.APPNAME.argo.project_source_repostoapps.APPNAME.argo.project.source_repos
And we've REMOVED:
apps.APPNAME.argo.part_of_app_of_apps- this was mostly used internally, we think
Here's an example of an updated cert-manager app with the new config:
apps:
cert_manager:
# ! NOTE: you currently can't set this to false. It is necessary to deploy
# most of our supported Argo CD apps since they often have TLS enabled either
# for pod connectivity or ingress
enabled: true
description: |
[link=https://cert-manager.io/]cert-manager[/link] let's you use LetsEncrypt to generate TLS certs for all your apps with ingress.
smol-k8s-lab supports initialization by creating two [link=https://cert-manager.io/docs/concepts/issuer/]ClusterIssuers[/link] for both staging and production using a provided email address as the account ID for acme.
# Initialize of the app through smol-k8s-lab
init:
# Deploys staging and prod ClusterIssuers and prompts you for
# cert-manager.argo.secret_keys if they were not set. Switch to false if
# you don't want to deploy any ClusterIssuers
enabled: true
argo:
secret_keys:
# Used for letsencrypt-staging, to generate certs
email: ""
# git repo to install the Argo CD app from
repo: "https://codeberg.org/open-engineering/argocd-apps"
# path in the argo repo to point to. Trailing slash very important!
path: "cert-manager/"
# either the branch or tag to point at in the argo repo above
revision: main
# namespace to install the k8s app in
namespace: "cert-manager"
# recurse directories in the provided git repo
directory_recursion: false
# source repos for cert-manager CD App Project (in addition to argo.repo)
project:
source_repos:
- https://charts.jetstack.io
destination:
# automatically includes the app's namespace and argocd's namespace
namespaces:
- kube-system
Under the hood
Note: this project is not officially affiliated with any of the below tooling or applications.
Supported k8s distributions
We always install the latest version of Kubernetes that is available from the distro's startup script.
| Distro | Description |
|---|---|
k3s |
The certified Kubernetes distribution built for IoT & Edge computing |
k3d |
TESTING PHASE k3s in docker 🐳 |
KinD |
kind is a tool for running local Kubernetes clusters using Docker container “nodes”. kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI. |
We tend to test first on k3s first, then the other distros. k3d support coming soon.
Default Installed Applications
All of these can be disabled with the exception of Argo CD, which is optional, but if not installed, smol-k8s-lab will only install: MetalLB, nginx-ingress, and cert-manager.
| Application | Description | Initialization Supported |
|---|---|---|
metallb |
Loadbalancer and IP Address pool manager for metal | ✅ |
cilium |
Gateway API (a bit clunky right now, but getting better!) | ✅ |
Traefik |
Traefik is both an Ingress and Gateway API controller and we're currently switching to supporting it in addition to cilium since ingress-nginx is deprecated. | ✅ |
cert-manager |
For SSL/TLS certificates | ✅ |
Argo CD |
Gitops - Continuous Deployment | ✅ |
Argo CD Appset Secret Plugin |
Gitops - Continuous Deployment | ✅ |
ESO |
external-secrets-operator integrates external secret management systems like Bitwarden | ❌ |
Bitwarden ESO Provider |
Bitwarden external-secrets-operator provider | ✅ |
OpenBao |
OpenBao (Hashicorp Vault fork) - it's a Secrets Store. | ✅ |
ZITADEL |
An identity provider and OIDC provider to provide SSO | ✅ |
Prometheus Stack |
Prometheus monitoring and logging stack using loki/alloy, alert manager, and grafana | ✅ |
For a complete list of installable applications, checkout the default apps docs. To install your own custom apps, you can check out an example via the config file or learn how to do it via the tui.
Phases
Below we define each of the phases your app can be installed in.
Bootstrap
no metrics.
- metallb.yaml
- prometheus_crds.yaml
- cilium.yaml or traefik.yaml
- traefik.yaml or traefik.yaml
- ingress_nginx.yaml (deprecated 😞)
- certmanager.yaml
- k8tz.yaml
operator
yes metrics.
- external_secrets_operator.yaml
- k8up.yaml
- cnpg_operator.yaml
cd
needs both bootstrap and operators...
- argocd.yaml
auth
required for any sort of thing that uses oidc
- zitadel.yaml
default
requires bootstrap, operators, cd, and auth
- forgejo.yaml
Status
Somewhat stable and being actively supported, so if you'd like to contribute or just found a :bug:, feel free to open an issue (and/or pull request), and we'll try to take a look ASAP!
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file smol_k8s_lab-11.0.0.tar.gz.
File metadata
- Download URL: smol_k8s_lab-11.0.0.tar.gz
- Upload date:
- Size: 224.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/2.4.1 CPython/3.11.2 Linux/6.1.0-51-amd64
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
617e5f887cc5ec1cd9074b7e5e72e071ad9fc12726c4b6b22b511a02b34707e6
|
|
| MD5 |
bf6d6f5ed173ead43fc521684aec49b9
|
|
| BLAKE2b-256 |
3791d7272ea7b5629b456e8238b869d99953d0139a2c937eabdf0d5719043cca
|
File details
Details for the file smol_k8s_lab-11.0.0-py3-none-any.whl.
File metadata
- Download URL: smol_k8s_lab-11.0.0-py3-none-any.whl
- Upload date:
- Size: 329.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/2.4.1 CPython/3.11.2 Linux/6.1.0-51-amd64
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b693af06da799495511657db868ec35a113c281d51575f133c8f4efb13672611
|
|
| MD5 |
4087bb672a0cc8216915cb528e7d3f53
|
|
| BLAKE2b-256 |
475e5c38d94c35d92c19682d6e815d6bd10a70a3066bfca39c1fa2059d8facc1
|