Cluster services

k3s (cluster software)
step-ca (certificate manager)
cert-manager (cluster to step gateway)
traefik (web proxy/gateway)
Prometheus (metrics)
Grafana (metrics display)
Something for logs
Something for alerts?

I want tenants to be able to use mTLS internally, so cert-manager needs two providers, one for step and one for Lets encrypt.

Cluster tenants

mail
- postfix (smtp server)
- dovecot (imap server and email storage)
- spamassassin (spam filter)
- opendkim
- opendmarc
osric.uk
shinjuspottery
fluffypeople
kamelion

Cluster Management

helm?

That looks like a bunch of namespaces and then mail has a small collection of pods.

I'd like to give everyone their own ipv6 address, and I can front the web servers with mythics ipv4 proxy, only running mail through native ipv4.

Still, I'm not sure what it gets me. I want to be able to throw up a new service (and take it down later) with close to zero effort. I've newly got that now, but there is a bunch of setup on the server to do.

(I also want to split osric.uk down into a bunch of tiny services, but I'm worried about the overhead of running c# vms)

I think I can setup a helm chat for "deploy to server", and then just change the names. I'll look at that.

Ipv6 address toy - use a range tag to adjust the subnet, and show the start/end addresses underneath

New cluster is up and routed on merit.

K3s, again, this time with a public /64 via a VPN (Linux routing is harder than it looks).

I'm annoyed that Ipv4 is so expensive, it would be nice to give merit it's own ip. Options:

Pay for L2TP from A&A (£10/month)
Forward the appropriate ports from wepiu to merit
Remember that the plan is to put the cluster on wepiu anyway, so can use wepiu's ip
Find out how much mythic charge for extra ips

Support stuff:

Container registry (zot seems to work)
Internal CA, for mtls (cert-manager is designed for this)
Edge proxy (traefik is built in, so I'll take it)
Monitoring/Logging/Tracing/alerting
OIDC
Outbound mail?

I've been playing with a new k3s install the last week or so (if you didn't pick that up from the last few posts), but I've been stalled for a few days. Tonight I discovered why.

K3s comes with traefik (an edge proxy), and a slightly complicated way of configuring it. K3s ships with a basic configuration, and then the cluster admin (me, in this case) needs to add an extra file to overwrite the default.

I thought I was going to need to reinstall the cluster every time I wanted to update the config, and I was dreading that as I tend to work in a "make a small change, measure it's effect" loop, and I didn't want to wait five minutes for the install to finish every time.

So that was wrong, all I need to do is run a kubectl apply command with the updated config file and k3s makes the change and restarts traefik.

I even wrapped the command in a Makefile, to make it even easier to deploy.

Conclusion: Automation really is the bees knees.