Ran out of infrastructure titles

Hell, where do I even start? My last "full" infrastructure update post was at the end of 2023, and we're now a third of the way through 2025. In that time, I've been doing a lot of tinkering and refining of my personal infrastructure, to the point I've very much lost track of what's changed besides the commits to my infra repository and posts to this very blog.

One of the big things I've done recently is entirely dropped Nix and NixOS from my toolset. This isn't a permenant decision, and I do keep an eye on it and the forks, but frankly I wasn't thrilled with some of the decisions that were being made around it and opted instead to move away from it entirely. This works great for me though - my infrastructure, at its heart, is a place for me to tinker and is full of pet server, not cattle servers, and sometimes the nature of Nix got in my way. My desktop runs Gentoo, which should really sums up the tinkering side of me. My servers are mostly running Debian now, which is very much a comfort Linux distro for me.

If Nix is something that interests you, go for it! I recommend the Lix fork. I do still like the concept of Nix, and I'll talk a bit about how I'm doing some declarative(ish) server configuration, but being able to tinker with a more "traditional" Linux system is more valuable to me at the moment.

My NAS (Vancouver) and primary Proxmox (Kelowna) server are both running Proxmox. They aren't configured for high availability, but configuring them into a sort of cluster means I only need to care about one web interface. Linux containers (LXCs) and virtual machines (VMs) are created depends on the workload requirements (storage versus performance) - for example Plex runs in an LXC on Vancouver with the GPU assigned and the media directory mapped to a directory inside the LXC, while Kelowna's primary focus is Windows-based stream boxes and Kubernetes control planes. Generally these LXCs and VMs require minimal maintenence to keep running, the odd apt update && apt upgrade here and there. To help simplify things, I've started investing a bit of time into setting up my own Salt master and minions. Salt is similar to Ansible, in the sense you use YAML to specify a state, but has a differening architecture, opting for a master/minion setup rather than Ansible's agentless SSH connections to each node. While Ansible is generally more popular among homelabbers, Salt is what is used at work to manage servers and so in the interest of actually learning Salt outside of the context of massive statefiles built up over years, I chose it instead. At the moment my highstate only cares about setting up the apt repository cache on the Debian and Ubuntu machines, but it's also incredibly useful to be able to run salt '*' pkg.upgrade refresh=True --async and have all the packages across the machines update.

Running a NAS off Proxmox isn't too difficult. Proxmox supports ZFS out of the box, which I was already using, and since Proxmox is a fairly vanilla Debian install with some niceties installed for managing VMs and containers, the experience is pretty close to what you'd expect - I can mount directories using either Samba (for my partner's Windows machine), SSHFS (my preferred mounting method), or NFS (only for my Kubernetes PVCs), and spin up new workloads pretty easily. The difficult part is mapping directories from the host to a container, while maintaining the user/group mapping so that other containers or machines on the network can access things smoothly.

An ever evolving list for sure. The Kubernetes directory in my infra repository is usually exactly what's running in my cluster, but on the Proxmox side of things I don't have the same versioned configuration. So a quick list of useful things on my Proxmox nodes:

• Harbour for caching container images. Generally really speeds up container creation in the Kubernetes cluster and avoids potential rate limiting.

• MySpeed for monitoring internet speeds by running a speed test every hour. This used to be a speedtest-exporter instance but MySpeed gives me a nice little web interface along with the Prometheus stats.

• Librespeed for testing connectivity back to my home network. Usually through Tailscale to get a sense of the latency and throughput to my NAS.

• Syncthing for syncing various things together. Not techncially self hosted but rather a node in the network. Does a continous sync of various directories on my desktop and phone which in turn get backed up offsite.

• Immich is one I've had my eye on since it was announced. I was on Photoprism for a while, but the user experience wasn't quite as polished. I do not, however, sync photos with the Immich iOS app, and instead use Syncthing to do so.

• Jellyfin as a backup to Plex. I do want to move off Plex at some point but I've struggled with some of the UX of the web and mobile apps availabe, and I'd love to be able to federate with other Jellyfin servers to share media libraries, much in the same way Plex can.

• Plex for accessing media. It's just... well it's not polished but it's a bit nicer than Jellyfin in some aspects. Just don't get me started on the busted download functionality.

• Forgejo for storing code! I sometimes mirror code to GitHub, but anything I work on will be pushed to my forgejo instance.

• Audiobookshelf for listening to podcasts. For a long while I was using PocketCasts, but was frustrated by their web API being slow to pick up new episodes.

• apt-cacher-ng for caching Debian and Ubuntu packages.

• HomeAssistant, although I'm mostly using it to bridge some cameras into HomeKit. Definetly not using it to the fullest extent.

• SaltGUI because I'm a Salt noob and having an interface rather than issuing commands manually is really helpful.

I route to these services with Caddy, one instance on each with the reverse proxy setup on whichever node the service is physically on. I held on to nginx for a long while, but when I swapped to Contour in my Kubernetes cluster (more on that later) I realised that hey, I can use Caddy. It's way, way, way easier.

In an effort to "give back" to the world, I've been slowly opening up some of the services I self host to the wider internet. Primarily I've been maintaining semi-public Nitter and Redlib instances, providing an alternative frontend, and by extension proxied connections, to Twitter and reddit. Unfortunately because of the absolute wave of machine learning scraping bots, I've had to challenge every request to these to make sure they're usable and not rate limited to high hell - while I was doing fine tuning of the filters of what to challenge, it got too overwhelming as bots shifted their fingerprints and user agents.

Are you also suffering from excessive bots scraping your sites? Try Anubis!

Since I host so many services for myself, logging in has become a bit of a pain. A long while ago, however, I setup my own authentik instance, which runs in the Kubernetes cluster. While authentik is perhaps quite a bit heavier and more flexible than I actually need, it does just work and has been audited. I have been considering a move to Pocket ID in the future, but am holding off until I has some third party auditing done since it's critical to auth and is exposed to the internet.

My Kubernetes cluster has grown as well. I've outgrown k3s and opted for Talos instead - somewhat satisfying the declarative void that Nix left. The cluster includes three control plane nodes (two on Kelowna, one on Vancouver), one worker node on each Proxmox machine, my ever-present Raspberry Pi 4b, and a VM on my desktop with my GPU passed through. Each VM has 6 cores and 8GiB of memory (the memory matching the Pi). This has given me ample compute for messing around with new self hosted apps or Kubernetes itself. If you've been following, you'll know I put a OnePlus 6T into the cluster not long ago, but it's missing from the list. I've had on-again off-again issues with it, usually requiring a re-flash of postmarketOS and rebuilding of a custom kernel to get working. Part of the issue was system updates would frequently bork the install, which I suppose is the risk that comes with running edge builds. The other part was the very... odd filesystem issues when it came to kubelet mounting directories or files, e.g for volumes or subPath volume mounts. I suspect it has something to do with the root filesystem being UFS, and figure an extenal SSD for kubelet data would help, but I haven't had a chance to experimenent. Thus, it is out of the cluster, for now.

Let's talk about Kubernetes, while I have you here. For a long while, my go-to ingress solution was fairly straightforward - throw ingress-nginx in the cluster as a DaemonSet with host networking, leverage the Ingress resource, done. But the writing is sort of on the wall when it comes to Ingress resources, and ingress-nginx specifically. The Kubernetes community considers the Ingress resource to be feature complete, and that's fair enough - it does what it needs to do fairly well, but the new hype is the Gateway API and the flexibility it offers. I may dive into my feelings on this elsewhere, but regardless, I made the decision to move to Contour, which uses Envoy under the hood. Wrapping my head around how Envoy and Contour are related, while other controllers also use Envoy, was a bit tricky, but the way I see it is like this - Contour is actually the configuration provider for Envoy, with opinionated defaults for its workload. At least, that's my current mental model.

Regardless - moving to Contour meant that my was of using the ingress-nginx controller had to change as well. Contour can use Ingress resources, but my configuration used a single wildcard certificate for all *.gmem.ca domains. I couldn't figure out how to properly configure this in Contour, and after getting frustrated the YOLO fuse in my brain tripped and I wrote some Python to turn all my Ingress resources into HTTPRoute resources from the Gateway API. I could then replicate the end product of what I had with ingress-nginx, albeit with some slightly more verbosity.

I mentioned that I have a small VM on my desktop to give some more compute to the cluster - this isn't anything special besides having my desktop's discreet GPU passed through to it for GPU workloads. I have yet to find a good use for this besides running Ollama and tinkering with machine learning, but even that is rarely used. It's also not always running, as I have to power it off to power on my gaming virtual machine since the GPU can only be used by one or the other. I don't really plan to do this long term, but it's kind of nice to be able to slice off a bit of compute that would otherwise be unsued when I'm not actively working something.

As part of this move, I had to change zero DNS records! This is because all traffic to my cluster flows through a dedicated HAProxy instance, which I keep in sync with my cluster with a bit of Python. Since the Contour deployment uses a NodePort service rather than host networking, the service is given assigned a random port that each node listens to, and then HAProxy sends traffic to whichever node has the least load (in theory). However, I have done significant work streamlining my DNS setup...

I've detailed this elsewhere, but as a quick recap, there are three possible answers for a DNS query to some of my domains, depending on my connectivity. If I'm on my Tailnet, the answer will likely be the relevant machine's Tailscale IP. If on my home subnet, it'll be the machine's local subnet IP. And if I'm out and about (and what most of you will hit), a Cloudflare Tunnel. To keep this all in order, I opted for OctoDNS - while Terraform is an option, OctoDNS configuration is YAML that is far easier to parse, which leads to two scripts in my infra repository. They take in the octoDNS configuration, and make changes to the relevant NextDNS (referral) profile, adding an rewrite from whatever the default is to whatever makes the most sense of the network I'm on. So for my Tailscale profile, *.ts.net entries, home profile .gmem.ca subdomains, and whatever else. My initial iteration of this system caused some headache, as the resolvers I was using didn't actually recurse, so rewrites hardcoded the IP address. In my effort to dual-stack my homelab and learn IPv6, this drove me a bit insane, and while looking at options I realised I could replace my dnsmasq "edge cache" instances with Unbound, which is a recursive DNS resolver! Problem solved! It's generally a little more robust on the DNS front.

You may be wondering why I run an edge cache for DNS. It's mostly for fun, as the speed to query my local instance is about 1/5th the time as querying NextDNS directly.

Kind of a boring part of my infrastructure at this point, so there's not much to mention. It just works. I have considered setting up Headscale as an escape hatch in case Tailscale implodes, and do keep an eye on alternatives, but frankly it's not something I think about much.

Keeping an eye on my infrastructure is typically easy - since I use it daily, if something breaks I'll know pretty much immediately. But sometimes something about to break can be detected, or I want some historical metrics to gauge whether I need to work on an optimisation. For that I continue to stick with Prom- actually, no! I've moved to victoriametrics, which has proven to be far more resource effecient than Prometheus, for the same functionality and longer data retention. An agent (vmagent) runs in my Kubernetes cluster, and ships metrics it scrapes off to the main instance, which is running on a small Hetzner ARM instance. This instance also runs Uptime Kuma, for some more basic uptime metrics and status pages, Grafana for viewing the metrics, and Healthchecks to notify me if jobs don't "check in" when they start and finish. All this together, with some alerting rules, means that I can at least be yelled at when something breaks rather than relying on me using it when it is broken. Various metrics exporters are scraped to gather data from their respective services, and most of my machines are running the node_exporter project so that their resource usages can be easily tracked. Because of the nature of the monitoring server, I haven't moved it off NixOS, and it continues to cruise along without issue.

Something that is key to this monitoring server is that it's completely remote, so that if my home network goes down I can still keep an eye on things elsewhere, or be notified that my home network is completely offline. It's still located in Europe, but if Europe goes down my monitoring server is the least of my problems.

Backups are fairly straightforward - anything important will get synced with or sent to Vancouver, which is running restic with backrest as a web frontend so I can easily setup new jobs and keep an eye on what's been backed up. Important data is then backed up nightly to an S3 compatible target, and since restic only stores changes to files I don't need to worry about things getting too unruly. It's already saved me a few times, when I needed to grab a long deleted file. It was as simple as using the restic mount command and browsing to a snapshot I knew would have the file.

If a backup fails, I'll get notified by the aforementioned Healthchecks instance, so I can dig in and find out why it's failed to complete or not run at all. Always test and verify your backups folks.

In hours of my life, a lot. But hardware wise... that's tricky. The hardware I have has been accumulated over the course of the last 5 years, either purchased new, bought second hand or burdened given for free. While I do have a PCPartPicker list of the hardware, the pricing isn't accurate, and doesn't account for supporting bits like my router, cables, etc. So the answer is probably somewhere between £4,000 and £6,000 at a random ballpark. It's difficult to keep track when it's accumulated slowly over time. And that's not how much a homelab costs anyways! There are a ton of options for getting your own up and running. It's less about the speed or power of the homelab, and more about the opportunities to learn and tinker. It is, after all, a lab.

Take whatever lessons you want from my setup. It's not really intended to be a blueprint for anyone, but is hopefully a nice source of inspiration. It's also never finished, since by nature of it being a lab (I emphasise that a lot) I am constantly expirementing. I'm sure I've missed something in this post, so if you have any questions feel free to ask! My goal is to setup a more regularly updated wiki rather than continously posting blog posts, so keep an eye out for that.

And yes, I make sure to give my servers plenty of headpats every day.