My First Post

2026-04-12T12:00:00+02:00

I've migrated this blog to be a part of my mono-repo which controls my homelab. The repo runs GitHub actions to automatically take modifications to my blog posts and render them with Pelican, a Python-based static blog generation package.

In order to stand out a bit, I created a custom Pelican theme using Claude code so the blog looks a bit better. My days of dealing with CSS are long behind me and I'm not a great designer, so I'm happy to use Claude for these types of tasks.

With this better workflow, I hope to be more motivated to create blog posts about my homelab this year!

My Home Lab v1.0/v1.5

2026-04-12T12:00:00+02:00

In 2024, I was generally ignorant of Kubernetes from a systems administration standpoint. Several years ago I resolved to increase my understanding of K8s by deploying my own bare-metal cluster onto my homelab from scratch (meaning: without kubernetes-in-a-box solutions like k3s, docker desktop, etc...). In this blog post, I will share the lessons that I have learned over the past few years and demonstrate how seemingly minor decisions made when first starting out can have outsized impacts on how your cluster works.

Initial Hardware Overview

My first crack at administering K8s came when I decided to take the plunge by converting all the VMs and applications on my Proxmox server into workloads in K8s. My server setup was:

Component	Description
OS	Proxmox
CPU	Intel i5-10400
RAM	32Gb DDR4
Primary Storage	1Tb PCIe 3.0 NVMe
Secondary Storage	x4 2Tb WD-Red NAS HDDs

K8s Cluster v0.1

The Proxmox server was turned into a 1-node K8s cluster running the following stack:

Component	Description
Container Runtime	runc
Container Network	Calico
Container Storage	local-provisioner
Load Balancing	MetalLB
Ingress	ingress-nginx

This, I felt, was a fairly simple yet complete approach. While I had no "real" container storage interface, a one-node kubernetes cluster has no real need of a more complex storage solution, and I was happy to just be running MetalLB-backed services to expose things to my local network using BGP with my Ubiquity EdgeRouter 12.

While this solution was a servicable way of running my very first workloads and services inside of Kubernetes, I quickly outgrew it in my desire to hew as closely as reasonable to the technology stack at CoreWeave as a priority.

I purchased several additional nodes between June and December and began constructing what would end up being a 5-node kubernetes cluster.

Getting to Infrastructure as Code

(future)

40Gbps of Infiniband Ethernet

Towards the back half of 2024 I began purchasing Mellanox ConnectX-3 cards, specifically the CX354A FCBT card. It is available cheaply: ~$15 for the card and another ~$15 for the Direct Attach Copper (DAC) cable will give you a functional 40Gbps Ethernet card if you've got a "Smart" infiniband router like the Mellanox SX3036 (~$160). "Simply":

add the license for ethernet gateway to the SX3036
turn the SX 3036 Host Channels into Ethernet mode
purchase a cable that will bust down the 40Gbps to 4 10Gbps channels
connect to your 10gbps router

and you'll have 40Gbps ethernet for cheaper than the aggregate cost of converting several hosts to 10Gbps ethernet equipment.

Adding GPU Nodes

(future)

Hyperconverged Ceph

I strongly recommend that you not use Rook to deploy a Ceph cluster on the same infrastructure you are leveraging to run Kubernetes. Kubernetes nodes are (supposed to be) fungible in a way which Ceph does not handle well unless you're operating at scales far beyond what most homelab budgets are meant for.

My stabliity experience with Ceph was rather miserable: A host would go down, perhaps from Out-of-memory issues due to a poorly coded or large workload on a host. That outage, if it was prolonged, would cause Ceph to re-balance disks. Then the host would come back, and the cluster would re-re-balance itself. This back-and-forth in an early homelab cluster with non-homogenous equipment lead to overworked disks on consumer-grade systems. Combined with early failures of older equipment this caused even more stability issues which eventually lead me to migrate off of Ceph for my Container Storage Interface. It became too regular of an occurance for services in my network to be down and maintenance of Ceph was distracting from my core goal of learning practical Kubernetes administration.

Migrating to NFS

When I decided to leave Ceph, I was leaving a rather featureful ecosystem where adding disks was as easy as finding an open bay and SATA port on one of 5 desktops. Adding disks scaled IOPS as well as storage, and the only growth boundary was my power budget since surplus desktops and hardware was cheap into late 2024 and early 2025.

In leaving Ceph, I could go to other clustered storage systems, but as mentioned: hyperconvergence in the homelab is more of a liability than an asset. I resolved to migrate to either a single non-Kubernetes storage mule like a disk shelf, or an all-in-one solution. I eventually chose the latter, opting for a Frankenstien's monster solution of an old Supermicro disk shelf with a newer SuperMicro motherboard.

Interestingly, SuperMicro chassis are ATX standards-compliant and the screw layout is user-changable (easily, I might add). I was able to place a dual-CPU, 256Gb DDR4 system in the old chassis with the same power supplies and with zero pain.

I installed Proxmox, shoved in some surplus 12Tb SAS drives, made ZFS pools, installed NFS, exported it to the cluster, and then installed the NFS container storage interface and was off to the races. That is, until I had to migrate the existing Persistent Volumes to the new StorageClass.

Crypt0s.com