🔧 My Homelab: from a single Raspberry Pi to a Proxmox cluster and a dedicated NAS

In 2020, I began my homelab journey (even though I never heard of the term “homelab” at that point) with a simple yet powerful setup: Home Assistant running on Proxmox on a NUC8i3BEH. My Home Assistant instance, with over 100 automations and about 20 addons (docker containers), had outgrown the capabilities of my Raspberry Pi 4, leading me to upgrade to the NUC8i3BEH. You can check out my Home Assistant configuration.

This system was more than sufficient for running Home Assistant—arguably overkill. However, because I passionately develop addons and custom components, I needed the ability to quickly restart and reload everything while debugging. For a long time, I ran Proxmox, mainly because the Home Assistant community recommended it, but I only used it to host one extra Ubuntu VM and Home Assistant OS.

In 2022, I stumbled upon the Proxmox VE Helper-Scripts by tteck, which got me interested in the whole homelabber/self-hosting community. These scripts opened a new world of possibilities. With them, I expanded my Proxmox environment to run 25 different VMs and LXC containers, each performing a variety of tasks, from DNS servers and reverse proxies to local dashboards and backup software.

Initially, I used an external 2 TB drive for storage, but it quickly became insufficient as my data needs grew. After some research, I found the HP EliteDesk 800 G4 SFF, a renewed enterprise-grade desktop with slots for two 3.5-inch drives. At approximately $160 on Amazon, it offered excellent specs for a budget-friendly price. I decided to create a TrueNAS VM on this machine using Proxmox and passed through the two 16TB renewed enterprise-level HDDs (via the PCIe SATA controller), which I had purchased from serverpartdeals.com. I chose these drives based on their reliability as reported in the BackBlaze drive stats.

When I got the HP, I created a Proxmox cluster and was very excited about how easy it was to migrate containers and VMs from one machine to the other. However, I soon encountered a significant drawback: enterprise drives are notably loud. Since I had my Proxmox “cluster” (just the two machines) running behind the TV, the loudness was noticeable and disruptive. This noise was just the beginning of my troubles. The NFS server on the TrueNAS VM often crashed, causing the entire Proxmox system to lock up. Initially, I mounted the NFS drives directly on the Proxmox host and used bind-mounts to pass through the drives. Due to the locking behavior of NFS, any problem would cause the entire system to lock up. To prevent this, I started to use autofs to mount the drives in the LXC containers and VMs instead. This worked fine until Proxmox started its periodic backups. Any issue with the NFS during backup caused the whole process to hang, locking up the VMs and LXC containers. This could be partially avoided by using the “stop” backup mode, but it still resulted in almost daily debugging sessions where I had to restart Proxmox-related services.

After running all services as LXC containers, I realized that updating them and keeping up with all security updates was labor-intensive. So, I started to look into migrating everything to Docker Compose. This transition turned out great because I could manage my config files in a git repo and keep the relevant data separate, streamlining updates and maintenance.

To resolve these issues and because I was having a lot of fun, I decided to build a dedicated NAS running TrueNAS Scale, not that I really needed one, but mostly because I could. This new setup included high-performance components, ensuring both stability and quiet operation. I used “Manufacturer Recertified” Western Digital Ultrastar DC HC550 HDDs, known for their reliability and performance. I have 6 x 16TB = 96TB now with the option to add 2 more 3.5" drives, 2 nvme M.2 drives, and 1 SATA 2.5" SSD. Now, I run all services that previously relied on NFS locally on TrueNAS via a Debian systemd-nspawn container managed with a new script called Jailmaker, which leverages the systemd-nspawn program in TrueNAS Scale (not to be confused with FreeBSD jails). Inside this systemd-nspawn container, I run Debian with Docker. For managing Docker Compose files, I use Dockge, which provides a nice web UI for easy management.

Along the way, I got familiar with ZFS and grew to love it for its amazing snapshotting and data integrity features. ZFS has been a game-changer, providing robust data protection and ease of management. Initially, I used mirrored vdevs, but I recently switched to RAIDZ2 for the added storage capacity and the promise of OpenZFS 2.3 supporting expansion of RAIDZ vdevs. I also use ZFS for my Proxmox VMs and LXC containers, whose backups I store on the NAS via rsync in a cronjob to another jail running plain Debian (no more NFS on the Proxmox host for me…)

This journey from a simple NUC running Home Assistant to a Proxmox cluster and dedicated NAS has been very time-consuming but a lot of fun. The lessons learned and the improvements in stability and performance have made the effort worthwhile. If you’re considering building your own homelab, investing in reliable hardware and being prepared for a bit of trial and error can make all the difference.

Lessons learned

• Run your services in Docker containers, not LXC containers, for easier management and updates.
• Do not have virtualized NAS (TrueNAS VM) on Proxmox, it’s a bad idea if you are also sharing the drives with Proxmox via NFS. There are many folks on the internet who run their NAS in a VM on Proxmox, seemingly without issues, but I was not one of them.
• Do not encrypt the root ZFS dataset! Encrypt its children instead.
• Use passphrase encryption instead of a generated key. It allows you to use the UI and rotating keys.
• Do not remove the physical drive that is passed through in a Proxmox VM, because Proxmox will not boot properly afterwards.
• Do not use random 100 character passwords for your Proxmox, you have only 60 seconds to enter it during boot.
• When making Proxmox backups, send them to a different physical machine, not only the same machine that is being backed up.
• When building a NAS or home server, consider noise levels, especially if the equipment will be in a living area. Enterprise-grade drives can be surprisingly loud.
• When using Intel I226-V 2.5GbE controllers, be prepared for potential issues.
• When planning your NAS storage size needs, remember to account for the extra space needed for redundancy (e.g., in RAIDZ2). Additionally, keep in mind that ZFS performs optimally when drives aren’t filled to capacity. It’s wise to plan for more storage than you think you’ll immediately need.
• Buy as many used enterprise-grade hardware components as you can, they are often cheaper and more reliable than consumer-grade components.

Components

For completeness, here are the components for each machine in my homelab:

NUC Components

HP Components

Notes on the HP components:

• Initially, I planned to use this machine as a NAS because it has two 3.5" drive bays, 2 M.2 slots, and the disc drive bay can be converted to a 2.5" drive bay. However, the 2 x 16TB drives I bought quickly filled up because I used a mirrored vdev, so I decided to build a dedicated NAS instead.

NAS Components

Notes on the NAS components:

• The 13500T is a 35W TDP CPU I bought on eBay, which will help keep the system cool and quiet.
• I bought the motherboard from AliExpress because it seems to be the only mini-ITX motherboard that supports 8 SATA drives, and also comes with 2x2.5GbE ports, 3xM.2 slots, and a full sized PCIe slot.
• The motherboard uses Intel I225-V 2.5GbE controllers, which are notoriously bad, initially iperf3 maxed out at 100Mbps. The internet is full with people complaining about these controllers auto-negotiating to 100Mbps, however, for me it showed 1Gbps. After trying all suggestions I could find (there are few), as a last resort I connecting the NAS directly to my ASUS XT8 router instead of the NETGEAR GS305 switch, and now I got 1Gbps speeds. So I bought a new managed TL-SG108E switch, which now also gives me 1Gbps speeds. I’m not sure what the issue was, but I’m happy it’s resolved.