Nothing Was Using the Cores: Three Idle Arm64 Boards, One HA k3s Cluster, and a BSD Build Farm

A.C. Jokela — Sun, 05 Jul 2026 01:30:00 GMT

The plan, at first, was to buy my way there. I wanted a three-node cluster, and the tidy way to get one is three identical machines — so I priced out two more Orange Pi 5 Max boards to stand beside the one I already had: same chip, same memory, same clean symmetry. Then I looked at the total.

We are, as I write this, deep in a memory and storage shortage, and the prices show it. DRAM and NAND have both climbed to where two more maxed-out single-board computers — sixteen gigabytes of LPDDR and a fast terabyte of NVMe apiece — added up to real money, for hardware that would spend most of its life, like everything else on that shelf, doing nothing. Buying idle capacity to set it next to idle capacity is a special kind of foolish, and I could feel that before I could quite justify it.

So before spending anything, I did the cheaper thing first — the thing I should have done to begin with. I took stock of what I already owned.

I had three small computers on a shelf, and on any given afternoon they were doing almost nothing.

That is not a complaint about the hardware. The hardware is lovely. There was an Orange Pi 5 Max — an RK3588 board, eight cores split four-and-four between big Cortex-A76s and little A55s, sixteen gigabytes of RAM, a neural processing unit, and a one-terabyte NVMe drive bolted underneath. Next to it, a Raspberry Pi 5, four cores and eight gigabytes and its own terabyte of WD_BLACK NVMe, quietly serving a web page. And a Compute Module 5 on a carrier, four cores, another sixteen gigabytes, two terabytes of NVMe — the fastest storage of the three and the least to do.

Add it up and it's sixteen cores, forty gigabytes of memory, four terabytes of solid-state storage. And the honest utilization graph, most of the time, was a flat line near the floor. The Orange Pi woke up when I asked it to run a local language model on its NPU. The Pi answered HTTP. The CM5 held its breath. Three machines, each doing one modest thing, each idle in the gaps — which is to say, idle almost always.

I have written before on this blog about hauling systems software onto hardware it was never supposed to run on, and there's a particular feeling that precedes those projects: a gap that's infuriating rather than merely inconvenient, a mismatch between what a machine plainly could do and what it was actually doing. This was that feeling, pointed inward at my own shelf. The boards were not underpowered. They were under-asked. And three of anything, on a network together, with fast local disks and nothing to prove — that is not three appliances. That is a cluster that hasn't been told yet.

So I told it.

Three boxes, one quorum

The goal was a real high-availability Kubernetes cluster, not a toy. For three small Arm boards that means k3s — Rancher's single-binary Kubernetes, which folds the API server, scheduler, controller manager, a container runtime, and networking into one process that starts in seconds and sips memory. On a fleet of SBCs it is the difference between "Kubernetes runs here" and "Kubernetes runs here and also leaves you room to run anything else."

High availability, specifically, is the reason there are three boards in this story and not two. Kubernetes keeps its entire state in etcd, and etcd is a consensus system: it stays available only as long as a strict majority of its members can see each other. With two nodes, a majority is two — lose either and the cluster's brain goes read-only. With three, a majority is two out of three, and you can lose any single node — reboot it, unplug it, drop its switch port — and the cluster keeps making decisions. Three is the smallest number where "highly available" stops being marketing. k3s can run etcd embedded across the server nodes, so all three boards became servers, all three became voting members of the quorum, and all three stayed schedulable for real work. No dedicated control plane sitting idle; the control plane is the workers.

Getting there was mostly smooth, with one wrinkle that everyone who has tried this on Raspberry Pi hardware eventually meets. Kubernetes needs the kernel's memory cgroup controller to enforce per-container memory limits, and the Raspberry Pi OS kernel ships with it switched off by default — a small performance optimization for people who aren't running containers, and a hard wall for people who are. The kubelet simply refuses to start. The fix is one line appended to /boot/firmware/cmdline.txt — cgroup_enable=memory cgroup_memory=1 — and a reboot, which I automated across the Pis with Ansible so the change was recorded rather than remembered. The Orange Pi's Armbian kernel already had the controller enabled, so it needed nothing. That asymmetry — same architecture, same job, different kernel defaults — is the texture of a homelab built from whatever excellent boards happened to be on sale.

I pointed k3s's data directory at the NVMe on each node instead of the default location on the boot media. etcd is unusually sensitive to storage latency — it fsyncs every write it commits — and the difference between a consensus log on an SD card and one on an NVMe drive is the difference between a cluster that feels sluggish and one that feels like a server. It also spares the SD and eMMC cards the write amplification that eventually kills them. Fast local disk is the one thing all three boards had in surplus, and the control plane was the workload most grateful to use it.

A short while later, kubectl get nodes returned three lines, all Ready, all control-plane,etcd, and the flat utilization line finally had somewhere to go. But an empty cluster is just a more elaborate way of being idle. The point was never Kubernetes. The point was the thing the Orange Pi had quietly been doing all along, badly.

NAME              STATUS   ROLES                AGE   VERSION
orangepi5-max     Ready    control-plane,etcd   25h   v1.36.2+k3s1
raspberrypi-cm5   Ready    control-plane,etcd   25h   v1.36.2+k3s1
rpi5              Ready    control-plane,etcd   25h   v1.36.2+k3s1

The build servers that never slept

Here is what had actually been running on that Orange Pi, underneath the occasional language model: three virtual machines, on all the time, existing only to compile software.

I maintain two projects that care a great deal about running natively on the BSDs. It is not because there is a huge BSD following for these projects. In fact, there are probably no dedicated BSD users. I do, however, have a soft spot for the BSDs. For nearly two decades, FreeBSD was my server operating system of choice. When I was into vintage hardware, like Sun Microsystems workstations from the 1990s, or the MicroVAX II I once owned, NetBSD was my preferred operating system.

One of the projects is a small programming language and its compiler, written in C, that I like to ship as real binaries for FreeBSD, OpenBSD, and NetBSD rather than telling people to build it themselves. The other is a ballistics engine written in Rust with the same ambition. Cross-compiling to the BSDs from Linux is possible in theory and miserable in practice — you need each target's system libraries, its linker quirks, its idea of where things live — so the honest way to get a native FreeBSD Arm64 binary is to build it on FreeBSD Arm64. The three VMs were exactly that: a FreeBSD guest, an OpenBSD guest, and a NetBSD guest, each an aarch64 machine running under QEMU with KVM acceleration, each holding a full toolchain, each reachable over SSH so a build script could push source in and pull a binary out.

They worked. And they were awake twenty-four hours a day to be useful for the few minutes a week I actually cut a release. Three virtual machines, two gigabytes of RAM apiece, pinned CPU, humming along through the night so that occasionally a compiler could run. It's the same waste as the idle cluster, just wearing a more respectable coat: capacity committed to a workload that is almost never present.

Once there was a cluster underneath them, the reshape was obvious. A build is a job — it starts, it runs, it produces an artifact, it ends. That is precisely the shape Kubernetes was built around. The always-on VMs wanted to become on-demand Jobs: nothing running when there's nothing to build, and when a build is needed, a pod springs up, does the work on real cores, and disappears.

The catch is the part that makes this a story instead of a paragraph. A Kubernetes Job runs a Linux container — a process sharing the host's Linux kernel. You cannot boot a FreeBSD kernel, or an OpenBSD one, or a NetBSD one, inside an ordinary container; there is no BSD kernel there to boot. The real BSD environment lives in those VM disk images, and the only honest way to build on a BSD is to run the BSD. So the design that survived contact with reality was a VM inside the Job: a privileged pod that mounts the host's /dev/kvm, boots the golden BSD image with QEMU at native speed, waits for the guest's SSH to come up, runs the project's own build script over that SSH connection exactly as before, copies the finished binary out to a persistent volume, and powers the guest down. The VM didn't go away. It stopped being a thing that's always on and became a thing that exists for the length of one build.

The pile of small, specific lies

The plan was clean. The path from plan to working was paved with the kind of bugs that only appear when real hardware stops being polite, and I want to walk through them, because each one is a small lesson and together they are the whole texture of the job.

The first was the most alarming and the most educational. I booted the FreeBSD image inside a pod, and QEMU came up spinning at four hundred percent CPU while the guest's console filled with synchronous exception panics — the Arm equivalent of a program tripping over an instruction the processor won't execute. The image booted fine on the host directly. It crashed reliably inside the pod. The difference was the RK3588's defining feature and, here, its trap: it is a big.LITTLE chip, four fast A76 cores and four efficient A55 cores, and they do not implement exactly the same instructions. QEMU was passing the host CPU straight through to the guest, and the Linux scheduler, seeing an unpinned QEMU thread, was free to migrate the virtual CPU from a big core to a little core mid-execution. The guest would start an instruction the A76 supports, get moved to an A55 that doesn't, and fall over. The host's own VM launch scripts had known this all along — they pinned QEMU with taskset -c 4-7, confining it to the four A76 cores — and my pod hadn't. One taskset and the synchronous exceptions vanished. The cluster's cleverest hardware feature was, for this workload, a thing to carefully switch off.

The second bug was a quiet philosophical one. My plan referred throughout to the BSD disk images as "golden images" — static, pristine templates you overlay a scratch copy on top of and never mutate. When I went to make that scratch overlay, QEMU refused: the image was locked, held open read-write by another process. The other process was the always-on VM. The "golden image" was the live disk of the running build server — not a template at all, but the working drive of a machine that was, at that moment, mid-thought. Overlaying a copy-on-write layer on a disk that's being written underneath you is not a snapshot; it's a data-corruption generator. And when I did force a copy, it failed its filesystem check on boot, because the source had been running for thirteen days without a clean shutdown and its dirty bit had never been cleared. The fix was to do the boring, correct thing I'd been skipping: cleanly shut each VM down, take a genuine point-in-time snapshot of the now-consistent disk into a real golden image, and restart the VM. The builds overlay those — static, clean, never open by anything else. It cost a couple of minutes of downtime per guest and bought a foundation that isn't quietly rotting.

Then the small, specific lies of virtual hardware. The first pod boot that got past the CPU pinning died instantly with QEMU complaining it couldn't find efi-virtio.rom — a network-card boot ROM that isn't packaged in a slim container image. The host scripts, again, already knew: they set romfile= empty on the virtual NIC, disabling the ROM the build doesn't need. Add that, and the guest booted further — and then OpenBSD, specifically, hung partway through, wedged on device probing. FreeBSD had sailed through the same configuration. The difference was vectors=0, another flag the host scripts carried and mine didn't, which disables MSI-X interrupts on the virtual devices; OpenBSD's virtio drivers, under emulation, needed the older interrupt style to come up at all. FreeBSD tolerated the modern path. OpenBSD did not. The host's launch scripts, it turned out, were a compressed archive of exactly these lessons, learned once and encoded as flags I had to re-learn by leaving them out.

The BSDs then taught me about their own disks. My build script staged source into the guest's home directory and immediately hit "No space left on device." OpenBSD, by long tradition, slices its disk into many small partitions, and its root filesystem — where a naive home directory lives — was four hundred megabytes and already full. NetBSD does the same, with a root barely over a gigabyte. FreeBSD hands you one big root and doesn't care. So the builder learned to build where the space actually is: on OpenBSD, in /usr/obj, the partition the system reserves for build output; on NetBSD, in /usr/pkg, its roomy second disk. Rust made this worse before it made it better, because Cargo unpacks its entire crate registry into a cache under the home directory, and on NetBSD that cache alone overflowed the tiny root the instant a real dependency graph arrived. Pointing CARGO_HOME at the big partition too, per guest, finally let a Rust project build on a NetBSD machine whose root filesystem couldn't have held its own dependencies.

The last of them wasn't a build bug at all; it was a networking cleanup that turned into an accidental repair. NetBSD's VM, unlike its siblings, had been bridged directly onto my LAN, while FreeBSD and OpenBSD used QEMU's simpler user-mode networking. I wanted them consistent, so I converted NetBSD to user-mode too — and in restarting it discovered that the bridged setup would have failed on its very next boot regardless, because a QEMU package update had silently stripped the CAP_NET_ADMIN capability off the bridge helper, the small privileged binary that lets an unprivileged VM attach to a bridge. The VM had been running for so long it hadn't been restarted since the update landed. The cleanup I wanted to do had been sitting on top of a latent failure I didn't know about, and converting away from bridging both simplified the design and defused a bomb.

None of these were deep. Every one was a mundane mistake or a mundane environmental fact, wearing for an afternoon the costume of something profound. That is the entire pattern of this kind of work, and I've made peace with it: the exotic theory that flatters your understanding almost always sits on top of a boring cause that flatters nothing. taskset. romfile=. A partition that's full. A capability an update took away.

When it was done, the payoff was concrete. A single command — make bsd-build PROJECT=lattice OS=freebsd — now fetches the latest source straight from GitHub, boots the golden FreeBSD image in a pod on the Orange Pi, compiles the project inside it, and drops a genuine aarch64 FreeBSD executable back onto my workstation, stripped and ready. The same command builds for OpenBSD and NetBSD, for both projects, and a release variant does all of it and publishes. Six native BSD binaries, produced by boards that used to keep three virtual machines awake all night to do the same thing on demand for no one.

Naming the tenants

With the builds turned into jobs, the always-on VMs had no reason to exist. I stopped and disabled them — reversibly; the golden images remain, and a systemctl away is the old world back — and the Orange Pi handed back roughly six gigabytes of RAM and its four A76 cores. Which returns us, neatly, to the theme this whole exercise keeps circling: reclaimed idle capacity is only worth reclaiming if you then ask something of it.

So here is the honest census of what actually runs on the cluster now — the jobs, named plainly.

On-demand BSD binary builds are the anchor tenant, the workload the whole thing was justified by: FreeBSD, OpenBSD, and NetBSD Arm64 binaries for the language compiler and the ballistics engine, each build a pod that boots a real BSD, compiles, and exits. Zero footprint between releases; real cores during them.

A full observability stack is the second tenant, and it exists partly because a cluster you can't see is a cluster you can't trust. It's the standard Prometheus-and-Grafana pairing, deployed the k3s-native way, with node exporters on every board reporting not just CPU and memory and disk but the thing that actually matters on fanless Arm boards crammed with NVMe: temperature. SBCs throttle when they cook, and a build farm that silently halves its own clock speed under thermal load is a build farm that lies to you about why it's slow. Grafana lives at a name on my LAN — grafana.localnet, resolved by an Unbound server on a little FreeBSD box that also runs my DNS — behind an ingress controller carefully arranged not to collide with the web page one of the Pis was already serving on port eighty. The stack pins its stateful pieces to the CM5, the node with the two-terabyte drive and the least else to do. The board that was holding its breath now holds the metrics.

Continuous fuzzing of the compiler is the newest tenant, and the one that most makes me glad the cores were sitting there. The language project ships nine libFuzzer harnesses — for the compiler front-end, three different bytecode VMs, the lexer, the formatter, and the JSON, TOML, and YAML parsers — and a fuzzer is the most gluttonous, most embarrassingly-parallel workload I own. It will eat every idle cycle you're willing to feed it and ask for more, forever, and every cycle it eats is a cycle spent trying to crash your code before a user does. On a cluster whose defining problem was too many idle cores, that is a nearly perfect fit. Each target gets its own continuous pod, spread across the three boards, building itself from the latest source and grinding against a corpus that persists and compounds on the NVMe — the first of the nine are coming online as I write this. And the results go somewhere I'll actually see them: when a fuzzer finds a crash, a small watcher reproduces it, fingerprints it so the same bug doesn't file itself twice, and opens a Jira ticket — the real bug tracker the project already uses — with the sanitizer backtrace and a base64 reproducer attached, tagged to the right component. Live throughput and coverage stream into the same Grafana the rest of the cluster reports to; a nightly job renders coverage as HTML behind that same ingress. Idle cores in; deduplicated, actionable bug reports out.

And underneath all of it, still, the Orange Pi's NPU runs a local language model when I ask it to — the original tenant, never evicted, just no longer the only one. Kubernetes shares the boards with it rather than owning them outright, which is exactly the relationship a homelab wants: the cluster is a tenant of the hardware, not its landlord.

The capacity was always there

There's a version of this post that's a shopping list — buy these boards, run these commands, get a cluster. I don't think that's the true version. The commands were the easy part; they always are. The hard part was the pile of small specific lies the hardware tells when you finally push it, and the honest bookkeeping of what's really running versus what's merely installed.

But the part I keep turning over isn't technical at all. It's that none of this required new hardware. Every core, every gigabyte, every terabyte of fast disk in this cluster was already sitting on my shelf, already powered, already mostly doing nothing. The Orange Pi was awake all night to compile software a few minutes a week. The CM5 was holding its breath. The utilization graph was a flat line near the floor, and I'd looked at it a hundred times without seeing it as a to-do item.

What changed wasn't the machines. It was the question I asked them. Three boards that each did one small thing became a single system that builds native binaries for three operating systems, watches its own temperature, and fuzzes a compiler around the clock — and the only thing I added was the willingness to treat idle capacity as a debt worth collecting.