TinyComputers.io (Posts about ci)

Nothing Was Using the Cores: Three Idle Arm64 Boards, One HA k3s Cluster, and a BSD Build Farm

A.C. Jokela — Sun, 05 Jul 2026 01:30:00 GMT

Update (August 2026): This cluster has since grown from three nodes to eight, and it is no longer all Arm — two x86-64 boxes have joined the quorum's company, along with the NanoPC-T6N I reviewed a few weeks later. One of the two projects those BSD builds exist for — the ballistics engine — now has a post of its own: what it actually computes, how you test physics when there is no oracle to check it against, and why it ships to thirteen platforms.

The plan, at first, was to buy my way there. I wanted a three-node cluster, and the tidy way to get one is three identical machines — so I priced out two more Orange Pi 5 Max boards to stand beside the one I already had: same chip, same memory, same clean symmetry. Then I looked at the total.

We are, as I write this, deep in a memory and storage shortage, and the prices show it. DRAM and NAND have both climbed to where two more maxed-out single-board computers — sixteen gigabytes of LPDDR and a fast terabyte of NVMe apiece — added up to real money, for hardware that would spend most of its life, like everything else on that shelf, doing nothing. Buying idle capacity to set it next to idle capacity is a special kind of foolish, and I could feel that before I could quite justify it.

So before spending anything, I did the cheaper thing first — the thing I should have done to begin with. I took stock of what I already owned.

I had three small computers on a shelf, and on any given afternoon they were doing almost nothing.

That is not a complaint about the hardware. The hardware is lovely. There was an Orange Pi 5 Max — an RK3588 board, eight cores split four-and-four between big Cortex-A76s and little A55s, sixteen gigabytes of RAM, a neural processing unit, and a one-terabyte NVMe drive bolted underneath. Next to it, a Raspberry Pi 5, four cores and eight gigabytes and its own terabyte of WD_BLACK NVMe, quietly serving a web page. And a Compute Module 5 on a carrier, four cores, another sixteen gigabytes, two terabytes of NVMe — the fastest storage of the three and the least to do.

Add it up and it's sixteen cores, forty gigabytes of memory, four terabytes of solid-state storage. And the honest utilization graph, most of the time, was a flat line near the floor. The Orange Pi woke up when I asked it to run a local language model on its NPU. The Pi answered HTTP. The CM5 held its breath. Three machines, each doing one modest thing, each idle in the gaps — which is to say, idle almost always.

I have written before on this blog about hauling systems software onto hardware it was never supposed to run on, and there's a particular feeling that precedes those projects: a gap that's infuriating rather than merely inconvenient, a mismatch between what a machine plainly could do and what it was actually doing. This was that feeling, pointed inward at my own shelf. The boards were not underpowered. They were under-asked. And three of anything, on a network together, with fast local disks and nothing to prove — that is not three appliances. That is a cluster that hasn't been told yet.

So I told it.

Three boxes, one quorum

The goal was a real high-availability Kubernetes cluster, not a toy. For three small Arm boards that means k3s — Rancher's single-binary Kubernetes, which folds the API server, scheduler, controller manager, a container runtime, and networking into one process that starts in seconds and sips memory. On a fleet of SBCs it is the difference between "Kubernetes runs here" and "Kubernetes runs here and also leaves you room to run anything else."

High availability, specifically, is the reason there are three boards in this story and not two. Kubernetes keeps its entire state in etcd, and etcd is a consensus system: it stays available only as long as a strict majority of its members can see each other. With two nodes, a majority is two — lose either and the cluster's brain goes read-only. With three, a majority is two out of three, and you can lose any single node — reboot it, unplug it, drop its switch port — and the cluster keeps making decisions. Three is the smallest number where "highly available" stops being marketing. k3s can run etcd embedded across the server nodes, so all three boards became servers, all three became voting members of the quorum, and all three stayed schedulable for real work. No dedicated control plane sitting idle; the control plane is the workers.

Getting there was mostly smooth, with one wrinkle that everyone who has tried this on Raspberry Pi hardware eventually meets. Kubernetes needs the kernel's memory cgroup controller to enforce per-container memory limits, and the Raspberry Pi OS kernel ships with it switched off by default — a small performance optimization for people who aren't running containers, and a hard wall for people who are. The kubelet simply refuses to start. The fix is one line appended to /boot/firmware/cmdline.txt — cgroup_enable=memory cgroup_memory=1 — and a reboot, which I automated across the Pis with Ansible so the change was recorded rather than remembered. The Orange Pi's Armbian kernel already had the controller enabled, so it needed nothing. That asymmetry — same architecture, same job, different kernel defaults — is the texture of a homelab built from whatever excellent boards happened to be on sale.

I pointed k3s's data directory at the NVMe on each node instead of the default location on the boot media. etcd is unusually sensitive to storage latency — it fsyncs every write it commits — and the difference between a consensus log on an SD card and one on an NVMe drive is the difference between a cluster that feels sluggish and one that feels like a server. It also spares the SD and eMMC cards the write amplification that eventually kills them. Fast local disk is the one thing all three boards had in surplus, and the control plane was the workload most grateful to use it.

A short while later, kubectl get nodes returned three lines, all Ready, all control-plane,etcd, and the flat utilization line finally had somewhere to go. But an empty cluster is just a more elaborate way of being idle. The point was never Kubernetes. The point was the thing the Orange Pi had quietly been doing all along, badly.

NAME              STATUS   ROLES                AGE   VERSION
orangepi5-max     Ready    control-plane,etcd   25h   v1.36.2+k3s1
raspberrypi-cm5   Ready    control-plane,etcd   25h   v1.36.2+k3s1
rpi5              Ready    control-plane,etcd   25h   v1.36.2+k3s1

The build servers that never slept

Here is what had actually been running on that Orange Pi, underneath the occasional language model: three virtual machines, on all the time, existing only to compile software.

I maintain two projects that care a great deal about running natively on the BSDs. It is not because there is a huge BSD following for these projects. In fact, there are probably no dedicated BSD users. I do, however, have a soft spot for the BSDs. For nearly two decades, FreeBSD was my server operating system of choice. When I was into vintage hardware, like Sun Microsystems workstations from the 1990s, or the MicroVAX II I once owned, NetBSD was my preferred operating system.

One of the projects is a small programming language and its compiler, written in C, that I like to ship as real binaries for FreeBSD, OpenBSD, and NetBSD rather than telling people to build it themselves. The other is a ballistics engine written in Rust with the same ambition. Cross-compiling to the BSDs from Linux is possible in theory and miserable in practice — you need each target's system libraries, its linker quirks, its idea of where things live — so the honest way to get a native FreeBSD Arm64 binary is to build it on FreeBSD Arm64. The three VMs were exactly that: a FreeBSD guest, an OpenBSD guest, and a NetBSD guest, each an aarch64 machine running under QEMU with KVM acceleration, each holding a full toolchain, each reachable over SSH so a build script could push source in and pull a binary out.

They worked. And they were awake twenty-four hours a day to be useful for the few minutes a week I actually cut a release. Three virtual machines, two gigabytes of RAM apiece, pinned CPU, humming along through the night so that occasionally a compiler could run. It's the same waste as the idle cluster, just wearing a more respectable coat: capacity committed to a workload that is almost never present.

Once there was a cluster underneath them, the reshape was obvious. A build is a job — it starts, it runs, it produces an artifact, it ends. That is precisely the shape Kubernetes was built around. The always-on VMs wanted to become on-demand Jobs: nothing running when there's nothing to build, and when a build is needed, a pod springs up, does the work on real cores, and disappears.

The catch is the part that makes this a story instead of a paragraph. A Kubernetes Job runs a Linux container — a process sharing the host's Linux kernel. You cannot boot a FreeBSD kernel, or an OpenBSD one, or a NetBSD one, inside an ordinary container; there is no BSD kernel there to boot. The real BSD environment lives in those VM disk images, and the only honest way to build on a BSD is to run the BSD. So the design that survived contact with reality was a VM inside the Job: a privileged pod that mounts the host's /dev/kvm, boots the golden BSD image with QEMU at native speed, waits for the guest's SSH to come up, runs the project's own build script over that SSH connection exactly as before, copies the finished binary out to a persistent volume, and powers the guest down. The VM didn't go away. It stopped being a thing that's always on and became a thing that exists for the length of one build.

The pile of small, specific lies

The plan was clean. The path from plan to working was paved with the kind of bugs that only appear when real hardware stops being polite, and I want to walk through them, because each one is a small lesson and together they are the whole texture of the job.

The first was the most alarming and the most educational. I booted the FreeBSD image inside a pod, and QEMU came up spinning at four hundred percent CPU while the guest's console filled with synchronous exception panics — the Arm equivalent of a program tripping over an instruction the processor won't execute. The image booted fine on the host directly. It crashed reliably inside the pod. The difference was the RK3588's defining feature and, here, its trap: it is a big.LITTLE chip, four fast A76 cores and four efficient A55 cores, and they do not implement exactly the same instructions. QEMU was passing the host CPU straight through to the guest, and the Linux scheduler, seeing an unpinned QEMU thread, was free to migrate the virtual CPU from a big core to a little core mid-execution. The guest would start an instruction the A76 supports, get moved to an A55 that doesn't, and fall over. The host's own VM launch scripts had known this all along — they pinned QEMU with taskset -c 4-7, confining it to the four A76 cores — and my pod hadn't. One taskset and the synchronous exceptions vanished. The cluster's cleverest hardware feature was, for this workload, a thing to carefully switch off.

The second bug was a quiet philosophical one. My plan referred throughout to the BSD disk images as "golden images" — static, pristine templates you overlay a scratch copy on top of and never mutate. When I went to make that scratch overlay, QEMU refused: the image was locked, held open read-write by another process. The other process was the always-on VM. The "golden image" was the live disk of the running build server — not a template at all, but the working drive of a machine that was, at that moment, mid-thought. Overlaying a copy-on-write layer on a disk that's being written underneath you is not a snapshot; it's a data-corruption generator. And when I did force a copy, it failed its filesystem check on boot, because the source had been running for thirteen days without a clean shutdown and its dirty bit had never been cleared. The fix was to do the boring, correct thing I'd been skipping: cleanly shut each VM down, take a genuine point-in-time snapshot of the now-consistent disk into a real golden image, and restart the VM. The builds overlay those — static, clean, never open by anything else. It cost a couple of minutes of downtime per guest and bought a foundation that isn't quietly rotting.

Then the small, specific lies of virtual hardware. The first pod boot that got past the CPU pinning died instantly with QEMU complaining it couldn't find efi-virtio.rom — a network-card boot ROM that isn't packaged in a slim container image. The host scripts, again, already knew: they set romfile= empty on the virtual NIC, disabling the ROM the build doesn't need. Add that, and the guest booted further — and then OpenBSD, specifically, hung partway through, wedged on device probing. FreeBSD had sailed through the same configuration. The difference was vectors=0, another flag the host scripts carried and mine didn't, which disables MSI-X interrupts on the virtual devices; OpenBSD's virtio drivers, under emulation, needed the older interrupt style to come up at all. FreeBSD tolerated the modern path. OpenBSD did not. The host's launch scripts, it turned out, were a compressed archive of exactly these lessons, learned once and encoded as flags I had to re-learn by leaving them out.

The BSDs then taught me about their own disks. My build script staged source into the guest's home directory and immediately hit "No space left on device." OpenBSD, by long tradition, slices its disk into many small partitions, and its root filesystem — where a naive home directory lives — was four hundred megabytes and already full. NetBSD does the same, with a root barely over a gigabyte. FreeBSD hands you one big root and doesn't care. So the builder learned to build where the space actually is: on OpenBSD, in /usr/obj, the partition the system reserves for build output; on NetBSD, in /usr/pkg, its roomy second disk. Rust made this worse before it made it better, because Cargo unpacks its entire crate registry into a cache under the home directory, and on NetBSD that cache alone overflowed the tiny root the instant a real dependency graph arrived. Pointing CARGO_HOME at the big partition too, per guest, finally let a Rust project build on a NetBSD machine whose root filesystem couldn't have held its own dependencies.

The last of them wasn't a build bug at all; it was a networking cleanup that turned into an accidental repair. NetBSD's VM, unlike its siblings, had been bridged directly onto my LAN, while FreeBSD and OpenBSD used QEMU's simpler user-mode networking. I wanted them consistent, so I converted NetBSD to user-mode too — and in restarting it discovered that the bridged setup would have failed on its very next boot regardless, because a QEMU package update had silently stripped the CAP_NET_ADMIN capability off the bridge helper, the small privileged binary that lets an unprivileged VM attach to a bridge. The VM had been running for so long it hadn't been restarted since the update landed. The cleanup I wanted to do had been sitting on top of a latent failure I didn't know about, and converting away from bridging both simplified the design and defused a bomb.

None of these were deep. Every one was a mundane mistake or a mundane environmental fact, wearing for an afternoon the costume of something profound. That is the entire pattern of this kind of work, and I've made peace with it: the exotic theory that flatters your understanding almost always sits on top of a boring cause that flatters nothing. taskset. romfile=. A partition that's full. A capability an update took away.

When it was done, the payoff was concrete. A single command — make bsd-build PROJECT=lattice OS=freebsd — now fetches the latest source straight from GitHub, boots the golden FreeBSD image in a pod on the Orange Pi, compiles the project inside it, and drops a genuine aarch64 FreeBSD executable back onto my workstation, stripped and ready. The same command builds for OpenBSD and NetBSD, for both projects, and a release variant does all of it and publishes. Six native BSD binaries, produced by boards that used to keep three virtual machines awake all night to do the same thing on demand for no one.

Naming the tenants

With the builds turned into jobs, the always-on VMs had no reason to exist. I stopped and disabled them — reversibly; the golden images remain, and a systemctl away is the old world back — and the Orange Pi handed back roughly six gigabytes of RAM and its four A76 cores. Which returns us, neatly, to the theme this whole exercise keeps circling: reclaimed idle capacity is only worth reclaiming if you then ask something of it.

So here is the honest census of what actually runs on the cluster now — the jobs, named plainly.

On-demand BSD binary builds are the anchor tenant, the workload the whole thing was justified by: FreeBSD, OpenBSD, and NetBSD Arm64 binaries for the language compiler and the ballistics engine, each build a pod that boots a real BSD, compiles, and exits. Zero footprint between releases; real cores during them.

A full observability stack is the second tenant, and it exists partly because a cluster you can't see is a cluster you can't trust. It's the standard Prometheus-and-Grafana pairing, deployed the k3s-native way, with node exporters on every board reporting not just CPU and memory and disk but the thing that actually matters on fanless Arm boards crammed with NVMe: temperature. SBCs throttle when they cook, and a build farm that silently halves its own clock speed under thermal load is a build farm that lies to you about why it's slow. Grafana lives at a name on my LAN — grafana.localnet, resolved by an Unbound server on a little FreeBSD box that also runs my DNS — behind an ingress controller carefully arranged not to collide with the web page one of the Pis was already serving on port eighty. The stack pins its stateful pieces to the CM5, the node with the two-terabyte drive and the least else to do. The board that was holding its breath now holds the metrics.

Continuous fuzzing of the compiler is the newest tenant, and the one that most makes me glad the cores were sitting there. The language project ships nine libFuzzer harnesses — for the compiler front-end, three different bytecode VMs, the lexer, the formatter, and the JSON, TOML, and YAML parsers — and a fuzzer is the most gluttonous, most embarrassingly-parallel workload I own. It will eat every idle cycle you're willing to feed it and ask for more, forever, and every cycle it eats is a cycle spent trying to crash your code before a user does. On a cluster whose defining problem was too many idle cores, that is a nearly perfect fit. Each target gets its own continuous pod, spread across the three boards, building itself from the latest source and grinding against a corpus that persists and compounds on the NVMe — the first of the nine are coming online as I write this. And the results go somewhere I'll actually see them: when a fuzzer finds a crash, a small watcher reproduces it, fingerprints it so the same bug doesn't file itself twice, and opens a Jira ticket — the real bug tracker the project already uses — with the sanitizer backtrace and a base64 reproducer attached, tagged to the right component. Live throughput and coverage stream into the same Grafana the rest of the cluster reports to; a nightly job renders coverage as HTML behind that same ingress. Idle cores in; deduplicated, actionable bug reports out.

And underneath all of it, still, the Orange Pi's NPU runs a local language model when I ask it to — the original tenant, never evicted, just no longer the only one. Kubernetes shares the boards with it rather than owning them outright, which is exactly the relationship a homelab wants: the cluster is a tenant of the hardware, not its landlord.

The capacity was always there

There's a version of this post that's a shopping list — buy these boards, run these commands, get a cluster. I don't think that's the true version. The commands were the easy part; they always are. The hard part was the pile of small specific lies the hardware tells when you finally push it, and the honest bookkeeping of what's really running versus what's merely installed.

But the part I keep turning over isn't technical at all. It's that none of this required new hardware. Every core, every gigabyte, every terabyte of fast disk in this cluster was already sitting on my shelf, already powered, already mostly doing nothing. The Orange Pi was awake all night to compile software a few minutes a week. The CM5 was holding its breath. The utilization graph was a flat line near the floor, and I'd looked at it a hundred times without seeing it as a to-do item.

What changed wasn't the machines. It was the question I asked them. Three boards that each did one small thing became a single system that builds native binaries for three operating systems, watches its own temperature, and fuzzes a compiler around the clock — and the only thing I added was the willingness to treat idle capacity as a debt worth collecting.

The cluster was already there. It just hadn't been told yet.

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

A.C. Jokela — Thu, 18 Jun 2026 19:45:00 GMT

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

I have a Rust project — a ballistics engine — that ships binaries for an embarrassing number of platforms. Linux on x86 and ARM, Windows, FreeBSD, NetBSD, and, because someone always asks and because I like the discipline of it, OpenBSD. Most of those targets are easy: a build matrix, a cross-compiler, a Docker image with the right musl toolchain, and the artifacts fall out the other end. The BSDs are where the matrix stops being polite.

For FreeBSD and NetBSD on arm64 I already had a clean answer: spin up a real Graviton instance, build natively, collect the binary, terminate the instance. A build costs a few cents and finishes in minutes, because the hardware is real ARM and there's no emulation tax. I'd written a little orchestrator for exactly this — launch, build, tear down — and it worked beautifully for two of the three BSDs.

OpenBSD was the one that wouldn't fall in line. And the reason is so clean, so completely a consequence of what OpenBSD is, that it took me a while to stop treating it as a bug to be worked around and start treating it as a fact to be designed around.

This is the story of building native OpenBSD/arm64 binaries on a cloud that OpenBSD, in the most literal sense, cannot see.

Why native, and why that's suddenly hard

The honest first question is whether native builds matter at all. Couldn't I cross-compile OpenBSD/arm64 binaries from Linux and skip the whole circus?

For a trivial program, maybe. For anything real, no. Cross-compiling to OpenBSD means reproducing its libc, its linker behavior, its system call conventions, and its packaging quirks in a toolchain that lives somewhere else — and then trusting that the binary you produced behaves the way a binary built on OpenBSD would. The entire value of shipping an OpenBSD artifact is the claim "this was built and linked on the platform it runs on." A cross-compiled binary that mostly works is worse than no binary at all, because it fails in ways your users discover and you can't reproduce.

So: native. Build it on OpenBSD/arm64, on real ARM silicon, the way the platform's own users would.

The old way to get there was emulation — QEMU on an x86 host, software-emulating an ARM CPU. It works, and it's miserably slow: with no hardware virtualization to lean on, every guest instruction gets translated, and a four-minute build stretches to twenty-five. I did not want to go back.

The new way was supposed to be the Graviton instance: real ARM cores, native speed, terminate when done. It works for FreeBSD. It works for NetBSD. And then you try it with OpenBSD and walk straight into the wall.

The wall: OpenBSD has no driver for AWS's network card

Here is the entire problem, and it is almost funny once you see it.

Modern EC2 instances present their network to the guest through the Elastic Network Adapter — ena, a custom Amazon device with a custom driver. Every cloud-friendly OS ships that driver: Linux has it, FreeBSD has it, NetBSD's community AMIs have it. It is the thing that lets an instance have a network at all.

OpenBSD's arm64 kernel does not have an ena driver.

This isn't an oversight, and it isn't a slight. OpenBSD is a small, fiercely curated tree. Drivers get in because someone runs the hardware, writes clean code, and maintains it — and effectively nobody runs OpenBSD as a first-class EC2 guest, so the driver for Amazon's proprietary NIC simply isn't there. The project doesn't chase cloud hardware the way a commercial OS does, and that restraint is a feature, not a failing. But it has a consequence: an OpenBSD/arm64 instance on EC2 boots, comes up, and finds no network interface it understands. The NIC is right there on the PCI bus, and the kernel has nothing to bind to it.

An EC2 instance you cannot reach over the network is not an instance. It's a billable void. There's no SSH, no console you can usefully drive, no way to hand it a build and get a binary back. The OS runs perfectly; it just can't talk to the one piece of hardware that makes a cloud instance a thing you can use.

I sat with that for a bit. You can't add the driver from outside. You can't give OpenBSD a NIC it understands by configuring the instance differently, because AWS only offers ena. The platform and the OS disagree about reality at the level of the network card, and neither one is going to move.

So I stopped trying to make them agree.

Stop asking OpenBSD to talk to AWS

The move, once it arrived, was obvious in the way these things always are in retrospect: don't let OpenBSD touch the cloud's hardware at all.

Put a Linux host on the metal. Linux has the ena driver; it brings the network up without complaint. Then run OpenBSD as a virtual machine inside that Linux host, under QEMU with KVM acceleration — and hand the guest the kind of hardware OpenBSD does understand: virtio. OpenBSD has had solid virtio drivers for years. It sees a virtio network card and a virtio disk and is perfectly happy, never knowing or caring that the physical NIC underneath is an Amazon device it couldn't have spoken to directly.

The Linux host owns the ena and the public IP. The OpenBSD guest lives behind it on a private virtual network, reachable through a port forward. The two layers each get hardware they recognize, and the impedance mismatch — the missing driver — simply stops mattering, because the only thing that ever talks to the ena is Linux.

It's the same shape as a hundred other infrastructure problems: when two layers can't agree, you don't force them, you put a translator between them. Here the translator is an entire hypervisor, and the thing being translated is the basic question of what a network card is.

This also quietly solves the speed problem that made x86 emulation unbearable. Because the host is itself ARM, KVM can run the ARM guest at near-native speed — no instruction translation, just virtualization. OpenBSD/arm64 on an ARM host under KVM boots in seconds and builds at real-hardware pace. The thing that was slow about the old emulation approach was never "virtualization"; it was cross-architecture emulation. Match the architectures and the tax disappears.

Why it has to be bare metal

There's a catch, and it's the reason this costs more than a few cents.

KVM needs hardware virtualization extensions, and on AWS those are only exposed to the guest OS on bare-metal instances. On an ordinary virtualized Graviton instance, the Nitro hypervisor already owns the CPU's virtualization layer; your kernel can't open /dev/kvm because something else is already standing where KVM would stand. Try to accelerate QEMU on a normal instance and it silently falls back to the slow software path — right back to the emulation tax I was trying to escape.

So the host has to be .metal. The cheapest bare-metal Graviton AWS sells is a1.metal — first-generation Graviton, sixteen Cortex-A72 cores, and, crucially, a real /dev/kvm exposed to the host kernel. It's about forty-one cents an hour on demand. That's more than the few-cents-per-build the native FreeBSD path costs, but a build spins the instance up, runs, and terminates in fifteen or twenty minutes, so a release still costs well under a dollar.

I picked Ubuntu for the Linux host — its QEMU and UEFI firmware packages are exactly what the recipe wants — pointed it at the right ARM AMI, and gave it a roomy root volume. The host comes up, apt installs QEMU and the AAVMF UEFI firmware, and now I have an ARM machine with KVM, ready to boot an OpenBSD guest. That part was easy.

The part that ate an afternoon was getting OpenBSD to actually attach its own disks.

Two device-tree gotchas that ate an afternoon

QEMU's generic ARM machine type, virt, gives you a clean, modern, device-tree-described board. You hand it virtio devices and it wires them up. I did the obvious thing — a virtio block device for the installer image, a virtio block device for the target disk, a virtio network device — booted, and watched the OpenBSD installer come up, reach the disk-selection step, and announce:

Available disks are: none.

No disks. Also, quietly, no network interface. Both virtio devices were there — UEFI had even booted the installer kernel off one of them — and yet the running OpenBSD kernel saw neither.

The cause is a transport mismatch that's easy to miss. On the virt machine, virtio-blk-device and virtio-net-device use the virtio-MMIO transport — devices described in the device tree, sitting at fixed memory addresses. UEFI can boot from those. OpenBSD's arm64 kernel, it turns out, does not attach them. What it does attach is virtio-PCI — the same virtio devices presented as PCI devices on the PCIe controller the kernel already probes. Swap virtio-blk-device for virtio-blk-pci and virtio-net-device for virtio-net-pci, and suddenly the disks appear as sd0 and sd1 and the NIC as vio0. One word per device, and the installer can see the world.

That fixed the disks. The very next boot hung.

Not crashed — hung. The kernel probed all the devices, printed them, and then stopped cold, right before mounting its root filesystem. The host's load average sat at zero: the guest CPU wasn't spinning, it was blocked, waiting for something that was never going to come. The something was an interrupt.

virtio-PCI devices default to MSI-X — message-signaled interrupts, delivered on ARM through the GIC's Interrupt Translation Service. On this first-generation Graviton host, under KVM, that MSI path doesn't deliver. The guest arms its interrupts, waits for the first one, and waits forever. The fix is to tell QEMU to give each virtio device zero MSI-X vectors, forcing it back to plain old pin-based (INTx) interrupts, which ride the GIC's distributor directly and work fine:

-device virtio-blk-pci,drive=hd0,bootindex=1,vectors=0
-device virtio-net-pci,netdev=net0,vectors=0

With vectors=0 on every virtio device, the device line in the boot log changes from msix to a plain irq, the kernel gets its interrupt, mounts root, and reaches the installer prompt. Two one-line changes — the PCI transport and the disabled MSI-X — separated by an afternoon of staring at a boot that stopped in exactly the same place every time.

Neither of these is in the friendly "run OpenBSD under QEMU" recipes you'll find, because those recipes assume a host where MSI-X works and where someone already knew to use the PCI transport. The combination of this OS on this generation of silicon under this hypervisor is niche enough that you find the edges yourself.

Installing an OS through a serial straw

The virt machine has no graphics — no VGA, no framebuffer worth the name. OpenBSD's installer comes up on the serial console, which QEMU helpfully wires to standard out. That's fine for a human at a keyboard. It's less fine when you're trying to make the whole thing repeatable, because the OpenBSD installer is a long, interactive conversation: hostname, network, disk, sets, passwords, two dozen prompts.

OpenBSD has an answer for this — autoinstall, where a response file answers every prompt and the install runs unattended. I tried it first, serving the response file over HTTP from the Linux host. It failed in an instructive way: QEMU's built-in user-mode network doesn't advertise an autoinstall server in its DHCP lease, and when the installer falls back to asking where the response file is, it tries to fetch it before the guest's network is actually up. The fetch fails, autoinstall aborts, and you're dropped to a shell. The unattended path assumes a network environment QEMU's toy DHCP doesn't provide.

So the install came down to driving the interactive program by hand — except I didn't drive it. I let Claude drive, programmatically: QEMU running inside a tmux session, the agent feeding the serial console one answer at a time and reading the screen back between each step. It's a strange way to install an operating system — an AI typing into a virtual serial port through a terminal multiplexer over SSH — and it fumbled in telling ways. It stuffed answers ahead of the prompts and desynced, at one point confidently typing vio0 into the DNS domain name field because it had guessed the question order and the installer disagreed; a mismatched password pair followed, and I watched it back out and resynchronize. Once it was reading each prompt instead of predicting it, the rest was deterministic, and it pulled the install sets straight off the installer image so the guest never needed working DNS at all. Whole-disk GPT, auto layout, sets from the local media, and a few minutes later: CONGRATULATIONS! Your OpenBSD install has been successfully completed!

Drop the installer image, reboot from the disk alone, and the installed system comes up on its own — generates its host keys, starts sshd, reaches a login prompt. uname confirms it: OpenBSD 7.9, arm64, on a "QEMU KVM Virtual Machine." A real, persistent OpenBSD/arm64 system, running on a cloud that can't host it directly.

Making it ephemeral: the golden image and the one-button build

A hand-installed VM is a proof of concept, not a build server. The point was always to make this disappear into a command — spin the whole stack up, build, collect the artifact, spin it down, and pay pennies.

So I did the slow part exactly once. I took that freshly installed guest, added the toolchain a build needs — git, the Rust compiler — and a build key whose public half lives in the image's authorized_keys. Then I shut it down cleanly and compressed the disk: a 20 GB virtual volume holding maybe two gigs of real data squeezed down to an 832 MB qcow2. That golden image went into S3, in the same region as the metal hosts, so pulling it back down is fast and free.

Around it I wrote an orchestrator — the OpenBSD sibling of the FreeBSD/NetBSD tool I already had. One command does the whole dance: launch an a1.metal, wait for it, install QEMU, pull the golden image from S3 through a presigned URL (so the host needs no AWS credentials of its own), boot the OpenBSD guest with the exact virtio-PCI-and-vectors=0 incantation the afternoon taught me, and wait for the guest's SSH to answer. The guest is reachable only through a port forward on the host, so the orchestrator jumps through the host to reach it — the build key never leaves my laptop, and no extra port is ever opened to the world. Then it pipes the build into the guest, runs it natively, copies the binary back out, checksums it, and — in a finally block, so it happens whether the build succeeds, fails, or throws — terminates the metal host.

Spin up, build, collect, spin down. The instance exists for the length of one build and then it's gone, and the bill is loose change.

The last wall: the build that wouldn't build

I ran the orchestrator end to end, and it was beautiful right up until the part that mattered. It launched the host, pulled the image, booted OpenBSD, SSH'd in, started the build — and the build died compiling a TLS library, deep in my engine's dependency tree.

I knew exactly what it was. Or I thought I did. My ballistics engine has an optional "online" feature, on by default, that pulls in an HTTP client and with it a whole TLS stack — rustls, the ring crypto backend, the works. Of course that was the thing that wouldn't build on an OS this far off the beaten path. I turned the feature off, the build went green, and I jotted down a tidy little lesson about exotic platforms and the liabilities of convenient defaults.

The lesson was wrong, and I found out a few hours later. The release binary that already ships for this exact platform has that entire TLS stack compiled into it — rustls builds on OpenBSD/arm64 perfectly well. So that was never the wall. The real wall was disk. OpenBSD's installer auto-layout splits the disk into a fistful of small partitions, and the build was running in /tmp — about a gigabyte. A Rust release target/ directory does not fit in a gigabyte, and rustls happens to be a large crate, so the partition filled mid-compile and the compiler fell over with an error that looked like a code problem. Turning off the feature "fixed" it only because a leaner build squeaked under the limit. The actual fix was one line — build in /usr/obj, the roomy partition OpenBSD hands you for exactly this — and with room to work, the full build, TLS and all, finishes fine.

Here's the part that stings, and the reason I'm telling it on myself. I am not new to this. I've dabbled in the BSDs for a long time — FreeBSD was my daily driver in the early 2000s — and I know OpenBSD well enough that its installer holds no surprises; it has barely changed in twenty years. "OpenBSD carves the disk into small partitions and you do your building in /usr/obj" is not arcana to me, it's reflex. A version of me at the keyboard would have read compiler died, OpenBSD, /tmp and smelled out of disk before finishing the line. But I wasn't at the keyboard — I'd handed it to the same agent that, a few screens earlier, had typed vio0 into the DNS field. An agent has read about OpenBSD without ever once running out of disk on it, so it had no instinct for the boring failure. It reached instead for the exotic one sitting right there in the dependency graph — the explanation that was plausible, pre-loaded, and a little flattering, because if the crypto stack won't compile you're wrestling something genuinely hard, rather than something you'd have caught in 2003.

And that is the trap worth naming, sharper than "niche platforms are hard." On an unfamiliar system every failure presents as exotic; the strange explanation announces itself and the mundane one doesn't. The further off the beaten path you are, the more deliberately you have to rule out the boring thing first — because out here everything wears the platform's costume, and an out-of-disk error in a one-gigabyte /tmp looks exactly like a dependency that can't be built, right up until someone who's been burned before tells you it's just disk.

What it actually was about

Step back and the whole project rhymes with a pattern I keep meeting. The headline problem — "build OpenBSD/arm64 binaries on AWS" — has a one-sentence answer once you have it: run OpenBSD as a KVM guest on a bare-metal Graviton host. That sentence is the easy part. Everything real lived in the gap between two systems that were each, on their own terms, completely correct.

OpenBSD is correct to omit a driver for hardware almost none of its users run; its restraint is exactly why it's OpenBSD. AWS is correct to present its network through a custom adapter built for the scale it operates at. Neither is wrong. They simply grew up with no expectation of meeting, and the moment you ask them to share a machine, every quiet assumption each one made surfaces at once — the missing driver, the MMIO-versus-PCI transport, the MSI-X interrupts that don't land on first-gen silicon, the DHCP lease that doesn't carry what the installer hoped, the cramped /tmp masquerading as a dependency that wouldn't compile.

The hypervisor in the middle isn't really there to virtualize a CPU. It's there to be the place where two designs that disagree about what hardware is can both be right at the same time. OpenBSD gets virtio and is happy. AWS gets ena and is happy. The seam between them holds because something whose entire job is to translate sits exactly on it.

And the build server that came out the other end is genuinely good: a single command that conjures an ARM machine, runs a native OpenBSD build at full speed, and dissolves the machine when it's done, for the price of a vending-machine snack. The operating system that couldn't see the network now turns out release binaries on the very cloud it can't directly run on — because at no point does anyone ask it to see the network. That was never going to work. The trick was to stop asking.