The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

A.C. Jokela — Thu, 18 Jun 2026 19:45:00 GMT

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

I have a Rust project — a ballistics engine — that ships binaries for an embarrassing number of platforms. Linux on x86 and ARM, Windows, FreeBSD, NetBSD, and, because someone always asks and because I like the discipline of it, OpenBSD. Most of those targets are easy: a build matrix, a cross-compiler, a Docker image with the right musl toolchain, and the artifacts fall out the other end. The BSDs are where the matrix stops being polite.

For FreeBSD and NetBSD on arm64 I already had a clean answer: spin up a real Graviton instance, build natively, collect the binary, terminate the instance. A build costs a few cents and finishes in minutes, because the hardware is real ARM and there's no emulation tax. I'd written a little orchestrator for exactly this — launch, build, tear down — and it worked beautifully for two of the three BSDs.

OpenBSD was the one that wouldn't fall in line. And the reason is so clean, so completely a consequence of what OpenBSD is, that it took me a while to stop treating it as a bug to be worked around and start treating it as a fact to be designed around.

This is the story of building native OpenBSD/arm64 binaries on a cloud that OpenBSD, in the most literal sense, cannot see.

Why native, and why that's suddenly hard

The honest first question is whether native builds matter at all. Couldn't I cross-compile OpenBSD/arm64 binaries from Linux and skip the whole circus?

For a trivial program, maybe. For anything real, no. Cross-compiling to OpenBSD means reproducing its libc, its linker behavior, its system call conventions, and its packaging quirks in a toolchain that lives somewhere else — and then trusting that the binary you produced behaves the way a binary built on OpenBSD would. The entire value of shipping an OpenBSD artifact is the claim "this was built and linked on the platform it runs on." A cross-compiled binary that mostly works is worse than no binary at all, because it fails in ways your users discover and you can't reproduce.

So: native. Build it on OpenBSD/arm64, on real ARM silicon, the way the platform's own users would.

The old way to get there was emulation — QEMU on an x86 host, software-emulating an ARM CPU. It works, and it's miserably slow: with no hardware virtualization to lean on, every guest instruction gets translated, and a four-minute build stretches to twenty-five. I did not want to go back.

The new way was supposed to be the Graviton instance: real ARM cores, native speed, terminate when done. It works for FreeBSD. It works for NetBSD. And then you try it with OpenBSD and walk straight into the wall.

The wall: OpenBSD has no driver for AWS's network card

Here is the entire problem, and it is almost funny once you see it.

Modern EC2 instances present their network to the guest through the Elastic Network Adapter — ena, a custom Amazon device with a custom driver. Every cloud-friendly OS ships that driver: Linux has it, FreeBSD has it, NetBSD's community AMIs have it. It is the thing that lets an instance have a network at all.

OpenBSD's arm64 kernel does not have an ena driver.

This isn't an oversight, and it isn't a slight. OpenBSD is a small, fiercely curated tree. Drivers get in because someone runs the hardware, writes clean code, and maintains it — and effectively nobody runs OpenBSD as a first-class EC2 guest, so the driver for Amazon's proprietary NIC simply isn't there. The project doesn't chase cloud hardware the way a commercial OS does, and that restraint is a feature, not a failing. But it has a consequence: an OpenBSD/arm64 instance on EC2 boots, comes up, and finds no network interface it understands. The NIC is right there on the PCI bus, and the kernel has nothing to bind to it.

An EC2 instance you cannot reach over the network is not an instance. It's a billable void. There's no SSH, no console you can usefully drive, no way to hand it a build and get a binary back. The OS runs perfectly; it just can't talk to the one piece of hardware that makes a cloud instance a thing you can use.

I sat with that for a bit. You can't add the driver from outside. You can't give OpenBSD a NIC it understands by configuring the instance differently, because AWS only offers ena. The platform and the OS disagree about reality at the level of the network card, and neither one is going to move.

So I stopped trying to make them agree.

Stop asking OpenBSD to talk to AWS

The move, once it arrived, was obvious in the way these things always are in retrospect: don't let OpenBSD touch the cloud's hardware at all.

Put a Linux host on the metal. Linux has the ena driver; it brings the network up without complaint. Then run OpenBSD as a virtual machine inside that Linux host, under QEMU with KVM acceleration — and hand the guest the kind of hardware OpenBSD does understand: virtio. OpenBSD has had solid virtio drivers for years. It sees a virtio network card and a virtio disk and is perfectly happy, never knowing or caring that the physical NIC underneath is an Amazon device it couldn't have spoken to directly.

The Linux host owns the ena and the public IP. The OpenBSD guest lives behind it on a private virtual network, reachable through a port forward. The two layers each get hardware they recognize, and the impedance mismatch — the missing driver — simply stops mattering, because the only thing that ever talks to the ena is Linux.

It's the same shape as a hundred other infrastructure problems: when two layers can't agree, you don't force them, you put a translator between them. Here the translator is an entire hypervisor, and the thing being translated is the basic question of what a network card is.

This also quietly solves the speed problem that made x86 emulation unbearable. Because the host is itself ARM, KVM can run the ARM guest at near-native speed — no instruction translation, just virtualization. OpenBSD/arm64 on an ARM host under KVM boots in seconds and builds at real-hardware pace. The thing that was slow about the old emulation approach was never "virtualization"; it was cross-architecture emulation. Match the architectures and the tax disappears.

Why it has to be bare metal

There's a catch, and it's the reason this costs more than a few cents.

KVM needs hardware virtualization extensions, and on AWS those are only exposed to the guest OS on bare-metal instances. On an ordinary virtualized Graviton instance, the Nitro hypervisor already owns the CPU's virtualization layer; your kernel can't open /dev/kvm because something else is already standing where KVM would stand. Try to accelerate QEMU on a normal instance and it silently falls back to the slow software path — right back to the emulation tax I was trying to escape.

So the host has to be .metal. The cheapest bare-metal Graviton AWS sells is a1.metal — first-generation Graviton, sixteen Cortex-A72 cores, and, crucially, a real /dev/kvm exposed to the host kernel. It's about forty-one cents an hour on demand. That's more than the few-cents-per-build the native FreeBSD path costs, but a build spins the instance up, runs, and terminates in fifteen or twenty minutes, so a release still costs well under a dollar.

I picked Ubuntu for the Linux host — its QEMU and UEFI firmware packages are exactly what the recipe wants — pointed it at the right ARM AMI, and gave it a roomy root volume. The host comes up, apt installs QEMU and the AAVMF UEFI firmware, and now I have an ARM machine with KVM, ready to boot an OpenBSD guest. That part was easy.

The part that ate an afternoon was getting OpenBSD to actually attach its own disks.

Two device-tree gotchas that ate an afternoon

QEMU's generic ARM machine type, virt, gives you a clean, modern, device-tree-described board. You hand it virtio devices and it wires them up. I did the obvious thing — a virtio block device for the installer image, a virtio block device for the target disk, a virtio network device — booted, and watched the OpenBSD installer come up, reach the disk-selection step, and announce:

Available disks are: none.

No disks. Also, quietly, no network interface. Both virtio devices were there — UEFI had even booted the installer kernel off one of them — and yet the running OpenBSD kernel saw neither.

The cause is a transport mismatch that's easy to miss. On the virt machine, virtio-blk-device and virtio-net-device use the virtio-MMIO transport — devices described in the device tree, sitting at fixed memory addresses. UEFI can boot from those. OpenBSD's arm64 kernel, it turns out, does not attach them. What it does attach is virtio-PCI — the same virtio devices presented as PCI devices on the PCIe controller the kernel already probes. Swap virtio-blk-device for virtio-blk-pci and virtio-net-device for virtio-net-pci, and suddenly the disks appear as sd0 and sd1 and the NIC as vio0. One word per device, and the installer can see the world.

That fixed the disks. The very next boot hung.

Not crashed — hung. The kernel probed all the devices, printed them, and then stopped cold, right before mounting its root filesystem. The host's load average sat at zero: the guest CPU wasn't spinning, it was blocked, waiting for something that was never going to come. The something was an interrupt.

virtio-PCI devices default to MSI-X — message-signaled interrupts, delivered on ARM through the GIC's Interrupt Translation Service. On this first-generation Graviton host, under KVM, that MSI path doesn't deliver. The guest arms its interrupts, waits for the first one, and waits forever. The fix is to tell QEMU to give each virtio device zero MSI-X vectors, forcing it back to plain old pin-based (INTx) interrupts, which ride the GIC's distributor directly and work fine:

-device virtio-blk-pci,drive=hd0,bootindex=1,vectors=0
-device virtio-net-pci,netdev=net0,vectors=0

With vectors=0 on every virtio device, the device line in the boot log changes from msix to a plain irq, the kernel gets its interrupt, mounts root, and reaches the installer prompt. Two one-line changes — the PCI transport and the disabled MSI-X — separated by an afternoon of staring at a boot that stopped in exactly the same place every time.

Neither of these is in the friendly "run OpenBSD under QEMU" recipes you'll find, because those recipes assume a host where MSI-X works and where someone already knew to use the PCI transport. The combination of this OS on this generation of silicon under this hypervisor is niche enough that you find the edges yourself.

Installing an OS through a serial straw

The virt machine has no graphics — no VGA, no framebuffer worth the name. OpenBSD's installer comes up on the serial console, which QEMU helpfully wires to standard out. That's fine for a human at a keyboard. It's less fine when you're trying to make the whole thing repeatable, because the OpenBSD installer is a long, interactive conversation: hostname, network, disk, sets, passwords, two dozen prompts.

OpenBSD has an answer for this — autoinstall, where a response file answers every prompt and the install runs unattended. I tried it first, serving the response file over HTTP from the Linux host. It failed in an instructive way: QEMU's built-in user-mode network doesn't advertise an autoinstall server in its DHCP lease, and when the installer falls back to asking where the response file is, it tries to fetch it before the guest's network is actually up. The fetch fails, autoinstall aborts, and you're dropped to a shell. The unattended path assumes a network environment QEMU's toy DHCP doesn't provide.

So the install came down to driving the interactive program by hand — except I didn't drive it. I let Claude drive, programmatically: QEMU running inside a tmux session, the agent feeding the serial console one answer at a time and reading the screen back between each step. It's a strange way to install an operating system — an AI typing into a virtual serial port through a terminal multiplexer over SSH — and it fumbled in telling ways. It stuffed answers ahead of the prompts and desynced, at one point confidently typing vio0 into the DNS domain name field because it had guessed the question order and the installer disagreed; a mismatched password pair followed, and I watched it back out and resynchronize. Once it was reading each prompt instead of predicting it, the rest was deterministic, and it pulled the install sets straight off the installer image so the guest never needed working DNS at all. Whole-disk GPT, auto layout, sets from the local media, and a few minutes later: CONGRATULATIONS! Your OpenBSD install has been successfully completed!

Drop the installer image, reboot from the disk alone, and the installed system comes up on its own — generates its host keys, starts sshd, reaches a login prompt. uname confirms it: OpenBSD 7.9, arm64, on a "QEMU KVM Virtual Machine." A real, persistent OpenBSD/arm64 system, running on a cloud that can't host it directly.

Making it ephemeral: the golden image and the one-button build

A hand-installed VM is a proof of concept, not a build server. The point was always to make this disappear into a command — spin the whole stack up, build, collect the artifact, spin it down, and pay pennies.

So I did the slow part exactly once. I took that freshly installed guest, added the toolchain a build needs — git, the Rust compiler — and a build key whose public half lives in the image's authorized_keys. Then I shut it down cleanly and compressed the disk: a 20 GB virtual volume holding maybe two gigs of real data squeezed down to an 832 MB qcow2. That golden image went into S3, in the same region as the metal hosts, so pulling it back down is fast and free.

Around it I wrote an orchestrator — the OpenBSD sibling of the FreeBSD/NetBSD tool I already had. One command does the whole dance: launch an a1.metal, wait for it, install QEMU, pull the golden image from S3 through a presigned URL (so the host needs no AWS credentials of its own), boot the OpenBSD guest with the exact virtio-PCI-and-vectors=0 incantation the afternoon taught me, and wait for the guest's SSH to answer. The guest is reachable only through a port forward on the host, so the orchestrator jumps through the host to reach it — the build key never leaves my laptop, and no extra port is ever opened to the world. Then it pipes the build into the guest, runs it natively, copies the binary back out, checksums it, and — in a finally block, so it happens whether the build succeeds, fails, or throws — terminates the metal host.

Spin up, build, collect, spin down. The instance exists for the length of one build and then it's gone, and the bill is loose change.

The last wall: the build that wouldn't build

I ran the orchestrator end to end, and it was beautiful right up until the part that mattered. It launched the host, pulled the image, booted OpenBSD, SSH'd in, started the build — and the build died compiling a TLS library, deep in my engine's dependency tree.

I knew exactly what it was. Or I thought I did. My ballistics engine has an optional "online" feature, on by default, that pulls in an HTTP client and with it a whole TLS stack — rustls, the ring crypto backend, the works. Of course that was the thing that wouldn't build on an OS this far off the beaten path. I turned the feature off, the build went green, and I jotted down a tidy little lesson about exotic platforms and the liabilities of convenient defaults.

The lesson was wrong, and I found out a few hours later. The release binary that already ships for this exact platform has that entire TLS stack compiled into it — rustls builds on OpenBSD/arm64 perfectly well. So that was never the wall. The real wall was disk. OpenBSD's installer auto-layout splits the disk into a fistful of small partitions, and the build was running in /tmp — about a gigabyte. A Rust release target/ directory does not fit in a gigabyte, and rustls happens to be a large crate, so the partition filled mid-compile and the compiler fell over with an error that looked like a code problem. Turning off the feature "fixed" it only because a leaner build squeaked under the limit. The actual fix was one line — build in /usr/obj, the roomy partition OpenBSD hands you for exactly this — and with room to work, the full build, TLS and all, finishes fine.

Here's the part that stings, and the reason I'm telling it on myself. I am not new to this. I've dabbled in the BSDs for a long time — FreeBSD was my daily driver in the early 2000s — and I know OpenBSD well enough that its installer holds no surprises; it has barely changed in twenty years. "OpenBSD carves the disk into small partitions and you do your building in /usr/obj" is not arcana to me, it's reflex. A version of me at the keyboard would have read compiler died, OpenBSD, /tmp and smelled out of disk before finishing the line. But I wasn't at the keyboard — I'd handed it to the same agent that, a few screens earlier, had typed vio0 into the DNS field. An agent has read about OpenBSD without ever once running out of disk on it, so it had no instinct for the boring failure. It reached instead for the exotic one sitting right there in the dependency graph — the explanation that was plausible, pre-loaded, and a little flattering, because if the crypto stack won't compile you're wrestling something genuinely hard, rather than something you'd have caught in 2003.

And that is the trap worth naming, sharper than "niche platforms are hard." On an unfamiliar system every failure presents as exotic; the strange explanation announces itself and the mundane one doesn't. The further off the beaten path you are, the more deliberately you have to rule out the boring thing first — because out here everything wears the platform's costume, and an out-of-disk error in a one-gigabyte /tmp looks exactly like a dependency that can't be built, right up until someone who's been burned before tells you it's just disk.

What it actually was about

Step back and the whole project rhymes with a pattern I keep meeting. The headline problem — "build OpenBSD/arm64 binaries on AWS" — has a one-sentence answer once you have it: run OpenBSD as a KVM guest on a bare-metal Graviton host. That sentence is the easy part. Everything real lived in the gap between two systems that were each, on their own terms, completely correct.

OpenBSD is correct to omit a driver for hardware almost none of its users run; its restraint is exactly why it's OpenBSD. AWS is correct to present its network through a custom adapter built for the scale it operates at. Neither is wrong. They simply grew up with no expectation of meeting, and the moment you ask them to share a machine, every quiet assumption each one made surfaces at once — the missing driver, the MMIO-versus-PCI transport, the MSI-X interrupts that don't land on first-gen silicon, the DHCP lease that doesn't carry what the installer hoped, the cramped /tmp masquerading as a dependency that wouldn't compile.

The hypervisor in the middle isn't really there to virtualize a CPU. It's there to be the place where two designs that disagree about what hardware is can both be right at the same time. OpenBSD gets virtio and is happy. AWS gets ena and is happy. The seam between them holds because something whose entire job is to translate sits exactly on it.

And the build server that came out the other end is genuinely good: a single command that conjures an ARM machine, runs a native OpenBSD build at full speed, and dissolves the machine when it's done, for the price of a vending-machine snack. The operating system that couldn't see the network now turns out release binaries on the very cloud it can't directly run on — because at no point does anyone ask it to see the network. That was never going to work. The trick was to stop asking.

TinyComputers.io (Posts about uefi)

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal