TinyComputers.io (Posts about graviton)

The Driver Nobody Wrote: OpenBSD's ena(4) Works Now — and Can't Go Upstream

A.C. Jokela — Fri, 26 Jun 2026 23:30:00 GMT

The Driver Nobody Wrote: OpenBSD's ena(4) Works Now — and Can't Go Upstream

A week ago I ended a post about writing OpenBSD's missing ena(4) driver with a confession disguised as a list.

The driver worked, I said, in the one sense that genuinely mattered — the ENA device protocol functioned, implemented from scratch, on real Graviton hardware, proven by the most boring fix in the file. And it did not work in nearly every other sense. So I wrote each of those senses down, because the exciting half of "it works" is forever trying to eat the honest half, and I wanted the honest half on the record. No userland DHCP — I'd driven the round-trip from a kernel thread, not from ifconfig. No disk install — I was booting the install ramdisk out of RAM. Scaffolding everywhere — polling threads, a packet injector, a keep-alive timer I'd built for a problem that turned out not to exist. One DHCP exchange, which is not throughput, not stability, not the dozen edge cases a NIC driver owes the world. Not reviewed. Not submitted. Would not survive tech@, nor should it.

This is the post where I cross them off. Every one.

And then I walk straight into the single wall that crossing them off can't move — a wall that has nothing to do with the code, and everything to do with who, or what, wrote it.

The list, crossed off

Start with the headline item, the one that would let me retire the ridiculous machine from the first post in this series — the bare-metal Graviton running OpenBSD as a QEMU guest behind a virtio shim, because OpenBSD couldn't see the real network card. The milestone was a normal OpenBSD instance: a real disk install, booting multiuser, with the ENA adapter as its only network interface, that I could SSH into over that interface like any other server.

That works now. There is an OpenBSD/arm64 instance on a Graviton2 that boots off an EBS volume, brings up ena0 from a DHCP lease, starts sshd, and accepts my key over the wire. No shim. No emulation. No QEMU host underneath it pretending to be hardware. The kernel talks to Amazon's network card directly, because it finally has a driver that knows how.

Getting there cost me one more bug of exactly the kind the last post was about — and I want to tell it quickly, because it rhymes.

Once the driver could do RAM-disk DHCP, the disk install should have been a formality: partition the EBS volume, extract the sets, write a bootloader, reboot into a real system. It installed fine. It booted fine. And then, the instant it tried to send a packet from the installed system rather than the ramdisk, the device went into FATAL_ERROR and took the interface down with it — reliably, every boot, a few hundred microseconds into the first real transmit.

I had, by this point, learned the lesson the doorbell taught me, and I tried very hard to apply it: look for the boring cause first. But the boring causes all checked out. The transmit ring was set up correctly. The descriptors were well-formed. The doorbell — the memory-mapped register write that tells the device "I've queued work for you," the same doorbell that named the last post — was firing at the right offset with the right value. I spent a day convinced it was a memory-barrier problem, the kind of thing that only shows up on real ARM with real out-of-order completion and never in emulation, because on the QEMU path the same code had been flawless.

It was not a barrier. It was that I was submitting transmit descriptors to the device's submission queue in a batch built for the low-latency LLQ path — the "low-latency queue" mode where you push packet headers directly into device memory — and the Graviton2 instance I was installing onto doesn't use LLQ. It uses the older host-memory path, where the device reads descriptors back out of host RAM, one at a time, and expects them submitted one at a time, each with its own doorbell, in a way the batched path violated. The device wasn't crashing on a subtle race. It was crashing because I was speaking the wrong dialect of its own protocol and it had no polite way to say so.

The fix was to submit host-path descriptors per-descriptor, the way the device's own reference code does, the way the documentation says in a sentence I had read and not absorbed. One commit. The FATAL vanished. The installed system transmitted, and kept transmitting, and I SSH'd into a native OpenBSD/arm64 box on AWS for the first time.

That is the entire pattern of this project in one paragraph: an exotic theory that flatters your understanding, sitting on top of a mundane mistake that flatters nothing. I had a second helping of it on the disk side too — the NVMe driver OpenBSD uses for the EBS volume itself wouldn't create its I/O queues until I clamped a maximum-queue-size field the AWS device reports in a way the stock driver didn't expect. Another one-line fix for a thing that looked, for an afternoon, like it might be deep.

So: disk install, done. SSH-in over ena0, done. The QEMU shim from the first post is, as of this writing, retired — I can build native OpenBSD/arm64 binaries on a real OpenBSD/arm64 instance now, the way the platform's own users would, on a cloud the platform supposedly cannot run on.

The scaffolding is gone too. The polling threads, the packet injector, the diagnostic printfs, the keep-alive timer for the imaginary problem — all stripped out, the real fixes separated from the detritus, the tree readable. It is not the same artifact I described last time as "full of scaffolding, none of it belonging in code anyone else should read." It's a driver now, not a demo with a driver inside it.

Faster than it had any right to be

The last post's most embarrassing admission was the testing: one DHCP round-trip is not throughput. So I went and got throughput, and then I went and got more of it than I expected.

The first honest number was 684 megabits per second — a single queue, a single CPU, real TCP over the real device, measured rather than imagined. That was already a strange feeling, watching a number that meant the data path was not just functional but fast enough to be useful, on a driver I'd been afraid to claim could pass a single packet a week earlier.

Then I built the parts a real NIC driver actually needs, the ones I'd listed as missing. Checksum offload, so the device computes and verifies IPv4/TCP/UDP checksums instead of the CPU — gated on the device advertising the feature, because not every ENA generation does. MTU control and jumbo frames. A watchdog that notices when the device has wedged and tears the data path down and brings it back up without a reboot. And then the big one: multiple queues.

A modern NIC isn't one ring of packets; it's many, one per CPU, so that traffic for different flows lands on different cores and the whole machine scales instead of bottlenecking on a single interrupt. Wiring that up on OpenBSD/arm64 meant allocating per-queue interrupts through the ARM interrupt controller, pinning each queue to its own CPU, and — the part that took longest to get right — receive-side scaling, RSS, where the device itself hashes each incoming packet by its flow and steers it to the correct queue so that packet order within a flow is preserved while load spreads across cores.

RSS is configured by handing the device a hash key and an indirection table: a little array that maps hash buckets to queues. I filled mine with what I was sure were the right queue identifiers, tested it, and got intermittent connectivity — about nine successful connections out of fifteen, which is the worst possible result, because it means you're close enough to be wrong in a way that looks almost right. The indirection table, it turned out, doesn't want the queue's completion-ring index, which is what I'd put there. It wants the queue's submission-ring index, a different number that happens to coincide on some devices and not others. Half my hash buckets were steering to a queue that wasn't listening. One field, the wrong index, surfacing only as a statistical haze of dropped connections. I changed it; twenty connections out of twenty succeeded, and the traffic spread cleanly across both cores.

On a two-vCPU t4g.medium — a deliberately small instance, because the point was to prove the mechanism, not to win a benchmark — the driver now moves 3.16 gigabits per second, balanced across both CPUs, with RSS steering flows to cores and each core fielding its own interrupts. That's not a line-rate claim on a big instance; it's a small machine using both of the hands it has. But it's a multi-queue, RSS-steered, checksum-offloaded network driver doing the actual job, and five weeks ago the honest claim was "one DHCP packet, from a kernel thread."

There was one more thing I wanted, less a feature than a verdict. I'd written and tested everything against OpenBSD 7.9, the release. But OpenBSD's real life happens on -current, the rolling development branch, and a driver that only works against one frozen release is a museum piece. So I built the whole thing against -current — and it compiled and ran with zero source changes, full feature parity, multi-queue and RSS and all. The interfaces I was building against had held. That mattered more to me than the throughput number, because it meant the driver was written against OpenBSD as it actually is, not against a single snapshot I'd reverse-engineered my way into.

By every measure I'd set for myself in that closing list, the driver was done. Which is precisely when I learned where it could not go.

The wall that isn't technical

Here is the thing I believed, quietly, the entire time I was writing this driver: that if I made it good enough — really good, idiomatic OpenBSD, clean bus_dma and honest locking and no vendor-HAL slop, the kind of code that earns its place — there was a path, however narrow, to it going upstream. To ena(4) becoming part of OpenBSD, so that the next person who tries to boot OpenBSD on Graviton doesn't have to write what I wrote. That was never the reason I did it — I did it because the gap was infuriating and the problem was beautiful — but it was the daydream underneath, the one that makes you clean up the scaffolding instead of leaving it.

The daydream is dead, and it died for a reason I didn't see coming and can't really argue with.

OpenBSD doesn't accept AI-generated code — not out of taste, but because code a model wrote has no human author, and with no author there's nothing to hold the copyright and nothing to license under the BSD/ISC terms the tree is built from. A provenance gate, not a quality one. Being good was never the question.

And this driver is AI-assisted in a way I want to be precise about, because it isn't how my other AI-assisted projects work. When I built Lattice, my programming language, or the ballistics engine that kicked off this whole Graviton saga, I was in the loop the entire time. The ideas were mine; I held the design in my head and used the model the way you use a sharp colleague — to think through, to draft against, to argue with. Authorship was never in question, because I was the one making the decisions.

The ena driver was not that. It was hands-off in a way I'd never tried: I pointed an agent at the problem, told it to run in a loop and build an ENA driver for OpenBSD, and let it go. It decided what the driver needed. It decided how to structure the attach path, the queues, the locking. It decided when to read Linux's driver for intent, when FreeBSD's, when NetBSD's, when Amazon's ena-com. I set the direction and the hard constraint — port from BSD-licensed sources, never copy the GPL Linux code — then read what came back and steered when it drifted. But I did not hold this driver in my head the way I held Lattice. For long stretches I was the reviewer of something being authored where I couldn't watch, by something making the decisions a driver's author makes.

And I should be blunter than I was last time. In the Doorbell post I wrote that I built the keep-alive timer and I implemented the host-attributes handshake — clean, correct code, I called it. The honest version is that an agent wrote both while I read along. I let the "I" stand because the project and the loop were mine, and they still are — but the decisions a driver's author makes were the agent's, and I'd rather say that plainly here than let the earlier "I" keep implying otherwise.

Which is why, here, the policy stops being abstract. With Lattice I can sign my name to every architectural choice and mean it. With this, the honest answer to "who decided that?" is, often, the agent did. Machine-authored, human-directed — and directed is not the same word as wrote. I don't think OpenBSD is wrong to refuse it. I think it might be the cleanest example I have of exactly the thing they're refusing.

So ena(4) stays mine. An independent driver, openly AI-assisted, for people who want native OpenBSD on Graviton badly enough to point their kernel config at a tree that isn't the official one. The work-in-progress repo, now considerably less work-in-progress, is at github.com/ajokela/openbsd-ena — open for reading, open for building, open for forking, and closed, by its own nature, to the one destination I'd quietly been building it toward.

There's a strange grief in that I didn't anticipate. Not for the work — the work is done and it runs. For the commons. The natural arc of a thing like this is that you give it away into the shared pool so the next person inherits it, and the reward for doing it well is that it stops being yours and becomes everyone's. This one can't take that arc. It's good enough to belong to everyone and it will belong to no one, because the question "who wrote this?" no longer has an answer the commons can accept. The driver works. The driver has no author. Both of those are true, and the second one is the price of the first.

The gap I couldn't close

If I'm going to be honest about the wall, I have to be honest about the one place the engineering didn't close either.

Everything above is true on Graviton2 — the t4g family, AWS's first ARM generation, the host-memory data path. On Graviton3 — the c7g family, which uses the low-latency LLQ path and a newer revision of the virtual ENA device — the driver attaches, configures, brings up its admin queue, reads every device attribute, and then fails at the very first step of creating an I/O queue. The device rejects the CREATE_CQ command — the request to make a completion ring — with an unhelpful status code, before a single packet has had the chance to flow.

I spent a genuinely unreasonable amount of effort on this. I did the thing systematic debugging tells you to do when a system has multiple components and you can't see inside one of them: I went and instrumented the component that works. I booted Amazon's own Linux on a c7g instance and traced its stock, vendor-blessed ENA driver with bpftrace — every admin command it sends, in order, with its exact arguments, from attach to first packet — so I'd have a known-good transcript to diff mine against.

The transcript refuted, one by one, every theory I had. The bytes of my CREATE_CQ command are identical to Linux's. The interrupt vector layout is identical — same number of MSI-X vectors, same assignment. The completion-descriptor size is identical. The order I create things in doesn't matter — I tried Linux's order exactly and the first queue still fails, whichever kind it is, which means it's not an ordering bug but a missing prerequisite the device wants before it will make any queue at all. I thought I'd found it in the host-info block, the little structure where the driver tells the device about itself — Linux fills in capability flags I was leaving zero — so I filled them in. No change. I thought it was the RSS hash configuration Linux sets up before its queues; I read the device's own feature bitmap and found that this device, like the Graviton2 one, doesn't even allow the host to set the hash function. Refuted by the device itself.

Nine times I baked a kernel and booted it on real c7g hardware to test a hypothesis, and nine times the device said no in the same flat way, and I never found the prerequisite it's waiting for. I have it narrowed to a short list of things Linux does before its first queue that I don't — a couple of feature reads I skip, a second host-info push after negotiation — but "narrowed to a short list, each refuted or untested" is not "solved," and I'm not going to dress it up as solved. The driver runs beautifully on the previous ARM generation and stops at the threshold of the current one, and I wrote down exactly where the threshold is so that whoever picks this up next — me, on a better day, or someone who forks the tree — starts from a map instead of a mystery.

That's the honest shape of it. A driver that's finished on the hardware it's finished on, and has a precisely-documented hole on the hardware it isn't.

The only OpenBSD in the cloud

I'll end where the practical and the strange meet.

At some point, late, I got curious about what I'd actually built relative to what already existed — whether I'd spent two months reinventing something I could have launched in one click. So I searched. Every public Amazon Machine Image, across multiple regions, with "OpenBSD" anywhere in the name or description.

There are none. Not one. Zero community images, zero in the AWS Marketplace, in every region I checked. The entire BSD presence in Amazon's marketplace is FreeBSD — dozens of official, well-maintained, free images published by the FreeBSD Foundation, every release and architecture, ARM and x86, the way a first-class cloud citizen looks. OpenBSD isn't a paid option anyone's gouging for. It isn't an option at all. The only way to run OpenBSD on EC2 today is to build the image yourself, which is to say: to first solve the exact problem these three posts have been about.

So I finished the job. I took the driver, baked it into a real OpenBSD/arm64 image, and made the image self-configuring the way every cloud image is expected to be — it pulls your SSH key, its hostname, and an optional first-boot setup script from the instance metadata service, creates an unprivileged login user, regenerates its own host keys so every instance has a unique identity. OpenBSD doesn't ship the Linux cloud-init and never will; the community answer is a slim shell agent written years ago by an OpenBSD developer, and I wired it in. Launch the image with your own key pair, and you SSH in as a normal user thirty seconds later. It is, as far as I can tell, the only launch-and-go OpenBSD/arm64 image on AWS, because it's the only OpenBSD/arm64 image on AWS.

Which leaves me holding a small, sharp irony. The most finished thing I've built in a while — a driver, a tuned data path, a clean image, a working cloud-init story, the genuinely-only-one-of-its-kind artifact — is the one that can't go where work like this is supposed to go. I could publish it as a public image tomorrow and be the sole OpenBSD on the entire platform. And I'd be publishing it the way I have to publish all of it: off to the side, in my own tree, under my own name, with a note that says an AI helped write this and so it can never be yours, only borrowed.

The first post in this series put a translator between two systems that disagreed, so each could be right without meeting. The second taught one system to speak the other's language directly, register by register, and found the bug that mattered was the most boring one in the file. This one finishes that language — makes it fluent, fast, multi-queue, native, bootable, launchable — and discovers that fluency was never the thing standing between the work and its home. The doorbell rings. The device lives. The network is seen, on a cloud that couldn't see it a season ago.

And the driver that makes it so has no author, and no home, and runs perfectly anyway.

The Doorbell That Killed the Device: Writing OpenBSD's Missing ena(4) Driver

A.C. Jokela — Mon, 22 Jun 2026 01:30:00 GMT

The Doorbell That Killed the Device: Writing OpenBSD's Missing ena(4) Driver

Earlier last week, I wrote about a problem with a clean, almost funny shape: OpenBSD's arm64 kernel has no driver for the network card AWS puts in every EC2 instance, so an OpenBSD/arm64 instance boots into a billable void it can never reach. The fix in that post was to stop asking — to run OpenBSD as a KVM guest inside QEMU on a bare-metal Graviton host, hand it virtio devices it already understands, and let a Linux host own the real Amazon adapter. Put a translator on the seam, and two systems that disagree about what a network card is can both be right.

That solved the build-server problem. It did not solve the itch.

Because the actual missing piece is small and specific and writable: the driver. Amazon's adapter is called the Elastic Network Adapter — ena — and it's a documented device with a permissively licensed reference implementation. FreeBSD has an ena driver. NetBSD has one. The protocol is published. OpenBSD just doesn't have the code, because effectively nobody runs OpenBSD as a first-class EC2 guest and so nobody wrote it. "Nobody wrote it" is not the same as "can't be written." So I decided to write it.

This is the story of that — and, more than the previous post, it's a story about a specific kind of failure. The driver attached on the first serious try. Then the device started killing itself, silently, a few microseconds after every bring-up, and I spent the better part of two days proposing increasingly clever reasons why. The real reason turned out to be embarrassingly dull. And because I want this to be useful and not just triumphant, the last third of the post is about the word "working": what I can actually claim, and the uncomfortable distance between that and "done."

What an ENA driver has to do

A modern NIC isn't a thing you poke registers at to send a packet. It's a small message-passing computer that shares host memory with you: rings of descriptors in DMA memory, doorbell registers to announce new work, an interrupt path for completions. ENA has three kinds of queue, and all three matter to this story.

The admin queue is how the host configures the device — a submission ring and a completion ring in shared memory. Write a command (read attributes, set a feature, create an IO queue), ring the admin doorbell, wait for a completion. OpenBSD's driver polls for that completion rather than taking an interrupt, which keeps bring-up simple.

The AENQ — Asynchronous Event Notification Queue — is the device's back-channel: it announces link up/down and, critically, posts a keep-alive event about once a second. The keep-alive is a heartbeat; the host is expected to drain these and, by draining them, prove it's still paying attention.

The IO queues are the actual network — a submission ring you fill with buffers, a completion ring the device writes back. You create them with admin commands (CREATE_CQ, then CREATE_SQ), and once they exist and the link is up, you can move traffic.

Get all three right, in the right order, and the card works. Get the order subtly wrong and — as I'd learn — the card decides you're not a real driver and quietly bricks itself.

The reference is Amazon's ena-com, a hardware-abstraction layer shared across the Linux and FreeBSD drivers. Its BSD-licensed parts — ena-com itself and FreeBSD's driver on top — are fair to read and port; the Linux driver is GPL, kept strictly read-only, a thing to consult for intent and never to copy. Writing the OpenBSD version means rewriting all of it in OpenBSD's idiom anyway — bus_dma(9) for the rings, pci(9) for attachment, ifnet/ifq for the stack. The protocol is the spec; the code is yours.

Phase zero: it attaches

The first milestone was just attachment, and it went well enough that I'd half-convinced myself the hard work was behind me. The driver resets the device, sets up the admin submission and completion rings, and issues commands. GET_FEATURE(DEVICE_ATTRIBUTES) comes back with the real MAC and maximum MTU — proof the admin queue works end to end, DMA is coherent, the device is listening. The console prints the line I'd been chasing: ena0 ... ENA ver 0.10 ... address 12:xx:xx:xx:xx:xx, on a real Graviton instance. The card was talking.

And then, every single time, a few seconds later, it stopped talking. The first attempt to create an IO queue — CREATE_CQ, the command that begins turning a configured device into a working network interface — would sit there and time out. No completion. No error. The admin queue that had just answered four commands flawlessly had gone silent.

When I finally added code to read the device's status register at each step, the shape of it came into focus and got worse. DEV_STS reads 0x1 — ready — through reset, through the admin handshake, through reading device attributes. Then, somewhere shortly after, it reads 0x21. Bit five is set. FATAL_ERROR. The device had, of its own accord, entered a fault state and was now refusing all further work. That's why CREATE_CQ vanished: you can't drive a device that's already decided it's dead.

A healthy card, healthy through every step I could see, that turns to stone the moment I look away. That's the wall.

Five wrong theories

Here is the part I'm telling on myself, because it's the honest center of the whole thing.

When you don't know why a device faults, the device gives you almost nothing — a single bit that says "something is wrong" and not a syllable about what. So you reason from the reference code: what does the working driver do that mine doesn't? And the trouble with that question, on hardware this unfamiliar, is that it has too many plausible answers. Every difference between my driver and ena-com looks like it could be the one that matters.

Theory one: the keep-alive watchdog. The device sends heartbeats; the host must drain them; if the host stops, surely the device fences off the unresponsive driver and faults. My driver had no periodic task draining the AENQ — it relied on a single interrupt that, I could see, fired exactly once and went quiet. This was a beautiful theory. It explained the timing. It matched a real mechanism FreeBSD implements. I built a proper one-second timer to drain the queue, mirroring FreeBSD's ena_timer_service, complete with a mutex to keep the timer and the interrupt from racing on a multi-core guest. It was clean, correct code. It fixed nothing. The device faulted on exactly the same schedule, and the diagnostic I'd added showed the AENQ had processed zero events — there was nothing to drain. I had carefully solved a problem that wasn't happening.

Theory two: the MMIO response region. ENA has a "readless" register mode backed by a small DMA region; maybe the device faulted without it. I added it. It didn't help — and worse, I'd added it before I started reading the status register, so for an embarrassingly long time I was carrying an unvalidated change that could have been the cause itself. (It wasn't.)

Theory three: host attributes. Both reference drivers register a 4 KB "host info" page right after admin init — SET_FEATURE(HOST_ATTR_CONFIG), a "yes, a real driver lives here" handshake. My driver skipped it. This had to be what the device validated before deciding I was legitimate. I implemented it properly. The status register read 0x1 right after it succeeded — and faulted anyway, at the same point as always.

Theory four: a stale completion. Maybe the admin queue was reading the wrong completion slot, so the feature data showing the device supporting zero AENQ event groups — a suspicious value — was garbage from an uninitialized ring. I instrumented the completion path down to the command IDs and phase bits. It was reading the right slot. The suspicious value was real.

Theory five: interrupt ordering. I'd unmasked the device's interrupt after a particular doorbell write; the reference does it before. I swapped the order. The device faulted one line later than before, which felt like progress and was not.

Somewhere in the middle I did what I increasingly do with a problem that has too many branches: I handed it to a fleet of AI agents, one per hypothesis, reading the reference trees in parallel, and had them synthesize a ranked root cause. The synthesis came back confident, specific, and wrong. What saved it was the same harness's adversarial step — three more agents told to refute the conclusion, and all three did, pointing out the timing didn't fit and the real anomaly was being hand-waved. The machine talked itself out of its own clever answer. That's the part worth remembering: not "the AI solved it," but "the AI proposed something plausible and the only thing that caught it was forcing a second pass that tried to tear it down."

What none of the five theories were was boring enough.

The doorbell

I gave up on theories and did the dumb, mechanical thing I should have done first. I made the driver poll the status register in a tight loop after every single register write in the bring-up, printing the exact moment the fault bit flipped. Not "is it healthy at the end" — which write kills it.

The answer came back in one line, and it was not ambiguous. Healthy after writing the queue's base address. Healthy after writing its size. Healthy after the feature commands. Then:

STS->FATAL 0x21 at [post-AENQ_HEAD_DB] iter=3

The fault appears about 150 microseconds after one specific write: the AENQ head doorbell — the register that tells the device "I've made the event ring's slots available; the queue is live." Not the interrupt unmask, which I'd swapped order on. Not any feature command. The doorbell that activates the event queue. Ring it, and the device dies.

The doorbell value was correct — I'd checked it against the reference a dozen times. The ring's address was correct. Its size was correct. So why would activating a correctly-configured queue kill a healthy device?

Because the queue wasn't really configured. It only looked like it was.

The most boring possible cause

I lined my driver's bring-up sequence up against ena-com's, write for write, and the difference was finally visible because I now knew exactly which write to care about.

ena-com registers the AENQ — writes its base address and size registers — inside ena_com_admin_init, in the same breath as the admin submission and completion queues, before the device is ever told initialization is finished. All three rings get registered together, as one atomic-feeling handshake, while the device is still in its "setting up" phase.

My driver registered the admin queues during admin init, exactly like the reference. But it registered the AENQ much later, in a separate function that ran after host attributes, after reading device attributes, after feature negotiation — long after the admin handshake had closed and the device considered itself up and running.

And here's the thing: writing those AENQ registers late worked, in the sense that the device accepted the writes and stayed healthy. The register values landed. The status bit stayed green. Everything looked fine. The device had quietly noted the address and size of a ring it had never actually wired into its event subsystem, because that wiring only happens during the init handshake I'd already finished. The AENQ was a ghost: registered on paper, uninitialized in the device's mind.

Then I rang the doorbell. "The event queue is live; start using it." The device went to use a subsystem that was never set up, and faulted. A hundred and fifty microseconds later, the bit flipped.

The fix is four lines moved earlier. I pulled the AENQ registration out of its late function and into admin init, immediately after the admin queues, exactly where ena-com does it. The later function kept only the parts that genuinely belong late — subscribing to event groups, and the final doorbell-and-unmask that says "go."

I rebuilt, booted, and watched the status register stay 0x1 straight through the doorbell. CREATE_CQ succeeded. CREATE_SQ succeeded. The link came up. The driver enqueued a hand-built DHCP DISCOVER, the device transmitted it, and a 590-byte IPv4 packet — the DHCP OFFER, a real reply from AWS's network — came back up the receive ring. Transmit and receive, on real silicon, for the first time.

There was one more gift in the logs, the kind that tells you a fix is right and not just lucky. Remember theory four — the device reporting zero supported AENQ event groups, the value I'd half-dismissed as a possible misread? With the AENQ now registered during init, that same query on the same hardware came back reporting all the groups supported. The zero had never been a VF limitation or a misread. It was the device telling me, in the one channel it had, that its event subsystem wasn't initialized — because I hadn't initialized it yet. One root cause had been wearing five costumes. The watchdog that had nothing to drain, the missing handshake, the suspicious zero, the doorbell fault — all of it was the single fact that I'd set up a queue in the wrong order, refracted through a device that can only ever tell you "something is wrong."

This is the same lesson the last post ended on, and I clearly didn't learn it hard enough the first time. Out here, far off the beaten path, every failure presents as exotic, because the strange explanation announces itself and the boring one doesn't. An out-of-disk error wears the costume of a dependency that won't compile. A queue initialized in the wrong order wears the costume of a keep-alive watchdog, a missing security handshake, and a device that lies about its capabilities. The further out you are, the more deliberately you have to rule out the dull thing first — and "I did the steps in the wrong order" is about as dull as it gets.

The loop that made it bearable

I have to mention the iteration speed, because for most of this saga it was the actual bottleneck, and fixing it is what turned a slog into something tractable.

The previous post's build server bakes a disk image, snapshots it, registers an AMI, boots a real EC2 instance, and reads the serial console — about eighteen minutes a turn. That's fine for building a binary. It is agony for debugging a driver, where you want to change one line and see what the hardware does. Eighteen minutes times the number of wrong theories above is a number I'd rather not compute.

So I built a faster loop, and it leans on the same bare-metal host the build server already needs. That a1.metal host has a real ARM SMMU — the IOMMU that makes device passthrough safe — and Linux's VFIO framework can hand a physical PCI device straight to a QEMU guest. So I attached a second network interface to the metal instance, bound it to vfio-pci on the Linux host (leaving the primary NIC alone, so I didn't saw off the SSH branch I was sitting on), and passed it through to the OpenBSD guest. Now the OpenBSD VM sees a real ENA device — actual Amazon silicon, vendor 1d0f, product ec20 — on its virtual PCI bus, and my driver attaches to that. No AMI bake. Build the kernel, reboot the guest, watch ena0 come up against real hardware. Two minutes a turn instead of eighteen.

There was one gotcha worth writing down, because the error is opaque: VFIO refused with failed to set iommu for container: Operation not permitted. ARM's SMMU here can't remap interrupts, so the kernel blocks passthrough by default — fixed with the module parameter allow_unsafe_interrupts=1, entirely fine for a trusted device on a machine I own by the hour.

That loop is also the honest reason the fix arrived when it did. The "poll after every write" instrumentation only became practical once I could run it and read it in two minutes. The clever theories flourished in the eighteen-minute dark; the boring method won the moment I could see in real time.

What "working" means

Here is where I have to be careful, because "I wrote a working ENA driver for OpenBSD" is a sentence that can mean five very different things, and only one of them is true today.

What is true: on a real EC2 Graviton2 instance, an OpenBSD/arm64 kernel with my driver attaches to the real ENA adapter, completes the full bring-up — admin queue, host attributes, device attributes, AENQ, IO queue creation — brings the link up, transmits a packet the device really puts on the wire, and receives the reply. The send and receive paths through the driver are the real ones: the transmit path is the same ifq enqueue the network stack uses, and the receive path is the same completion handler that would feed packets up to IP. A DHCP DISCOVER went out and an OFFER came back. The device protocol — the genuinely hard, genuinely undocumented-in-OpenBSD-idiom part — works, and it works on the production hardware, not just the passthrough rig.

First boot. OpenBSD/arm64 on a real Graviton2 EC2 instance, ena(4) attached, link up, DHCP round-trip completed against the real device.

What is not yet true, and I want to be exact about each one:

I drove that DHCP exchange from a kernel thread, not from userland. The test harness builds a DISCOVER packet in the kernel, hands it to the transmit ring, and watches the rings directly, because the minimal RAM-disk environment I'm booting doesn't have room for a real dhclient. The packets and the ring mechanics are real; the thing wrapping them is a debug scaffold, not ifconfig ena0 up; dhclient ena0; ping. I have not yet typed those three commands at a shell and watched them work. That's the next milestone, and until I've done it I won't claim the interface works "from userland," only that the driver's data paths do.

I am booting the OpenBSD install ramdisk, bsd.rd, which runs entirely from RAM. I have not done a full disk install and booted a persistent OpenBSD that comes up multiuser with ena0 as its only network interface and lets me SSH in over it. That — a normal OpenBSD instance you log into over the network it sees natively — is the milestone that would let me retire the QEMU-shim build server from the last post. I'm not there. I've proven the hard part is possible; I haven't assembled it into a system you'd actually run.

The driver is full of scaffolding. The status-register polling, the packet-injection thread, a dozen diagnostic printfs, the keep-alive timer I built for a problem that didn't exist — all still in the tree, behind a debug flag. None of it belongs in code anyone else should read. Before this is a contribution rather than a demo, that all comes out, the real fixes get separated from the detritus, and the whole thing gets the kind of review OpenBSD's tree rightly demands. It has had none of that. No OpenBSD developer has looked at a line of it. It is not submitted, not reviewed, and would not survive tech@ in its current state, nor should it. The work-in-progress tree, scaffolding and all, lives at github.com/ajokela/openbsd-ena — open for reading, not for trusting.

And the testing is thin. One DHCP round-trip is not throughput, not stability under load, not days of uptime, not the dozen edge cases — checksum offload, multi-queue, MTU changes, link flaps — a NIC driver has to handle before anyone trusts it. I've shown the path is real. I have not shown it's robust.

So: working in the sense that the central, doubted, genuinely difficult thing — does the device protocol function, correctly implemented from scratch, on real hardware — is now answered yes. Not working in the sense of something you'd deploy, or even the sense of something you'd ifconfig by hand yet. Both halves of that sentence are true and I don't want the exciting half to eat the honest one.

What it was about, again

The last post put a translator between two systems that disagreed, so each could be right without meeting. This one is the opposite move: no shim, just teaching one system to speak the other's language directly — a driver doing the actual work of turning OpenBSD's idea of a network interface into ENA's, register by register and ring by ring.

But the deeper rhyme isn't the architecture, it's the failure. Both times the headline problem had a one-sentence answer — "run it as a guest," "register the queue during init" — and both times that sentence was the easy part, with the real work in a gap where everything looked exotic and the truth was mundane. And both times the trap was the same: a plausible, faintly flattering, wrong explanation is far more available than the boring one underneath it — especially with a tireless machine happy to generate plausible explanations on demand. The machine is genuinely useful; it read three reference drivers in parallel and caught its own bad guess on the second pass. But it has no instinct for "you probably just did the steps out of order," because it has never spent an afternoon being humiliated by exactly that.

What I have now is a driver that makes OpenBSD see the network on a cloud that, a month ago, OpenBSD couldn't see at all. It is not done. It is, for the first time, possible — proven on the hardware, by the most boring fix in the file. The doorbell rings, and the device lives.

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

A.C. Jokela — Thu, 18 Jun 2026 19:45:00 GMT

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

I have a Rust project — a ballistics engine — that ships binaries for an embarrassing number of platforms. Linux on x86 and ARM, Windows, FreeBSD, NetBSD, and, because someone always asks and because I like the discipline of it, OpenBSD. Most of those targets are easy: a build matrix, a cross-compiler, a Docker image with the right musl toolchain, and the artifacts fall out the other end. The BSDs are where the matrix stops being polite.

For FreeBSD and NetBSD on arm64 I already had a clean answer: spin up a real Graviton instance, build natively, collect the binary, terminate the instance. A build costs a few cents and finishes in minutes, because the hardware is real ARM and there's no emulation tax. I'd written a little orchestrator for exactly this — launch, build, tear down — and it worked beautifully for two of the three BSDs.

OpenBSD was the one that wouldn't fall in line. And the reason is so clean, so completely a consequence of what OpenBSD is, that it took me a while to stop treating it as a bug to be worked around and start treating it as a fact to be designed around.

This is the story of building native OpenBSD/arm64 binaries on a cloud that OpenBSD, in the most literal sense, cannot see.

Why native, and why that's suddenly hard

The honest first question is whether native builds matter at all. Couldn't I cross-compile OpenBSD/arm64 binaries from Linux and skip the whole circus?

For a trivial program, maybe. For anything real, no. Cross-compiling to OpenBSD means reproducing its libc, its linker behavior, its system call conventions, and its packaging quirks in a toolchain that lives somewhere else — and then trusting that the binary you produced behaves the way a binary built on OpenBSD would. The entire value of shipping an OpenBSD artifact is the claim "this was built and linked on the platform it runs on." A cross-compiled binary that mostly works is worse than no binary at all, because it fails in ways your users discover and you can't reproduce.

So: native. Build it on OpenBSD/arm64, on real ARM silicon, the way the platform's own users would.

The old way to get there was emulation — QEMU on an x86 host, software-emulating an ARM CPU. It works, and it's miserably slow: with no hardware virtualization to lean on, every guest instruction gets translated, and a four-minute build stretches to twenty-five. I did not want to go back.

The new way was supposed to be the Graviton instance: real ARM cores, native speed, terminate when done. It works for FreeBSD. It works for NetBSD. And then you try it with OpenBSD and walk straight into the wall.

The wall: OpenBSD has no driver for AWS's network card

Here is the entire problem, and it is almost funny once you see it.

Modern EC2 instances present their network to the guest through the Elastic Network Adapter — ena, a custom Amazon device with a custom driver. Every cloud-friendly OS ships that driver: Linux has it, FreeBSD has it, NetBSD's community AMIs have it. It is the thing that lets an instance have a network at all.

OpenBSD's arm64 kernel does not have an ena driver.

This isn't an oversight, and it isn't a slight. OpenBSD is a small, fiercely curated tree. Drivers get in because someone runs the hardware, writes clean code, and maintains it — and effectively nobody runs OpenBSD as a first-class EC2 guest, so the driver for Amazon's proprietary NIC simply isn't there. The project doesn't chase cloud hardware the way a commercial OS does, and that restraint is a feature, not a failing. But it has a consequence: an OpenBSD/arm64 instance on EC2 boots, comes up, and finds no network interface it understands. The NIC is right there on the PCI bus, and the kernel has nothing to bind to it.

An EC2 instance you cannot reach over the network is not an instance. It's a billable void. There's no SSH, no console you can usefully drive, no way to hand it a build and get a binary back. The OS runs perfectly; it just can't talk to the one piece of hardware that makes a cloud instance a thing you can use.

I sat with that for a bit. You can't add the driver from outside. You can't give OpenBSD a NIC it understands by configuring the instance differently, because AWS only offers ena. The platform and the OS disagree about reality at the level of the network card, and neither one is going to move.

So I stopped trying to make them agree.

Stop asking OpenBSD to talk to AWS

The move, once it arrived, was obvious in the way these things always are in retrospect: don't let OpenBSD touch the cloud's hardware at all.

Put a Linux host on the metal. Linux has the ena driver; it brings the network up without complaint. Then run OpenBSD as a virtual machine inside that Linux host, under QEMU with KVM acceleration — and hand the guest the kind of hardware OpenBSD does understand: virtio. OpenBSD has had solid virtio drivers for years. It sees a virtio network card and a virtio disk and is perfectly happy, never knowing or caring that the physical NIC underneath is an Amazon device it couldn't have spoken to directly.

The Linux host owns the ena and the public IP. The OpenBSD guest lives behind it on a private virtual network, reachable through a port forward. The two layers each get hardware they recognize, and the impedance mismatch — the missing driver — simply stops mattering, because the only thing that ever talks to the ena is Linux.

It's the same shape as a hundred other infrastructure problems: when two layers can't agree, you don't force them, you put a translator between them. Here the translator is an entire hypervisor, and the thing being translated is the basic question of what a network card is.

This also quietly solves the speed problem that made x86 emulation unbearable. Because the host is itself ARM, KVM can run the ARM guest at near-native speed — no instruction translation, just virtualization. OpenBSD/arm64 on an ARM host under KVM boots in seconds and builds at real-hardware pace. The thing that was slow about the old emulation approach was never "virtualization"; it was cross-architecture emulation. Match the architectures and the tax disappears.

Why it has to be bare metal

There's a catch, and it's the reason this costs more than a few cents.

KVM needs hardware virtualization extensions, and on AWS those are only exposed to the guest OS on bare-metal instances. On an ordinary virtualized Graviton instance, the Nitro hypervisor already owns the CPU's virtualization layer; your kernel can't open /dev/kvm because something else is already standing where KVM would stand. Try to accelerate QEMU on a normal instance and it silently falls back to the slow software path — right back to the emulation tax I was trying to escape.

So the host has to be .metal. The cheapest bare-metal Graviton AWS sells is a1.metal — first-generation Graviton, sixteen Cortex-A72 cores, and, crucially, a real /dev/kvm exposed to the host kernel. It's about forty-one cents an hour on demand. That's more than the few-cents-per-build the native FreeBSD path costs, but a build spins the instance up, runs, and terminates in fifteen or twenty minutes, so a release still costs well under a dollar.

I picked Ubuntu for the Linux host — its QEMU and UEFI firmware packages are exactly what the recipe wants — pointed it at the right ARM AMI, and gave it a roomy root volume. The host comes up, apt installs QEMU and the AAVMF UEFI firmware, and now I have an ARM machine with KVM, ready to boot an OpenBSD guest. That part was easy.

The part that ate an afternoon was getting OpenBSD to actually attach its own disks.

Two device-tree gotchas that ate an afternoon

QEMU's generic ARM machine type, virt, gives you a clean, modern, device-tree-described board. You hand it virtio devices and it wires them up. I did the obvious thing — a virtio block device for the installer image, a virtio block device for the target disk, a virtio network device — booted, and watched the OpenBSD installer come up, reach the disk-selection step, and announce:

Available disks are: none.

No disks. Also, quietly, no network interface. Both virtio devices were there — UEFI had even booted the installer kernel off one of them — and yet the running OpenBSD kernel saw neither.

The cause is a transport mismatch that's easy to miss. On the virt machine, virtio-blk-device and virtio-net-device use the virtio-MMIO transport — devices described in the device tree, sitting at fixed memory addresses. UEFI can boot from those. OpenBSD's arm64 kernel, it turns out, does not attach them. What it does attach is virtio-PCI — the same virtio devices presented as PCI devices on the PCIe controller the kernel already probes. Swap virtio-blk-device for virtio-blk-pci and virtio-net-device for virtio-net-pci, and suddenly the disks appear as sd0 and sd1 and the NIC as vio0. One word per device, and the installer can see the world.

That fixed the disks. The very next boot hung.

Not crashed — hung. The kernel probed all the devices, printed them, and then stopped cold, right before mounting its root filesystem. The host's load average sat at zero: the guest CPU wasn't spinning, it was blocked, waiting for something that was never going to come. The something was an interrupt.

virtio-PCI devices default to MSI-X — message-signaled interrupts, delivered on ARM through the GIC's Interrupt Translation Service. On this first-generation Graviton host, under KVM, that MSI path doesn't deliver. The guest arms its interrupts, waits for the first one, and waits forever. The fix is to tell QEMU to give each virtio device zero MSI-X vectors, forcing it back to plain old pin-based (INTx) interrupts, which ride the GIC's distributor directly and work fine:

-device virtio-blk-pci,drive=hd0,bootindex=1,vectors=0
-device virtio-net-pci,netdev=net0,vectors=0

With vectors=0 on every virtio device, the device line in the boot log changes from msix to a plain irq, the kernel gets its interrupt, mounts root, and reaches the installer prompt. Two one-line changes — the PCI transport and the disabled MSI-X — separated by an afternoon of staring at a boot that stopped in exactly the same place every time.

Neither of these is in the friendly "run OpenBSD under QEMU" recipes you'll find, because those recipes assume a host where MSI-X works and where someone already knew to use the PCI transport. The combination of this OS on this generation of silicon under this hypervisor is niche enough that you find the edges yourself.

Installing an OS through a serial straw

The virt machine has no graphics — no VGA, no framebuffer worth the name. OpenBSD's installer comes up on the serial console, which QEMU helpfully wires to standard out. That's fine for a human at a keyboard. It's less fine when you're trying to make the whole thing repeatable, because the OpenBSD installer is a long, interactive conversation: hostname, network, disk, sets, passwords, two dozen prompts.

OpenBSD has an answer for this — autoinstall, where a response file answers every prompt and the install runs unattended. I tried it first, serving the response file over HTTP from the Linux host. It failed in an instructive way: QEMU's built-in user-mode network doesn't advertise an autoinstall server in its DHCP lease, and when the installer falls back to asking where the response file is, it tries to fetch it before the guest's network is actually up. The fetch fails, autoinstall aborts, and you're dropped to a shell. The unattended path assumes a network environment QEMU's toy DHCP doesn't provide.

So the install came down to driving the interactive program by hand — except I didn't drive it. I let Claude drive, programmatically: QEMU running inside a tmux session, the agent feeding the serial console one answer at a time and reading the screen back between each step. It's a strange way to install an operating system — an AI typing into a virtual serial port through a terminal multiplexer over SSH — and it fumbled in telling ways. It stuffed answers ahead of the prompts and desynced, at one point confidently typing vio0 into the DNS domain name field because it had guessed the question order and the installer disagreed; a mismatched password pair followed, and I watched it back out and resynchronize. Once it was reading each prompt instead of predicting it, the rest was deterministic, and it pulled the install sets straight off the installer image so the guest never needed working DNS at all. Whole-disk GPT, auto layout, sets from the local media, and a few minutes later: CONGRATULATIONS! Your OpenBSD install has been successfully completed!

Drop the installer image, reboot from the disk alone, and the installed system comes up on its own — generates its host keys, starts sshd, reaches a login prompt. uname confirms it: OpenBSD 7.9, arm64, on a "QEMU KVM Virtual Machine." A real, persistent OpenBSD/arm64 system, running on a cloud that can't host it directly.

Making it ephemeral: the golden image and the one-button build

A hand-installed VM is a proof of concept, not a build server. The point was always to make this disappear into a command — spin the whole stack up, build, collect the artifact, spin it down, and pay pennies.

So I did the slow part exactly once. I took that freshly installed guest, added the toolchain a build needs — git, the Rust compiler — and a build key whose public half lives in the image's authorized_keys. Then I shut it down cleanly and compressed the disk: a 20 GB virtual volume holding maybe two gigs of real data squeezed down to an 832 MB qcow2. That golden image went into S3, in the same region as the metal hosts, so pulling it back down is fast and free.

Around it I wrote an orchestrator — the OpenBSD sibling of the FreeBSD/NetBSD tool I already had. One command does the whole dance: launch an a1.metal, wait for it, install QEMU, pull the golden image from S3 through a presigned URL (so the host needs no AWS credentials of its own), boot the OpenBSD guest with the exact virtio-PCI-and-vectors=0 incantation the afternoon taught me, and wait for the guest's SSH to answer. The guest is reachable only through a port forward on the host, so the orchestrator jumps through the host to reach it — the build key never leaves my laptop, and no extra port is ever opened to the world. Then it pipes the build into the guest, runs it natively, copies the binary back out, checksums it, and — in a finally block, so it happens whether the build succeeds, fails, or throws — terminates the metal host.

Spin up, build, collect, spin down. The instance exists for the length of one build and then it's gone, and the bill is loose change.

The last wall: the build that wouldn't build

I ran the orchestrator end to end, and it was beautiful right up until the part that mattered. It launched the host, pulled the image, booted OpenBSD, SSH'd in, started the build — and the build died compiling a TLS library, deep in my engine's dependency tree.

I knew exactly what it was. Or I thought I did. My ballistics engine has an optional "online" feature, on by default, that pulls in an HTTP client and with it a whole TLS stack — rustls, the ring crypto backend, the works. Of course that was the thing that wouldn't build on an OS this far off the beaten path. I turned the feature off, the build went green, and I jotted down a tidy little lesson about exotic platforms and the liabilities of convenient defaults.

The lesson was wrong, and I found out a few hours later. The release binary that already ships for this exact platform has that entire TLS stack compiled into it — rustls builds on OpenBSD/arm64 perfectly well. So that was never the wall. The real wall was disk. OpenBSD's installer auto-layout splits the disk into a fistful of small partitions, and the build was running in /tmp — about a gigabyte. A Rust release target/ directory does not fit in a gigabyte, and rustls happens to be a large crate, so the partition filled mid-compile and the compiler fell over with an error that looked like a code problem. Turning off the feature "fixed" it only because a leaner build squeaked under the limit. The actual fix was one line — build in /usr/obj, the roomy partition OpenBSD hands you for exactly this — and with room to work, the full build, TLS and all, finishes fine.

Here's the part that stings, and the reason I'm telling it on myself. I am not new to this. I've dabbled in the BSDs for a long time — FreeBSD was my daily driver in the early 2000s — and I know OpenBSD well enough that its installer holds no surprises; it has barely changed in twenty years. "OpenBSD carves the disk into small partitions and you do your building in /usr/obj" is not arcana to me, it's reflex. A version of me at the keyboard would have read compiler died, OpenBSD, /tmp and smelled out of disk before finishing the line. But I wasn't at the keyboard — I'd handed it to the same agent that, a few screens earlier, had typed vio0 into the DNS field. An agent has read about OpenBSD without ever once running out of disk on it, so it had no instinct for the boring failure. It reached instead for the exotic one sitting right there in the dependency graph — the explanation that was plausible, pre-loaded, and a little flattering, because if the crypto stack won't compile you're wrestling something genuinely hard, rather than something you'd have caught in 2003.

And that is the trap worth naming, sharper than "niche platforms are hard." On an unfamiliar system every failure presents as exotic; the strange explanation announces itself and the mundane one doesn't. The further off the beaten path you are, the more deliberately you have to rule out the boring thing first — because out here everything wears the platform's costume, and an out-of-disk error in a one-gigabyte /tmp looks exactly like a dependency that can't be built, right up until someone who's been burned before tells you it's just disk.

What it actually was about

Step back and the whole project rhymes with a pattern I keep meeting. The headline problem — "build OpenBSD/arm64 binaries on AWS" — has a one-sentence answer once you have it: run OpenBSD as a KVM guest on a bare-metal Graviton host. That sentence is the easy part. Everything real lived in the gap between two systems that were each, on their own terms, completely correct.

OpenBSD is correct to omit a driver for hardware almost none of its users run; its restraint is exactly why it's OpenBSD. AWS is correct to present its network through a custom adapter built for the scale it operates at. Neither is wrong. They simply grew up with no expectation of meeting, and the moment you ask them to share a machine, every quiet assumption each one made surfaces at once — the missing driver, the MMIO-versus-PCI transport, the MSI-X interrupts that don't land on first-gen silicon, the DHCP lease that doesn't carry what the installer hoped, the cramped /tmp masquerading as a dependency that wouldn't compile.

The hypervisor in the middle isn't really there to virtualize a CPU. It's there to be the place where two designs that disagree about what hardware is can both be right at the same time. OpenBSD gets virtio and is happy. AWS gets ena and is happy. The seam between them holds because something whose entire job is to translate sits exactly on it.

And the build server that came out the other end is genuinely good: a single command that conjures an ARM machine, runs a native OpenBSD build at full speed, and dissolves the machine when it's done, for the price of a vending-machine snack. The operating system that couldn't see the network now turns out release binaries on the very cloud it can't directly run on — because at no point does anyone ask it to see the network. That was never going to work. The trick was to stop asking.