TinyComputers.io (Posts about kernel)

The Driver Nobody Wrote: OpenBSD's ena(4) Works Now — and Can't Go Upstream

A.C. Jokela — Fri, 26 Jun 2026 23:30:00 GMT

The Driver Nobody Wrote: OpenBSD's ena(4) Works Now — and Can't Go Upstream

A week ago I ended a post about writing OpenBSD's missing ena(4) driver with a confession disguised as a list.

The driver worked, I said, in the one sense that genuinely mattered — the ENA device protocol functioned, implemented from scratch, on real Graviton hardware, proven by the most boring fix in the file. And it did not work in nearly every other sense. So I wrote each of those senses down, because the exciting half of "it works" is forever trying to eat the honest half, and I wanted the honest half on the record. No userland DHCP — I'd driven the round-trip from a kernel thread, not from ifconfig. No disk install — I was booting the install ramdisk out of RAM. Scaffolding everywhere — polling threads, a packet injector, a keep-alive timer I'd built for a problem that turned out not to exist. One DHCP exchange, which is not throughput, not stability, not the dozen edge cases a NIC driver owes the world. Not reviewed. Not submitted. Would not survive tech@, nor should it.

This is the post where I cross them off. Every one.

And then I walk straight into the single wall that crossing them off can't move — a wall that has nothing to do with the code, and everything to do with who, or what, wrote it.

The list, crossed off

Start with the headline item, the one that would let me retire the ridiculous machine from the first post in this series — the bare-metal Graviton running OpenBSD as a QEMU guest behind a virtio shim, because OpenBSD couldn't see the real network card. The milestone was a normal OpenBSD instance: a real disk install, booting multiuser, with the ENA adapter as its only network interface, that I could SSH into over that interface like any other server.

That works now. There is an OpenBSD/arm64 instance on a Graviton2 that boots off an EBS volume, brings up ena0 from a DHCP lease, starts sshd, and accepts my key over the wire. No shim. No emulation. No QEMU host underneath it pretending to be hardware. The kernel talks to Amazon's network card directly, because it finally has a driver that knows how.

Getting there cost me one more bug of exactly the kind the last post was about — and I want to tell it quickly, because it rhymes.

Once the driver could do RAM-disk DHCP, the disk install should have been a formality: partition the EBS volume, extract the sets, write a bootloader, reboot into a real system. It installed fine. It booted fine. And then, the instant it tried to send a packet from the installed system rather than the ramdisk, the device went into FATAL_ERROR and took the interface down with it — reliably, every boot, a few hundred microseconds into the first real transmit.

I had, by this point, learned the lesson the doorbell taught me, and I tried very hard to apply it: look for the boring cause first. But the boring causes all checked out. The transmit ring was set up correctly. The descriptors were well-formed. The doorbell — the memory-mapped register write that tells the device "I've queued work for you," the same doorbell that named the last post — was firing at the right offset with the right value. I spent a day convinced it was a memory-barrier problem, the kind of thing that only shows up on real ARM with real out-of-order completion and never in emulation, because on the QEMU path the same code had been flawless.

It was not a barrier. It was that I was submitting transmit descriptors to the device's submission queue in a batch built for the low-latency LLQ path — the "low-latency queue" mode where you push packet headers directly into device memory — and the Graviton2 instance I was installing onto doesn't use LLQ. It uses the older host-memory path, where the device reads descriptors back out of host RAM, one at a time, and expects them submitted one at a time, each with its own doorbell, in a way the batched path violated. The device wasn't crashing on a subtle race. It was crashing because I was speaking the wrong dialect of its own protocol and it had no polite way to say so.

The fix was to submit host-path descriptors per-descriptor, the way the device's own reference code does, the way the documentation says in a sentence I had read and not absorbed. One commit. The FATAL vanished. The installed system transmitted, and kept transmitting, and I SSH'd into a native OpenBSD/arm64 box on AWS for the first time.

That is the entire pattern of this project in one paragraph: an exotic theory that flatters your understanding, sitting on top of a mundane mistake that flatters nothing. I had a second helping of it on the disk side too — the NVMe driver OpenBSD uses for the EBS volume itself wouldn't create its I/O queues until I clamped a maximum-queue-size field the AWS device reports in a way the stock driver didn't expect. Another one-line fix for a thing that looked, for an afternoon, like it might be deep.

So: disk install, done. SSH-in over ena0, done. The QEMU shim from the first post is, as of this writing, retired — I can build native OpenBSD/arm64 binaries on a real OpenBSD/arm64 instance now, the way the platform's own users would, on a cloud the platform supposedly cannot run on.

The scaffolding is gone too. The polling threads, the packet injector, the diagnostic printfs, the keep-alive timer for the imaginary problem — all stripped out, the real fixes separated from the detritus, the tree readable. It is not the same artifact I described last time as "full of scaffolding, none of it belonging in code anyone else should read." It's a driver now, not a demo with a driver inside it.

Faster than it had any right to be

The last post's most embarrassing admission was the testing: one DHCP round-trip is not throughput. So I went and got throughput, and then I went and got more of it than I expected.

The first honest number was 684 megabits per second — a single queue, a single CPU, real TCP over the real device, measured rather than imagined. That was already a strange feeling, watching a number that meant the data path was not just functional but fast enough to be useful, on a driver I'd been afraid to claim could pass a single packet a week earlier.

Then I built the parts a real NIC driver actually needs, the ones I'd listed as missing. Checksum offload, so the device computes and verifies IPv4/TCP/UDP checksums instead of the CPU — gated on the device advertising the feature, because not every ENA generation does. MTU control and jumbo frames. A watchdog that notices when the device has wedged and tears the data path down and brings it back up without a reboot. And then the big one: multiple queues.

A modern NIC isn't one ring of packets; it's many, one per CPU, so that traffic for different flows lands on different cores and the whole machine scales instead of bottlenecking on a single interrupt. Wiring that up on OpenBSD/arm64 meant allocating per-queue interrupts through the ARM interrupt controller, pinning each queue to its own CPU, and — the part that took longest to get right — receive-side scaling, RSS, where the device itself hashes each incoming packet by its flow and steers it to the correct queue so that packet order within a flow is preserved while load spreads across cores.

RSS is configured by handing the device a hash key and an indirection table: a little array that maps hash buckets to queues. I filled mine with what I was sure were the right queue identifiers, tested it, and got intermittent connectivity — about nine successful connections out of fifteen, which is the worst possible result, because it means you're close enough to be wrong in a way that looks almost right. The indirection table, it turned out, doesn't want the queue's completion-ring index, which is what I'd put there. It wants the queue's submission-ring index, a different number that happens to coincide on some devices and not others. Half my hash buckets were steering to a queue that wasn't listening. One field, the wrong index, surfacing only as a statistical haze of dropped connections. I changed it; twenty connections out of twenty succeeded, and the traffic spread cleanly across both cores.

On a two-vCPU t4g.medium — a deliberately small instance, because the point was to prove the mechanism, not to win a benchmark — the driver now moves 3.16 gigabits per second, balanced across both CPUs, with RSS steering flows to cores and each core fielding its own interrupts. That's not a line-rate claim on a big instance; it's a small machine using both of the hands it has. But it's a multi-queue, RSS-steered, checksum-offloaded network driver doing the actual job, and five weeks ago the honest claim was "one DHCP packet, from a kernel thread."

There was one more thing I wanted, less a feature than a verdict. I'd written and tested everything against OpenBSD 7.9, the release. But OpenBSD's real life happens on -current, the rolling development branch, and a driver that only works against one frozen release is a museum piece. So I built the whole thing against -current — and it compiled and ran with zero source changes, full feature parity, multi-queue and RSS and all. The interfaces I was building against had held. That mattered more to me than the throughput number, because it meant the driver was written against OpenBSD as it actually is, not against a single snapshot I'd reverse-engineered my way into.

By every measure I'd set for myself in that closing list, the driver was done. Which is precisely when I learned where it could not go.

The wall that isn't technical

Here is the thing I believed, quietly, the entire time I was writing this driver: that if I made it good enough — really good, idiomatic OpenBSD, clean bus_dma and honest locking and no vendor-HAL slop, the kind of code that earns its place — there was a path, however narrow, to it going upstream. To ena(4) becoming part of OpenBSD, so that the next person who tries to boot OpenBSD on Graviton doesn't have to write what I wrote. That was never the reason I did it — I did it because the gap was infuriating and the problem was beautiful — but it was the daydream underneath, the one that makes you clean up the scaffolding instead of leaving it.

The daydream is dead, and it died for a reason I didn't see coming and can't really argue with.

OpenBSD doesn't accept AI-generated code — not out of taste, but because code a model wrote has no human author, and with no author there's nothing to hold the copyright and nothing to license under the BSD/ISC terms the tree is built from. A provenance gate, not a quality one. Being good was never the question.

And this driver is AI-assisted in a way I want to be precise about, because it isn't how my other AI-assisted projects work. When I built Lattice, my programming language, or the ballistics engine that kicked off this whole Graviton saga, I was in the loop the entire time. The ideas were mine; I held the design in my head and used the model the way you use a sharp colleague — to think through, to draft against, to argue with. Authorship was never in question, because I was the one making the decisions.

The ena driver was not that. It was hands-off in a way I'd never tried: I pointed an agent at the problem, told it to run in a loop and build an ENA driver for OpenBSD, and let it go. It decided what the driver needed. It decided how to structure the attach path, the queues, the locking. It decided when to read Linux's driver for intent, when FreeBSD's, when NetBSD's, when Amazon's ena-com. I set the direction and the hard constraint — port from BSD-licensed sources, never copy the GPL Linux code — then read what came back and steered when it drifted. But I did not hold this driver in my head the way I held Lattice. For long stretches I was the reviewer of something being authored where I couldn't watch, by something making the decisions a driver's author makes.

And I should be blunter than I was last time. In the Doorbell post I wrote that I built the keep-alive timer and I implemented the host-attributes handshake — clean, correct code, I called it. The honest version is that an agent wrote both while I read along. I let the "I" stand because the project and the loop were mine, and they still are — but the decisions a driver's author makes were the agent's, and I'd rather say that plainly here than let the earlier "I" keep implying otherwise.

Which is why, here, the policy stops being abstract. With Lattice I can sign my name to every architectural choice and mean it. With this, the honest answer to "who decided that?" is, often, the agent did. Machine-authored, human-directed — and directed is not the same word as wrote. I don't think OpenBSD is wrong to refuse it. I think it might be the cleanest example I have of exactly the thing they're refusing.

So ena(4) stays mine. An independent driver, openly AI-assisted, for people who want native OpenBSD on Graviton badly enough to point their kernel config at a tree that isn't the official one. The work-in-progress repo, now considerably less work-in-progress, is at github.com/ajokela/openbsd-ena — open for reading, open for building, open for forking, and closed, by its own nature, to the one destination I'd quietly been building it toward.

There's a strange grief in that I didn't anticipate. Not for the work — the work is done and it runs. For the commons. The natural arc of a thing like this is that you give it away into the shared pool so the next person inherits it, and the reward for doing it well is that it stops being yours and becomes everyone's. This one can't take that arc. It's good enough to belong to everyone and it will belong to no one, because the question "who wrote this?" no longer has an answer the commons can accept. The driver works. The driver has no author. Both of those are true, and the second one is the price of the first.

The gap I couldn't close

If I'm going to be honest about the wall, I have to be honest about the one place the engineering didn't close either.

Everything above is true on Graviton2 — the t4g family, AWS's first ARM generation, the host-memory data path. On Graviton3 — the c7g family, which uses the low-latency LLQ path and a newer revision of the virtual ENA device — the driver attaches, configures, brings up its admin queue, reads every device attribute, and then fails at the very first step of creating an I/O queue. The device rejects the CREATE_CQ command — the request to make a completion ring — with an unhelpful status code, before a single packet has had the chance to flow.

I spent a genuinely unreasonable amount of effort on this. I did the thing systematic debugging tells you to do when a system has multiple components and you can't see inside one of them: I went and instrumented the component that works. I booted Amazon's own Linux on a c7g instance and traced its stock, vendor-blessed ENA driver with bpftrace — every admin command it sends, in order, with its exact arguments, from attach to first packet — so I'd have a known-good transcript to diff mine against.

The transcript refuted, one by one, every theory I had. The bytes of my CREATE_CQ command are identical to Linux's. The interrupt vector layout is identical — same number of MSI-X vectors, same assignment. The completion-descriptor size is identical. The order I create things in doesn't matter — I tried Linux's order exactly and the first queue still fails, whichever kind it is, which means it's not an ordering bug but a missing prerequisite the device wants before it will make any queue at all. I thought I'd found it in the host-info block, the little structure where the driver tells the device about itself — Linux fills in capability flags I was leaving zero — so I filled them in. No change. I thought it was the RSS hash configuration Linux sets up before its queues; I read the device's own feature bitmap and found that this device, like the Graviton2 one, doesn't even allow the host to set the hash function. Refuted by the device itself.

Nine times I baked a kernel and booted it on real c7g hardware to test a hypothesis, and nine times the device said no in the same flat way, and I never found the prerequisite it's waiting for. I have it narrowed to a short list of things Linux does before its first queue that I don't — a couple of feature reads I skip, a second host-info push after negotiation — but "narrowed to a short list, each refuted or untested" is not "solved," and I'm not going to dress it up as solved. The driver runs beautifully on the previous ARM generation and stops at the threshold of the current one, and I wrote down exactly where the threshold is so that whoever picks this up next — me, on a better day, or someone who forks the tree — starts from a map instead of a mystery.

That's the honest shape of it. A driver that's finished on the hardware it's finished on, and has a precisely-documented hole on the hardware it isn't.

The only OpenBSD in the cloud

I'll end where the practical and the strange meet.

At some point, late, I got curious about what I'd actually built relative to what already existed — whether I'd spent two months reinventing something I could have launched in one click. So I searched. Every public Amazon Machine Image, across multiple regions, with "OpenBSD" anywhere in the name or description.

There are none. Not one. Zero community images, zero in the AWS Marketplace, in every region I checked. The entire BSD presence in Amazon's marketplace is FreeBSD — dozens of official, well-maintained, free images published by the FreeBSD Foundation, every release and architecture, ARM and x86, the way a first-class cloud citizen looks. OpenBSD isn't a paid option anyone's gouging for. It isn't an option at all. The only way to run OpenBSD on EC2 today is to build the image yourself, which is to say: to first solve the exact problem these three posts have been about.

So I finished the job. I took the driver, baked it into a real OpenBSD/arm64 image, and made the image self-configuring the way every cloud image is expected to be — it pulls your SSH key, its hostname, and an optional first-boot setup script from the instance metadata service, creates an unprivileged login user, regenerates its own host keys so every instance has a unique identity. OpenBSD doesn't ship the Linux cloud-init and never will; the community answer is a slim shell agent written years ago by an OpenBSD developer, and I wired it in. Launch the image with your own key pair, and you SSH in as a normal user thirty seconds later. It is, as far as I can tell, the only launch-and-go OpenBSD/arm64 image on AWS, because it's the only OpenBSD/arm64 image on AWS.

Which leaves me holding a small, sharp irony. The most finished thing I've built in a while — a driver, a tuned data path, a clean image, a working cloud-init story, the genuinely-only-one-of-its-kind artifact — is the one that can't go where work like this is supposed to go. I could publish it as a public image tomorrow and be the sole OpenBSD on the entire platform. And I'd be publishing it the way I have to publish all of it: off to the side, in my own tree, under my own name, with a note that says an AI helped write this and so it can never be yours, only borrowed.

The first post in this series put a translator between two systems that disagreed, so each could be right without meeting. The second taught one system to speak the other's language directly, register by register, and found the bug that mattered was the most boring one in the file. This one finishes that language — makes it fluent, fast, multi-queue, native, bootable, launchable — and discovers that fluency was never the thing standing between the work and its home. The doorbell rings. The device lives. The network is seen, on a cloud that couldn't see it a season ago.

And the driver that makes it so has no author, and no home, and runs perfectly anyway.

The Doorbell That Killed the Device: Writing OpenBSD's Missing ena(4) Driver

A.C. Jokela — Mon, 22 Jun 2026 01:30:00 GMT

The Doorbell That Killed the Device: Writing OpenBSD's Missing ena(4) Driver

Earlier last week, I wrote about a problem with a clean, almost funny shape: OpenBSD's arm64 kernel has no driver for the network card AWS puts in every EC2 instance, so an OpenBSD/arm64 instance boots into a billable void it can never reach. The fix in that post was to stop asking — to run OpenBSD as a KVM guest inside QEMU on a bare-metal Graviton host, hand it virtio devices it already understands, and let a Linux host own the real Amazon adapter. Put a translator on the seam, and two systems that disagree about what a network card is can both be right.

That solved the build-server problem. It did not solve the itch.

Because the actual missing piece is small and specific and writable: the driver. Amazon's adapter is called the Elastic Network Adapter — ena — and it's a documented device with a permissively licensed reference implementation. FreeBSD has an ena driver. NetBSD has one. The protocol is published. OpenBSD just doesn't have the code, because effectively nobody runs OpenBSD as a first-class EC2 guest and so nobody wrote it. "Nobody wrote it" is not the same as "can't be written." So I decided to write it.

This is the story of that — and, more than the previous post, it's a story about a specific kind of failure. The driver attached on the first serious try. Then the device started killing itself, silently, a few microseconds after every bring-up, and I spent the better part of two days proposing increasingly clever reasons why. The real reason turned out to be embarrassingly dull. And because I want this to be useful and not just triumphant, the last third of the post is about the word "working": what I can actually claim, and the uncomfortable distance between that and "done."

What an ENA driver has to do

A modern NIC isn't a thing you poke registers at to send a packet. It's a small message-passing computer that shares host memory with you: rings of descriptors in DMA memory, doorbell registers to announce new work, an interrupt path for completions. ENA has three kinds of queue, and all three matter to this story.

The admin queue is how the host configures the device — a submission ring and a completion ring in shared memory. Write a command (read attributes, set a feature, create an IO queue), ring the admin doorbell, wait for a completion. OpenBSD's driver polls for that completion rather than taking an interrupt, which keeps bring-up simple.

The AENQ — Asynchronous Event Notification Queue — is the device's back-channel: it announces link up/down and, critically, posts a keep-alive event about once a second. The keep-alive is a heartbeat; the host is expected to drain these and, by draining them, prove it's still paying attention.

The IO queues are the actual network — a submission ring you fill with buffers, a completion ring the device writes back. You create them with admin commands (CREATE_CQ, then CREATE_SQ), and once they exist and the link is up, you can move traffic.

Get all three right, in the right order, and the card works. Get the order subtly wrong and — as I'd learn — the card decides you're not a real driver and quietly bricks itself.

The reference is Amazon's ena-com, a hardware-abstraction layer shared across the Linux and FreeBSD drivers. Its BSD-licensed parts — ena-com itself and FreeBSD's driver on top — are fair to read and port; the Linux driver is GPL, kept strictly read-only, a thing to consult for intent and never to copy. Writing the OpenBSD version means rewriting all of it in OpenBSD's idiom anyway — bus_dma(9) for the rings, pci(9) for attachment, ifnet/ifq for the stack. The protocol is the spec; the code is yours.

Phase zero: it attaches

The first milestone was just attachment, and it went well enough that I'd half-convinced myself the hard work was behind me. The driver resets the device, sets up the admin submission and completion rings, and issues commands. GET_FEATURE(DEVICE_ATTRIBUTES) comes back with the real MAC and maximum MTU — proof the admin queue works end to end, DMA is coherent, the device is listening. The console prints the line I'd been chasing: ena0 ... ENA ver 0.10 ... address 12:xx:xx:xx:xx:xx, on a real Graviton instance. The card was talking.

And then, every single time, a few seconds later, it stopped talking. The first attempt to create an IO queue — CREATE_CQ, the command that begins turning a configured device into a working network interface — would sit there and time out. No completion. No error. The admin queue that had just answered four commands flawlessly had gone silent.

When I finally added code to read the device's status register at each step, the shape of it came into focus and got worse. DEV_STS reads 0x1 — ready — through reset, through the admin handshake, through reading device attributes. Then, somewhere shortly after, it reads 0x21. Bit five is set. FATAL_ERROR. The device had, of its own accord, entered a fault state and was now refusing all further work. That's why CREATE_CQ vanished: you can't drive a device that's already decided it's dead.

A healthy card, healthy through every step I could see, that turns to stone the moment I look away. That's the wall.

Five wrong theories

Here is the part I'm telling on myself, because it's the honest center of the whole thing.

When you don't know why a device faults, the device gives you almost nothing — a single bit that says "something is wrong" and not a syllable about what. So you reason from the reference code: what does the working driver do that mine doesn't? And the trouble with that question, on hardware this unfamiliar, is that it has too many plausible answers. Every difference between my driver and ena-com looks like it could be the one that matters.

Theory one: the keep-alive watchdog. The device sends heartbeats; the host must drain them; if the host stops, surely the device fences off the unresponsive driver and faults. My driver had no periodic task draining the AENQ — it relied on a single interrupt that, I could see, fired exactly once and went quiet. This was a beautiful theory. It explained the timing. It matched a real mechanism FreeBSD implements. I built a proper one-second timer to drain the queue, mirroring FreeBSD's ena_timer_service, complete with a mutex to keep the timer and the interrupt from racing on a multi-core guest. It was clean, correct code. It fixed nothing. The device faulted on exactly the same schedule, and the diagnostic I'd added showed the AENQ had processed zero events — there was nothing to drain. I had carefully solved a problem that wasn't happening.

Theory two: the MMIO response region. ENA has a "readless" register mode backed by a small DMA region; maybe the device faulted without it. I added it. It didn't help — and worse, I'd added it before I started reading the status register, so for an embarrassingly long time I was carrying an unvalidated change that could have been the cause itself. (It wasn't.)

Theory three: host attributes. Both reference drivers register a 4 KB "host info" page right after admin init — SET_FEATURE(HOST_ATTR_CONFIG), a "yes, a real driver lives here" handshake. My driver skipped it. This had to be what the device validated before deciding I was legitimate. I implemented it properly. The status register read 0x1 right after it succeeded — and faulted anyway, at the same point as always.

Theory four: a stale completion. Maybe the admin queue was reading the wrong completion slot, so the feature data showing the device supporting zero AENQ event groups — a suspicious value — was garbage from an uninitialized ring. I instrumented the completion path down to the command IDs and phase bits. It was reading the right slot. The suspicious value was real.

Theory five: interrupt ordering. I'd unmasked the device's interrupt after a particular doorbell write; the reference does it before. I swapped the order. The device faulted one line later than before, which felt like progress and was not.

Somewhere in the middle I did what I increasingly do with a problem that has too many branches: I handed it to a fleet of AI agents, one per hypothesis, reading the reference trees in parallel, and had them synthesize a ranked root cause. The synthesis came back confident, specific, and wrong. What saved it was the same harness's adversarial step — three more agents told to refute the conclusion, and all three did, pointing out the timing didn't fit and the real anomaly was being hand-waved. The machine talked itself out of its own clever answer. That's the part worth remembering: not "the AI solved it," but "the AI proposed something plausible and the only thing that caught it was forcing a second pass that tried to tear it down."

What none of the five theories were was boring enough.

The doorbell

I gave up on theories and did the dumb, mechanical thing I should have done first. I made the driver poll the status register in a tight loop after every single register write in the bring-up, printing the exact moment the fault bit flipped. Not "is it healthy at the end" — which write kills it.

The answer came back in one line, and it was not ambiguous. Healthy after writing the queue's base address. Healthy after writing its size. Healthy after the feature commands. Then:

STS->FATAL 0x21 at [post-AENQ_HEAD_DB] iter=3

The fault appears about 150 microseconds after one specific write: the AENQ head doorbell — the register that tells the device "I've made the event ring's slots available; the queue is live." Not the interrupt unmask, which I'd swapped order on. Not any feature command. The doorbell that activates the event queue. Ring it, and the device dies.

The doorbell value was correct — I'd checked it against the reference a dozen times. The ring's address was correct. Its size was correct. So why would activating a correctly-configured queue kill a healthy device?

Because the queue wasn't really configured. It only looked like it was.

The most boring possible cause

I lined my driver's bring-up sequence up against ena-com's, write for write, and the difference was finally visible because I now knew exactly which write to care about.

ena-com registers the AENQ — writes its base address and size registers — inside ena_com_admin_init, in the same breath as the admin submission and completion queues, before the device is ever told initialization is finished. All three rings get registered together, as one atomic-feeling handshake, while the device is still in its "setting up" phase.

My driver registered the admin queues during admin init, exactly like the reference. But it registered the AENQ much later, in a separate function that ran after host attributes, after reading device attributes, after feature negotiation — long after the admin handshake had closed and the device considered itself up and running.

And here's the thing: writing those AENQ registers late worked, in the sense that the device accepted the writes and stayed healthy. The register values landed. The status bit stayed green. Everything looked fine. The device had quietly noted the address and size of a ring it had never actually wired into its event subsystem, because that wiring only happens during the init handshake I'd already finished. The AENQ was a ghost: registered on paper, uninitialized in the device's mind.

Then I rang the doorbell. "The event queue is live; start using it." The device went to use a subsystem that was never set up, and faulted. A hundred and fifty microseconds later, the bit flipped.

The fix is four lines moved earlier. I pulled the AENQ registration out of its late function and into admin init, immediately after the admin queues, exactly where ena-com does it. The later function kept only the parts that genuinely belong late — subscribing to event groups, and the final doorbell-and-unmask that says "go."

I rebuilt, booted, and watched the status register stay 0x1 straight through the doorbell. CREATE_CQ succeeded. CREATE_SQ succeeded. The link came up. The driver enqueued a hand-built DHCP DISCOVER, the device transmitted it, and a 590-byte IPv4 packet — the DHCP OFFER, a real reply from AWS's network — came back up the receive ring. Transmit and receive, on real silicon, for the first time.

There was one more gift in the logs, the kind that tells you a fix is right and not just lucky. Remember theory four — the device reporting zero supported AENQ event groups, the value I'd half-dismissed as a possible misread? With the AENQ now registered during init, that same query on the same hardware came back reporting all the groups supported. The zero had never been a VF limitation or a misread. It was the device telling me, in the one channel it had, that its event subsystem wasn't initialized — because I hadn't initialized it yet. One root cause had been wearing five costumes. The watchdog that had nothing to drain, the missing handshake, the suspicious zero, the doorbell fault — all of it was the single fact that I'd set up a queue in the wrong order, refracted through a device that can only ever tell you "something is wrong."

This is the same lesson the last post ended on, and I clearly didn't learn it hard enough the first time. Out here, far off the beaten path, every failure presents as exotic, because the strange explanation announces itself and the boring one doesn't. An out-of-disk error wears the costume of a dependency that won't compile. A queue initialized in the wrong order wears the costume of a keep-alive watchdog, a missing security handshake, and a device that lies about its capabilities. The further out you are, the more deliberately you have to rule out the dull thing first — and "I did the steps in the wrong order" is about as dull as it gets.

The loop that made it bearable

I have to mention the iteration speed, because for most of this saga it was the actual bottleneck, and fixing it is what turned a slog into something tractable.

The previous post's build server bakes a disk image, snapshots it, registers an AMI, boots a real EC2 instance, and reads the serial console — about eighteen minutes a turn. That's fine for building a binary. It is agony for debugging a driver, where you want to change one line and see what the hardware does. Eighteen minutes times the number of wrong theories above is a number I'd rather not compute.

So I built a faster loop, and it leans on the same bare-metal host the build server already needs. That a1.metal host has a real ARM SMMU — the IOMMU that makes device passthrough safe — and Linux's VFIO framework can hand a physical PCI device straight to a QEMU guest. So I attached a second network interface to the metal instance, bound it to vfio-pci on the Linux host (leaving the primary NIC alone, so I didn't saw off the SSH branch I was sitting on), and passed it through to the OpenBSD guest. Now the OpenBSD VM sees a real ENA device — actual Amazon silicon, vendor 1d0f, product ec20 — on its virtual PCI bus, and my driver attaches to that. No AMI bake. Build the kernel, reboot the guest, watch ena0 come up against real hardware. Two minutes a turn instead of eighteen.

There was one gotcha worth writing down, because the error is opaque: VFIO refused with failed to set iommu for container: Operation not permitted. ARM's SMMU here can't remap interrupts, so the kernel blocks passthrough by default — fixed with the module parameter allow_unsafe_interrupts=1, entirely fine for a trusted device on a machine I own by the hour.

That loop is also the honest reason the fix arrived when it did. The "poll after every write" instrumentation only became practical once I could run it and read it in two minutes. The clever theories flourished in the eighteen-minute dark; the boring method won the moment I could see in real time.

What "working" means

Here is where I have to be careful, because "I wrote a working ENA driver for OpenBSD" is a sentence that can mean five very different things, and only one of them is true today.

What is true: on a real EC2 Graviton2 instance, an OpenBSD/arm64 kernel with my driver attaches to the real ENA adapter, completes the full bring-up — admin queue, host attributes, device attributes, AENQ, IO queue creation — brings the link up, transmits a packet the device really puts on the wire, and receives the reply. The send and receive paths through the driver are the real ones: the transmit path is the same ifq enqueue the network stack uses, and the receive path is the same completion handler that would feed packets up to IP. A DHCP DISCOVER went out and an OFFER came back. The device protocol — the genuinely hard, genuinely undocumented-in-OpenBSD-idiom part — works, and it works on the production hardware, not just the passthrough rig.

First boot. OpenBSD/arm64 on a real Graviton2 EC2 instance, ena(4) attached, link up, DHCP round-trip completed against the real device.

What is not yet true, and I want to be exact about each one:

I drove that DHCP exchange from a kernel thread, not from userland. The test harness builds a DISCOVER packet in the kernel, hands it to the transmit ring, and watches the rings directly, because the minimal RAM-disk environment I'm booting doesn't have room for a real dhclient. The packets and the ring mechanics are real; the thing wrapping them is a debug scaffold, not ifconfig ena0 up; dhclient ena0; ping. I have not yet typed those three commands at a shell and watched them work. That's the next milestone, and until I've done it I won't claim the interface works "from userland," only that the driver's data paths do.

I am booting the OpenBSD install ramdisk, bsd.rd, which runs entirely from RAM. I have not done a full disk install and booted a persistent OpenBSD that comes up multiuser with ena0 as its only network interface and lets me SSH in over it. That — a normal OpenBSD instance you log into over the network it sees natively — is the milestone that would let me retire the QEMU-shim build server from the last post. I'm not there. I've proven the hard part is possible; I haven't assembled it into a system you'd actually run.

The driver is full of scaffolding. The status-register polling, the packet-injection thread, a dozen diagnostic printfs, the keep-alive timer I built for a problem that didn't exist — all still in the tree, behind a debug flag. None of it belongs in code anyone else should read. Before this is a contribution rather than a demo, that all comes out, the real fixes get separated from the detritus, and the whole thing gets the kind of review OpenBSD's tree rightly demands. It has had none of that. No OpenBSD developer has looked at a line of it. It is not submitted, not reviewed, and would not survive tech@ in its current state, nor should it. The work-in-progress tree, scaffolding and all, lives at github.com/ajokela/openbsd-ena — open for reading, not for trusting.

And the testing is thin. One DHCP round-trip is not throughput, not stability under load, not days of uptime, not the dozen edge cases — checksum offload, multi-queue, MTU changes, link flaps — a NIC driver has to handle before anyone trusts it. I've shown the path is real. I have not shown it's robust.

So: working in the sense that the central, doubted, genuinely difficult thing — does the device protocol function, correctly implemented from scratch, on real hardware — is now answered yes. Not working in the sense of something you'd deploy, or even the sense of something you'd ifconfig by hand yet. Both halves of that sentence are true and I don't want the exciting half to eat the honest one.

What it was about, again

The last post put a translator between two systems that disagreed, so each could be right without meeting. This one is the opposite move: no shim, just teaching one system to speak the other's language directly — a driver doing the actual work of turning OpenBSD's idea of a network interface into ENA's, register by register and ring by ring.

But the deeper rhyme isn't the architecture, it's the failure. Both times the headline problem had a one-sentence answer — "run it as a guest," "register the queue during init" — and both times that sentence was the easy part, with the real work in a gap where everything looked exotic and the truth was mundane. And both times the trap was the same: a plausible, faintly flattering, wrong explanation is far more available than the boring one underneath it — especially with a tireless machine happy to generate plausible explanations on demand. The machine is genuinely useful; it read three reference drivers in parallel and caught its own bad guess on the second pass. But it has no instinct for "you probably just did the steps out of order," because it has never spent an afternoon being humiliated by exactly that.

What I have now is a driver that makes OpenBSD see the network on a cloud that, a month ago, OpenBSD couldn't see at all. It is not done. It is, for the first time, possible — proven on the hardware, by the most boring fix in the file. The doorbell rings, and the device lives.

JokelaOS: Writing a Bare-Metal x86 Kernel from Scratch

A.C. Jokela — Tue, 10 Mar 2026 15:00:00 GMT

There's a moment early in any OS project where the serial port prints its first character and you realize that nothing you've written has a safety net. No libc. No kernel underneath. No syscall to fall back on. If the byte appears on the terminal, it's because you programmed the UART divisor latch, polled the line status register, and wrote to the data port. If it doesn't appear, you stare at register dumps until you find the mistake. There's no debugger; you haven't written one yet.

The closest thing I can compare it to is the first time I got a RetroShield Z80 talking over serial, that moment where a processor you wired up yourself pushes a character out of an emulated ACIA and it appears on your screen. The Z80 version involves physical hardware and solder. The x86 version is virtual (QEMU, a cross-compiler, and a Multiboot header), but the feeling is the same. You built the entire path from CPU to character. Nothing was given to you.

JokelaOS started there: a Multiboot header, a stack, and a call kmain. Everything that followed (GDT (Global Descriptor Table), IDT (Interrupt Descriptor Table), memory management, a network stack, preemptive multitasking, paging, user mode, a shell) was built one subsystem at a time, tested after every change, with no external code. No forks of existing kernels. No libc.

To be clear about what this is: JokelaOS is a toy. It's a learning project. The memory allocator is a linear scan. The scheduler has no concept of priority. The file system can't delete files. The user authentication stores passwords in plaintext in a static array. Nothing here is production-grade, and none of it is intended to be. The value is in the building: understanding what each subsystem actually does by writing it from scratch, making the mistakes, and fixing them with nothing between you and the hardware.

This is the story of what it takes to go from twenty lines of NASM to a kernel that boots, manages memory, runs user programs in Ring 3, handles syscalls, responds to pings, and gives you a command prompt.

The Target

JokelaOS targets 32-bit x86 (i686) and runs under QEMU. The toolchain is a cross-compiler (i686-elf-gcc, i686-elf-ld) with NASM for the assembly files. The C standard is gnu11; GNU extensions are required for inline assembly. There are no external libraries whatsoever, not even a freestanding string.h. Every memcpy, every memset, every printf-like function is written from scratch.

The only console is the serial port. COM1 at 0x3F8, 115200 baud, 8N1 (8 data bits, no parity, 1 stop bit). All kernel output goes through serial_printf(). This is a deliberate choice: serial is simpler than VGA text mode, works perfectly with QEMU's -serial stdio, and means the kernel's output appears directly in the host terminal. No framebuffer driver needed, no font rendering, no cursor management. Just bytes on a wire.

$ make run
qemu-system-i386 -kernel build/jokelaos.bin -serial stdio \
    -display none -device rtl8139,netdev=net0 \
    -netdev user,id=net0 -no-reboot

Kernel Architecture

JokelaOS is monolithic: everything runs in Ring 0, in one address space. When the network stack needs a page, it calls pmm_alloc_frame() directly. When the shell loads a program, the call chain goes through the loader, the PMM, and the paging subsystem without ever crossing an address space boundary. The trade-off is that a bug in the RTL8139 driver can corrupt the process table, and a buffer overrun in the serial handler can overwrite page tables. In a toy kernel written by one person, bugs are spectacular.

A microkernel would isolate those failures, but it would also triple the code before you could print a single character. You'd need working IPC before the serial driver could talk to anything. JokelaOS is monolithic because it's the simplest architecture to build and the easiest to debug: serial_printf() anywhere can see everything.

Booting: The First 33 Lines

The entire boot sequence fits in boot.asm. Multiboot v1 requires a magic number (0x1BADB002), flags, and a checksum in a specific header format. GRUB or QEMU's -kernel loader scans for this header, loads the binary, and jumps to _start in protected mode with paging disabled.

section .multiboot
align 4
    dd 0x1BADB002                           ; Multiboot magic
    dd 0x00000003                           ; Flags: page-align + memory map
    dd -(0x1BADB002 + 0x00000003)           ; Checksum

section .text
global _start
extern kmain

_start:
    mov esp, stack_top
    push 0
    popf                    ; Clear EFLAGS
    push ebx                ; Multiboot info struct pointer
    push eax                ; Multiboot magic number
    call kmain
    cli
.hang:
    hlt
    jmp .hang

That's it. Set up a stack, clear the flags register, push the two values the Multiboot spec guarantees (magic number in EAX, info struct pointer in EBX), and call C. If kmain ever returns, disable interrupts and halt forever.

The 16 KB stack is allocated in the BSS section, zeroed at load time. The linker script places the kernel at 1 MB (the standard x86 protected-mode load address), with .multiboot first so the bootloader can find the header within the first 8 KB of the binary.

Protection Rings: Hardware-Enforced Privilege

x86 protected mode provides four privilege levels, numbered 0 through 3, called rings. Ring 0 is the most privileged: the kernel runs here. Ring 3 is the least privileged: user programs run here. Rings 1 and 2 exist in the hardware but almost nobody uses them. Linux doesn't. Windows doesn't. JokelaOS doesn't. The practical x86 privilege model is two rings: kernel and user.

The ring system isn't a software convention. It's enforced by the CPU itself, in silicon. The processor tracks the Current Privilege Level (CPL), the ring the currently executing code belongs to, and checks it against every sensitive operation. A Ring 3 process that executes cli (disable interrupts), hlt (halt the CPU), lgdt (load a new GDT), or mov cr3 (change the page directory) triggers a General Protection Fault. The CPU literally refuses to execute the instruction. A Ring 3 process can't touch I/O ports unless the kernel has explicitly granted access through the I/O Permission Bitmap in the TSS. It can't modify its own segment registers to escalate privilege, because the CPU validates every segment load against the descriptor's DPL (Descriptor Privilege Level).

The only way for Ring 3 code to enter Ring 0 is through a gate: an interrupt gate, a trap gate, or a call gate. Gates are entries in the IDT or GDT that the kernel sets up in advance. They define the exact entry points where Ring 3 code can cross into Ring 0, what the new code and stack segments will be, and what privilege level is required to use them. There's no way for user code to jump to an arbitrary kernel address. It can only enter the kernel through the doors the kernel has built.

This is what makes an operating system an operating system rather than a library. Without ring separation, a buggy user program can corrupt kernel memory, disable interrupts, reprogram the PIC, or overwrite the page tables. With ring separation, the worst it can do is crash itself.

The mechanism that implements all of this is the Global Descriptor Table.

The GDT: Defining the World

The GDT defines memory segments: their base addresses, sizes, privilege levels, and whether they hold code or data. Each segment descriptor is an 8-byte structure with fields packed into non-obvious bit positions (a consequence of backward compatibility with the 286, which had a different descriptor format that the 386 had to extend without breaking).

JokelaOS uses a flat memory model: every segment covers the full 4 GB address space with base 0 and limit 0xFFFFFFFF. The segmentation hardware is effectively nullified, which is what you want on modern x86 where paging handles memory protection. But the GDT is still mandatory; the CPU requires it for the ring system to function. Even with flat segments, the DPL field in each descriptor is what tells the CPU "code using this segment is Ring 0" or "code using this segment is Ring 3."

The GDT has six entries:

Index	Selector	Purpose
0	0x00	Null descriptor (required by x86)
1	0x08	Kernel code (Ring 0, execute/read)
2	0x10	Kernel data (Ring 0, read/write)
3	0x18	User code (Ring 3, execute/read)
4	0x20	User data (Ring 3, read/write)
5	0x28	Task State Segment

Entries 1 and 2 are identical to entries 3 and 4 in every way except the DPL field: two bits in the access byte that say 00 (Ring 0) versus 11 (Ring 3). That two-bit difference is the entire kernel/user boundary.

When a user process runs, the CPU's CS register is loaded with 0x1B; that's selector 0x18 (pointing to GDT entry 3, the user code segment) OR'd with RPL 3 (the bottom two bits of the selector). The data segment registers get 0x23 (GDT entry 4, user data, RPL 3). The CPU sets CPL to match, and from that point on, every instruction is checked against Ring 3 privileges. The kernel runs with CS=0x08 (GDT entry 1, RPL 0) and DS=0x10 (GDT entry 2, RPL 0).

The TSS (Task State Segment) is the bridge between rings. When the CPU takes an interrupt while running Ring 3 code, it needs to switch to a Ring 0 stack, because you can't trust the user's stack pointer to be valid, and you certainly can't run kernel interrupt handlers on a user-controlled stack. The TSS holds the Ring 0 stack pointer (esp0). Every context switch updates the TSS with the current process's kernel stack, so the CPU always knows where to land when transitioning from user mode to kernel mode.

void gdt_init(void) {
    gdt_set_entry(0, 0, 0, 0, 0);                     // Null
    gdt_set_entry(1, 0, 0xFFFFFFFF, 0x9A, 0xCF);      // Kernel code
    gdt_set_entry(2, 0, 0xFFFFFFFF, 0x92, 0xCF);      // Kernel data
    gdt_set_entry(3, 0, 0xFFFFFFFF, 0xFA, 0xCF);      // User code
    gdt_set_entry(4, 0, 0xFFFFFFFF, 0xF2, 0xCF);      // User data
    // TSS entry built separately
    ...
}

The access byte 0x9A means: present, Ring 0, code segment, executable, readable. 0xFA means the same thing but Ring 3. These magic numbers come straight from the Intel manuals and they're the kind of thing you get wrong three times before you get right once.

Interrupts: Exceptions, IRQs, and the PIC

The IDT maps interrupt vectors to handler functions. JokelaOS sets up 256 entries: CPU exceptions (0-31), hardware IRQs (32-47), and the syscall gate (0x80).

The x86 PIC needs remapping. By default, the master PIC maps IRQs 0-7 to interrupt vectors 8-15, which collide with CPU exceptions (double fault is vector 8, for instance). The standard fix is to remap the master PIC to vectors 32-39 and the slave to 40-47. This requires sending four Initialization Command Words to each PIC in the correct sequence, the kind of hardware protocol that hasn't changed since the IBM PC/AT in 1984.

ISR stubs are written in NASM. Each one pushes an error code (or a dummy zero for exceptions that don't push one), pushes the interrupt number, saves all general-purpose registers, calls the C handler, restores registers, and does an iret. The stubs are generated with macros:

%macro ISR_NOERRCODE 1
global isr%1
isr%1:
    push dword 0        ; dummy error code
    push dword %1       ; interrupt number
    jmp isr_common
%endmacro

The C-side dispatcher checks the interrupt number. For exceptions (0-31), it prints the register state and halts, since there's no recovery from a page fault when you don't have a page fault handler yet. For IRQs (32-47), it calls the registered handler function and sends an EOI command to the PIC. For interrupt 0x80, it dispatches to the syscall handler.

One critical detail: interrupt 0x80 is set as a trap gate with DPL 3, not an interrupt gate. This means Ring 3 code can trigger it with int 0x80. All other interrupt gates are DPL 0, so a user program that tries to execute int 0x00 gets a General Protection Fault instead. This is the mechanism that makes syscalls work while keeping everything else protected.

Memory: Three Allocators

JokelaOS has three layers of memory management, each built on top of the previous one.

The Bump Allocator

The simplest possible allocator. A pointer starts at the first page boundary after the kernel image (_kernel_end from the linker script) and only moves forward. kmalloc(size) aligns the pointer to 16 bytes, returns it, and advances by size. There is no kfree(). Memory allocated with the bump allocator is permanent.

This sounds primitive, and it is. But it's also exactly right for kernel initialization. The GDT, IDT, page tables, file system metadata, user table; these are allocated once and never freed. The bump allocator handles all of them with zero fragmentation and zero overhead.

The Physical Memory Manager

Once the kernel needs to allocate and free pages dynamically (for process stacks, program code, page tables), it needs a real allocator. The PMM uses a bitmap: one bit per 4 KB physical frame, supporting up to 256 MB of RAM (65,536 frames, 8 KB bitmap).

Initialization parses the Multiboot memory map to find usable RAM regions, then marks everything from frame 0 through the end of the bump heap as reserved. This protects the IVT, BIOS data area, kernel image, and all bump-allocated structures from being handed out as free pages.

uint32_t pmm_alloc_frame(void) {
    for (uint32_t i = 0; i < total_frames; i++) {
        if (!(bitmap[i / 8] & (1 << (i % 8)))) {
            bitmap[i / 8] |= (1 << (i % 8));
            free_count--;
            return i * PAGE_SIZE;
        }
    }
    return 0;  // out of memory
}

Linear scan, no free lists, no buddy system. It's O(n) per allocation, which is fine when n is measured in thousands and allocations are infrequent. A production kernel would use something smarter. This kernel allocates a few dozen pages total.

Paging

With physical frames available, the kernel can enable paging. paging_init() builds a page directory and 32 page tables, identity-mapping the first 128 MB of physical memory (virtual address = physical address). The page directory goes into CR3, and setting the PG bit in CR0 turns the MMU on.

Identity mapping means the kernel doesn't need to worry about virtual-to-physical translation for its own code and data. Kernel pointers just work. When user processes need memory, the loader allocates physical frames and maps them into the process's address space with the PG_USER flag set, allowing Ring 3 access.

void paging_map_page(uint32_t virt, uint32_t phys, uint32_t flags) {
    uint32_t dir_idx = virt >> 22;
    uint32_t tbl_idx = (virt >> 12) & 0x3FF;

    if (!(page_directory[dir_idx] & PG_PRESENT)) {
        uint32_t tbl_frame = pmm_alloc_frame();
        memset((void *)tbl_frame, 0, PAGE_SIZE);
        page_directory[dir_idx] = tbl_frame | PG_PRESENT | PG_WRITE | flags;
    }

    uint32_t *table = (uint32_t *)(page_directory[dir_idx] & 0xFFFFF000);
    table[tbl_idx] = (phys & 0xFFFFF000) | flags;

    asm volatile("invlpg (%0)" : : "r"(virt) : "memory");
}

The invlpg instruction flushes the TLB entry for the mapped virtual address, which is critical. Without it, the CPU might use a stale translation from its cache and access the wrong physical page.

The Network Stack

JokelaOS has a working network stack, the one subsystem where "toy" undersells it slightly. It resolves ARP, constructs IPv4 packets with correct checksums, and handles ICMP echo request/reply with measured round-trip times. There's no TCP, no UDP, no sockets. But the packets that leave this kernel are real packets that traverse real networks.

The NIC is an emulated RTL8139, the simplest PCI Ethernet controller that QEMU supports. The driver initializes the chip by writing to its configuration registers: reset, enable transmitter and receiver, set up a receive ring buffer, configure the interrupt mask, and unmask IRQ 11. Packet transmission uses a four-descriptor TX ring; reception is interrupt-driven through the RTL8139's ring buffer.

PCI enumeration scans the configuration space to find the RTL8139 by vendor/device ID (0x10EC:0x8139), reads the I/O base address from BAR0, and enables bus mastering. This is the only driver in the system; there's no USB, no disk, no display. One NIC, one network.

The stack is layered:

Layer	Module	Purpose
Link	`ethernet.c`	Frame demux by EtherType
ARP	`arp.c`	Table + request/reply
Network	`ipv4.c`	Routing, header checksum
Transport	`icmp.c`	Echo reply + outgoing ping

On boot, the kernel sends an ARP request for the gateway (10.0.2.2, QEMU's default) and waits for the reply. Once the gateway's MAC address is resolved, the kernel can ping arbitrary hosts through QEMU's SLIRP NAT. A ping 10.1.1.1 from the shell constructs an ICMP echo request, wraps it in an IPv4 packet, wraps that in an Ethernet frame, and pushes it out through the RTL8139's TX ring. When the reply comes back, the receive ISR fires, the Ethernet layer demuxes by EtherType, the IP layer validates the checksum, and the ICMP handler matches the echo reply to the outstanding request and computes the RTT.

root$ ping 10.1.1.1
[ping] Pinging 10.1.1.1...
  64 bytes from 10.1.1.1: seq=1 time=4 ms
  64 bytes from 10.1.1.1: seq=2 time=2 ms
  64 bytes from 10.1.1.1: seq=3 time=3 ms
  64 bytes from 10.1.1.1: seq=4 time=2 ms

Getting here required writing every byte-order conversion (htons, htonl), every checksum computation (the IP header checksum is a one's complement sum of 16-bit words), every packet layout (Ethernet header is 14 bytes, IP header is 20, ICMP is 8 plus payload). None of this is hard individually. Together, it's a thousand places to put a byte in the wrong order.

Processes and Preemptive Multitasking

The process subsystem manages up to 16 processes in a static table. Each process has a state (UNUSED, READY, RUNNING, DEAD), a kernel stack pointer, and a user-mode entry point and stack.

Process creation doesn't follow the UNIX fork()/exec() model. There's no cloning of address spaces, no copy-on-write, no replacing the current process image. Instead, the loader allocates fresh physical frames for the program's code and stack, copies the flat binary into the code pages, and calls proc_create(), which allocates a 4 KB kernel stack and builds a fake stack frame on it. This stack frame is what context_switch() will "return" into on the process's first schedule; it contains saved registers and a return address pointing to proc_entry_user().

proc_entry_user() is a small assembly sequence that performs the Ring 0 to Ring 3 transition. It sets the data segment registers to the user data selector (0x23), pushes a fake interrupt frame (SS, ESP, EFLAGS with IF=1, CS, EIP), and executes iret. The CPU pops the frame, switches to Ring 3, and starts executing the user program. From the hardware's perspective, this looks identical to returning from an interrupt that happened to interrupt a user-mode program, which is exactly the trick.

static void proc_entry_user(void) {
    process_t *p = proc_current();
    asm volatile(
        "mov $0x23, %%ax \n"
        "mov %%ax, %%ds  \n"
        "mov %%ax, %%es  \n"
        "mov %%ax, %%fs  \n"
        "mov %%ax, %%gs  \n"
        "push $0x23      \n"   // SS
        "push %0         \n"   // ESP
        "pushf           \n"
        "pop %%eax       \n"
        "or $0x200, %%eax\n"   // Set IF
        "push %%eax      \n"   // EFLAGS
        "push $0x1B      \n"   // CS (user code)
        "push %1         \n"   // EIP
        "iret"
        : : "r"(p->user_esp), "r"(p->user_eip)
        : "eax", "memory"
    );
}

Context switching uses a simple assembly stub in switch.asm. It saves the callee-saved registers (EBP, EBX, ESI, EDI), stores ESP into the old process's slot, loads the new process's ESP, restores registers, and returns. The ret instruction pops the return address from the new stack and resumes where that process left off.

Scheduling is preemptive round-robin. The PIT fires at 1000 Hz. Every 10 ticks (10 ms), the IRQ handler calls proc_schedule(), which finds the next READY process and switches to it. If no user processes are ready, control stays with PID 0 (the kernel/shell). This is the minimum viable scheduler: no priorities, no time slices, no fairness guarantees. But it works: two user programs printing characters to serial run concurrently, interleaved by the timer.

Syscalls

User programs communicate with the kernel through int 0x80. The mechanism, a software interrupt that transitions from Ring 3 to Ring 0, is the same one Linux used on i386 before sysenter replaced it. The register convention is borrowed too: syscall number in EAX, arguments in EBX/ECX/EDX/ESI/EDI, return value in EAX. But that's where the resemblance ends.

JokelaOS is not a UNIX. The syscall numbers are custom (exit is 0, write is 1, getpid is 2, read is 3), not Linux's i386 table (where exit is 1, read is 3, write is 4, getpid is 20). There's no fork(), no exec(), no open(), no close(), no signals, no pipes. File descriptors 0 and 1 exist as concepts (stdin maps to the keyboard buffer, stdout maps to the serial port) but there's no file descriptor table behind them. The syscall handler just checks if (fd == 1) and calls serial_putchar(). The process model isn't UNIX either; there's no parent/child relationship, no wait(), no process groups. Processes are created by the loader and scheduled round-robin until they exit. It's closer to a microcontroller RTOS than to anything in the UNIX lineage.

Four syscalls are implemented:

Number	Name	Arguments	Description
0	SYS_EXIT	ebx=status	Terminate process
1	SYS_WRITE	ebx=fd, ecx=buf, edx=len	Write to serial (fd=1)
2	SYS_GETPID	-	Return current PID
3	SYS_READ	ebx=fd, ecx=buf, edx=len	Read from keyboard (fd=0)

This is enough to write programs that print output, read input, identify themselves, and exit cleanly. The syscall dispatcher validates file descriptors (only 0 and 1 are legal) and bounds-checks lengths. SYS_WRITE sends bytes to the serial port; SYS_READ drains the keyboard buffer non-blocking.

User programs are flat binaries: raw machine code with no headers, no relocations, no ELF parsing. The loader copies the binary to freshly allocated pages and jumps to byte zero. Programs that need to reference their own data use position-independent tricks:

call next       ; push EIP
next:
pop ebp         ; EBP = address of this instruction
lea ecx, [ebp + offset_to_data]

This is the same technique used by shellcode and position-independent code on x86. It works because call pushes the address of the next instruction, which gives you a known reference point relative to the code's actual load address.

The Shell

With all the subsystems in place, the shell ties them together into something interactive. shell_run() is the kernel's main loop after initialization. It presents a login prompt, authenticates against the user table, and drops into a command interpreter.

==============================
  JokelaOS v0.1
==============================

[ok] GDT loaded (ring 0 + ring 3 + TSS)
[ok] IDT loaded, PIC remapped
[ok] Multiboot confirmed
[ok] Multiboot info at 0x9500
[ok] Bump allocator ready

PCI 00:03.0 - vendor 10EC device 8139 (RTL8139)
[ok] RTL8139 ready, MAC=52:54:00:12:34:56, IRQ=11

[ok] ramfs: 2 files
[ok] Users: root, guest
[ok] PMM: 31269 free frames (122 MB)
[ok] Paging enabled (128 MB identity-mapped)
[ok] PIT timer @ 1000 Hz
[ok] Keyboard + serial input ready

JokelaOS is alive.

login: root
password: ****
root$

The shell supports: help, ls, run <program>, ps, mem, ping <ip>, uptime, whoami, and logout. The line editor handles backspace. Password input echoes asterisks. The run command loads a flat binary from ramfs, creates a process, and the scheduler picks it up on the next timer tick.

ps shows the process table:

root$ ps
  PID  STATE
    0  RUNNING
    1  READY
    2  DEAD

mem shows memory usage:

root$ mem
Heap used: 8832 bytes
PMM free:  31267 frames (122 MB)

The keyboard input path is worth noting. The PS/2 keyboard controller fires IRQ 1. The handler reads the scancode from port 0x60, converts it to ASCII using a US QWERTY lookup table (with shift modifier tracking), and drops it into a 256-byte circular buffer. Serial input takes the same path; the UART's receive interrupt (IRQ 4) reads the incoming byte and injects it into the keyboard buffer. This means the shell works identically whether you're typing on a PS/2 keyboard or through the QEMU serial console.

The RAM File System

User programs need to live somewhere. With no disk driver, the file system is purely in-memory. ramfs stores up to 32 files, each with a name (28 bytes), a data pointer, and a size. ramfs_create() allocates space with the bump allocator and copies the binary in. ramfs_find() does a linear search by name.

During boot, two test programs are embedded directly in kmain.c as byte arrays of hand-assembled x86 machine code. One prints the character '1' ten times; the other prints '2' ten times. Both use SYS_WRITE to output through the serial port and SYS_EXIT to terminate cleanly. They're loaded into ramfs, and run print1 from the shell executes them in user mode.

This is about as minimal as a file system gets. No directories, no permissions, no deletion. But it demonstrates the complete path from "bytes in kernel memory" to "user-mode process executing with its own address space."

What I Learned

The boot process is the hardest part. Not because the code is complex (boot.asm is 33 lines), but because when something goes wrong, you have zero diagnostic capability. The serial port isn't initialized yet. The IDT isn't loaded. If your Multiboot header checksum is wrong by one bit, QEMU silently fails. You're debugging with QEMU's -d int flag and reading hex dumps of interrupt frames.

x86 protected mode is an archaeology project. The PIC remapping sequence dates from the IBM PC/AT (1984). The GDT access bytes encode information in bit patterns designed for hardware that predates flat memory models. The TSS exists because Intel's original vision for the 286 involved hardware task switching that nobody ended up using. You're programming against forty years of backward compatibility, and every one of those layers is still there, still mandatory, still silently breaking things if you get it wrong.

The gap between "works in Ring 0" and "works in Ring 3" is enormous. A kernel that runs entirely in supervisor mode can be surprisingly simple. The moment you add user mode, you need: the TSS (so the CPU knows where the kernel stack is), Ring 3 GDT segments, trap gates for syscalls, a mechanism to build fake interrupt frames for the initial iret into user mode, and careful validation of every pointer that crosses the kernel boundary. Each of these is individually straightforward. Getting them all correct simultaneously is not.

Preemptive scheduling is simpler than it sounds. The concept (save state, pick next process, restore state) translates almost directly into code. The context switch is twelve instructions of assembly. The scheduler is a for loop. What makes it tricky is the interaction with everything else: the TSS must be updated, the interrupt must send EOI before switching, the process's kernel stack must be set up so that restoring registers and returning lands in the right place. The scheduler itself is trivial. The invariants it depends on are not.

Writing a network stack is an exercise in byte ordering. Ethernet is big-endian. x86 is little-endian. IP addresses, port numbers, checksums, packet lengths: every multi-byte field requires explicit conversion. Miss one htons() and your packets are valid-looking garbage. The RTL8139 driver, the ARP implementation, the IP checksum; each is maybe fifty lines. The debugging when a byte is swapped is hours.

The Numbers

JokelaOS in its current form:

Component	Files	Approximate LOC
Boot (ASM)	3	~120
Kernel core	16	~1,200
Drivers	2	~250
Network stack	5	~450
Total	26	~2,000

Two thousand lines for a kernel that boots, manages memory with paging, runs preemptive multitasking with Ring 3 isolation, handles interrupts, implements syscalls, has a working network stack, and provides an interactive shell. No line is borrowed from another project. Every byte is accounted for.

The entire thing builds in under a second and the binary is around 40 KB. make run goes from source to a running kernel in QEMU in about two seconds. This fast iteration cycle is what made the project possible; every subsystem was tested immediately after being written, and bugs were caught before they could compound.

What's Next

The point of JokelaOS was never to build a production operating system. The point was to understand what an operating system actually does: not in the abstract, not from a textbook diagram, but in the specific, concrete sense of "these bytes go into these ports in this order and then the hardware does this thing." Every subsystem in JokelaOS exists because I wanted to understand it, and the only way to truly understand a piece of systems software is to write it yourself.

The source code is on GitHub.