TinyComputers.io (Posts about qemu)

Four Partitions and a Borrowed Bootloader: NetBSD on the Milk-V Mars

A.C. Jokela — Sun, 19 Jul 2026 21:30:00 GMT

Four Partitions and a Borrowed Bootloader: NetBSD on the Milk-V Mars

As I write this, there is a Milk-V Mars on my network running NetBSD-current with a 28 GB root filesystem, gigabit Ethernet passing a ping flood at 0.0% loss, and an SSH prompt that answers like any other machine on the shelf. As far as I can tell it's the only documented NetBSD Mars in existence. The NetBSD wiki lists the hardware as supported, but publishes no image and no instructions, and neither does anyone else. Closing the gap between "supported" and "boots" took a weekend, three wrong theories, one self-inflicted filesystem corruption, and a serial console I put off hooking up for far too long. This is the whole story, and it ends with a one-line fix.

How a JH7110 finds an operating system

The JH7110's boot chain has four links. The BootROM in silicon reads the GPT from sector 1 of the card and hunts for a partition by type GUID (2E54B353-…) which holds the SPL. The SPL brings up DDR and loads a FIT image (OpenSBI + U-Boot) from a second type GUID, BC13C2FF-…. U-Boot finds an OS somehow (hold that thought) and NetBSD's bootriscv64.efi pulls the kernel off an FFS partition. Four partitions:

#	Contents	Found by
1	SPL	BootROM, via type GUID
2	OpenSBI + U-Boot FIT	SPL, via type GUID
3	EFI System Partition	U-Boot
4	NetBSD FFS root	the ESP's bootloader

Partitions 3 and 4 come wholesale from NetBSD's daily riscv64 image (nycdn.netbsd.org; JH7110 support lives only in -current). Assembly is sgdisk, mtools, and dd on a Mac: two minutes, scripted end-to-end in my repo's build-image.sh. The build phase's happiest find: NetBSD's image ships jh7110-milkv-mars.dtb, a device tree for this exact board, matched to its own drivers. Remember that file; it matters later.

Partitions 1 and 2 are the borrowed bootloader, and they're where everything went sideways.

The board itself: Mars V1.21, StarFive JH7110 in the middle, the 40-pin header along the top edge (serial console on pins 6, 8, and 10), and the tiny two-position boot DIP switch tucked against the USB3 stack.

MilkV's own practice pointed at a workaround: their Debian card's uEnv.txt contains, verbatim, # Fix missing bootcmd. The vendor patches its own firmware's environment at every boot. Following suit, my card grew a uEnv.txt hijacking their fdt_loaddtb hook into an EFI boot. QEMU validated it against the vendor's exact command sequence. The board still didn't boot. With no console, I turned the card into its own flight recorder: U-Boot has fatwrite, so the hook gained breadcrumb files written to the FAT partition at each stage. Boot, pull the card, read it back. Verdict, across multiple boots: zero breadcrumbs. The firmware on this board never even imported the file its own design patches with. At that point it made more sense to stop accommodating the vendor firmware entirely.

The transplant, and the board that "died"

If the vendor bootloader won't speak any language my card can write, replace the bootloader: Debian's u-boot-starfive package carries mainline 2025.01 for the VisionFive 2, the exact version the U-Boot docs bless for this family, and the vendor's own Mars firmware is literally a VF2 build with the filename changed. SPL into partition 1, FIT into partition 2.

And the board booted. DHCP lease with NetBSD's fingerprints (a locally-administered MAC and a dhcpcd DUID); port 22 open. Then, minutes later: gone. Every boot: appear, vanish. I had three theories (kernel panic, power sag, bad -current daily) and a day of DHCP-lease forensics that couldn't pick between them.

The serial console — a USB-UART hanging off a Raspberry Pi across the room, driven over SSH, after its own detour through baud-rate resets, a missing clocal that made writes block forever, and five stacked stty processes wedging the tty — settled it in ten minutes:

NetBSD/riscv (riscv64) (constty)

login:

A login prompt. On the board I'd declared dead. All three theories wrong at once: the board was never dying. What died was its network. The OS sat there healthy, unreachable, while I issued death certificates on the testimony of a router. The kernel had already named the killer at boot:

[ 1.000000] mcommphy0: autoconfiguration error: unknown drive strength (3 uA at 1800 mV)
[ 1.000000] eqos1: reset timeout!

A PHY misconfigured by unrecognized device-tree values, and eqos1, a second Ethernet controller on a board that has one port. The VisionFive 2 has two. The kernel was running the VF2 device tree. Mainline U-Boot had silently fallen back to its built-in FDT, and NetBSD, handed a description of the wrong board, drove the PHY into dropping four of every five frames, with the survivors crossing in a perfectly healthy 0.4 ms. At the U-Boot prompt:

StarFive # printenv fdtfile
## Error: "fdtfile" not defined
StarFive # setenv fdtfile starfive/jh7110-milkv-mars.dtb
StarFive # bootflow scan
...
riscv64# ping -c 10 -i 0.3 10.1.1.1 | tail -2
10 packets transmitted, 10 packets received, 0.0% packet loss

Zero percent packet loss. One environment variable had been the entire problem. env save wrote it to SPI flash, and every boot since has been autonomous and clean.

One empty string

Why was fdtfile never set? This is my favorite finding of the project. The VF2 board code composes it from the board's ID EEPROM, and the Mars's EEPROM is blank. The env template ends with "fdtfile=" CONFIG_DEFAULT_FDT_FILE, and the defconfig doesn't set CONFIG_DEFAULT_FDT_FILE, so the template renders as fdtfile=. U-Boot's env import treats an empty value as "delete the variable." There is no dedicated Mars defconfig upstream; Mars support is VF2-plus-EEPROM. Blank EEPROM, empty default, deleted variable, wrong device tree, 80% packet loss. The fix, compiled into the custom U-Boot that now ships in my repo (cross-built by build-uboot.sh on a Raspberry Pi):

CONFIG_DEFAULT_FDT_FILE="starfive/jh7110-milkv-mars.dtb"

One line — though, as is usually the way with these things, finding the line was the weekend's work.

The superblock that fought back

Two more fights before the finish line, both instructive. First, the kernel panicked on one heavily-loaded boot: a genuine NetBSD-current riscv bug, caught with a full backtrace over serial (pmap_segtab.c:948, assertion failed in the pagedaemon; report drafted for port-riscv). It retroactively explains the earlier "deaths" too: panic, then a reboot attempt that hangs, because (the second fight) the VF2-flavored firmware can't drive the Mars's PMIC. pmic_ops: cannot read pmic power register, and every software reboot on this board is actually a power cycle. Permanent caveat until someone builds Mars-aware firmware.

And then the filesystem resize humbled me. NetBSD's images grow their root on first boot and then run resize_root_postcmd="/sbin/reboot -n": reboot without syncing. That looked like a bug to me on a board where reboot hangs, so I emptied it. Days later I learned what it's for: after resizing a mounted-read-only root, the kernel still holds the old superblock in memory, and any later sync writes stale metadata over the resized filesystem. Mine did exactly that (BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE), and fsck, offered two truths, chose the stale one and un-resized the disk. The recovery (from the boot-failure shell, root mounted read-only): fsck_ffs -f -y, resize_ffs -y, verify, then exit via reboot -n, the exact no-sync reboot I'd deleted, so the fix reaches disk unclobbered. The next power cycle mounted 28 GB clean. Weird-looking safeties are load-bearing. Understand why a thing exists before you remove it, especially when it's one flag on a reboot command.

Where it stands

The scoreboard, finally all green where it counts: Boot chain: mainline U-Boot, EFI, Mars DTB from compiled-in default, fully autonomous. Ethernet: 0.0% loss, stable pinned MAC. Storage: 28 GB grown FFS, clean. Access: SSH from anywhere on my LAN, serial console rig for the deep end. Known caveats: power-cycle instead of reboot (PMIC), and a recurring kernel panic in the UVM layer under sustained heavy write load (two distinct backtraces captured over serial, pagedaemon/pmap and buffer-cache list corruption, pointing at a common riscv MD bug; upstream report in progress). Day-to-day use is stable; sustained bulk writes (extracting the 150,000-file pkgsrc tree, for instance) trigger it reliably.

Proof of life, over SSH: NetBSD 11.99.7, CPU reported as milkv,mars, 48 minutes of uptime, and neofetch installed from the riscv64 binary package repository.

Everything is scripted and reproducible in the repo, github.com/ajokela/milkv-netbsd; see the README for build instructions, the DIP-switch table, and known issues. build-image.sh builds the image (NetBSD daily + custom U-Boot + FAT16 ESP + your SSH key, two minutes on a Mac), build-uboot.sh reproduces the one-line-patched firmware on any Debian box, and customize-image.sh boots the image in QEMU and bakes in distribution defaults. The next step is the thing this niche deserves and doesn't have: a flashable image. Default credentials root/netbsd with a motd that yells at you to change them, in the grand SBC tradition; unique host keys and entropy generated on each board's own first boot; a persistent MAC minted on first touch. Download: netbsd-mars-20260719.img.xz (234 MB, SHA512) — flash with xz -dc netbsd-mars-20260719.img.xz | sudo dd of=/dev/rdiskN bs=1m, set the DIP switch to SD, and log in as root/netbsd.

[     1.000000] ------------------------------------------
[     1.000000]            kern_vtopdiff = 0xffffffbf03400000
[     1.000000]             memory_start = 0x        40060000
[     1.000000]               memory_end = 0x       240000000
[     1.000000]              memory_size = 0x       1fffa0000
[     1.000000]           kernstart_phys = 0x        fcc00000
[     1.000000]             kernend_phys = 0x        fd800000
[     1.000000]               msgbufaddr = 0xffffffe23fff0000
[     1.000000]    VM_MIN_KERNEL_ADDRESS = 0xffffffc000000000
[     1.000000]           kernstart_mega = 0xffffffc000000000
[     1.000000]                kernstart = 0xffffffc000000000
[     1.000000]                  kernend = 0xffffffc000c00000
[     1.000000]             kernend_mega = 0xffffffc000c00000
[     1.000000]    VM_MAX_KERNEL_ADDRESS = 0xffffffd000000000
[     1.000000]         pmap_direct_base = 0xffffffe000000000
[     1.000000] ------------------------------------------
[     1.000000] adding       0x40060000 -       0xfcc00000  to freelist 0
[     1.000000] adding       0xfd800000 -       0xfda00000  to freelist 0
[     1.000000] adding       0xfda0a000 -       0xfe6e8000  to freelist 0
[     1.000000] adding       0xfe6f5000 -       0xfe6f9000  to freelist 0
[     1.000000] adding       0xfe6fa000 -       0xfe6fb000  to freelist 0
[     1.000000] adding       0xfe6fd000 -       0xfe700000  to freelist 0
[     1.000000] adding       0xfe725000 -       0xff762000  to freelist 0
[     1.000000] adding       0xff763000 -       0xfff4a000  to freelist 0
[     1.000000] adding       0xfff4c000 -      0x23fff0000  to freelist 0
[     1.000000] common cpusets bs_pde 0xffffffc000a0b000 ASID max 0 kcpusets nkmempages limits: ffffffc004000000 - ffffffd000000000
[     1.000000] kva   : ffffffc004000000 - ffffffc2b9000000
[     1.000000] mpstart
[     1.000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
[     1.000000]     2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013,
[     1.000000]     2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023,
[     1.000000]     2024, 2025, 2026
[     1.000000]     The NetBSD Foundation, Inc.  All rights reserved.
[     1.000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
[     1.000000]     The Regents of the University of California.  All rights reserved.

[     1.000000] NetBSD 11.99.7 (GENERIC64) #0: Sat Jul 18 15:41:53 UTC 2026
[     1.000000]     mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/riscv/compile/GENERIC64
[     1.000000] total memory = 8179 MB
[     1.000000] avail memory = 7910 MB
[     1.000000] entropy: ready
[     1.000000] timecounter: Timecounters tick every 10.000 msec
[     1.000000] WARNING: module error: Built-in module `iic' can't find built-in dependency `i2c_bitbang'
[     1.000000] WARNING: module error: Built-in module `iic' prerequisite `i2c_bitbang' failed, error 2
[     1.000000] mainbus0 (root)
[     1.000000] simplebus0 at mainbus0: Milk-V Mars
[     1.000000] simplebus1 at simplebus0
[     1.000000] cpus0 at simplebus0
[     1.000000] simplebus2 at simplebus0
[     1.000000] cpu0 at cpus0: SiFive 7-Series Processor (E7, S7, U7 series) imp. 4210427
[     1.000000] cpu0: vendor 0x489 arch. 8000000000000007 imp. 4210427
[     1.000000] cpu1 at cpus0: SiFive 7-Series Processor (E7, S7, U7 series) imp. 4210427
[     1.000000] cpu1: vendor 0x489 arch. 8000000000000007 imp. 4210427
[     1.000000] cpu2 at cpus0: SiFive 7-Series Processor (E7, S7, U7 series) imp. 4210427
[     1.000000] cpu2: vendor 0x489 arch. 8000000000000007 imp. 4210427
[     1.000000] cpu3 at cpus0: SiFive 7-Series Processor (E7, S7, U7 series) imp. 4210427
[     1.000000] cpu3: vendor 0x489 arch. 8000000000000007 imp. 4210427
[     1.000000] intc0 at cpu0: local interrupt controller
[     1.000000] intc1 at cpu1: local interrupt controller
[     1.000000] intc2 at cpu2: local interrupt controller
[     1.000000] intc3 at cpu3: local interrupt controller
[     1.000000] fclock0 at simplebus0: 49152000 Hz fixed clock (tdm_ext)
[     1.000000] fclock1 at simplebus0: 12288000 Hz fixed clock (mclk_ext)
[     1.000000] fclock2 at simplebus0: 12288000 Hz fixed clock (i2srx_bclk_ext)
[     1.000000] fclock3 at simplebus0: 192000 Hz fixed clock (i2srx_lrck_ext)
[     1.000000] syscon0 at simplebus1: System Controller Registers
[     1.000000] jh7110syscon0 at simplebus1: JH7110 STG system controller
[     1.000000] fclock4 at simplebus0: 24000000 Hz fixed clock (osc)
[     1.000000] fclock5 at simplebus0: 12288000 Hz fixed clock (i2stx_bclk_ext)
[     1.000000] fclock6 at simplebus0: 192000 Hz fixed clock (i2stx_lrck_ext)
[     1.000000] fclock7 at simplebus0: 50000000 Hz fixed clock (gmac1_rmii_refin)
[     1.000000] fclock8 at simplebus0: 125000000 Hz fixed clock (gmac1_rgmii_rxin)
[     1.000000] syscon1 at simplebus1: System Controller Registers
[     1.000000] fclock9 at simplebus0: 50000000 Hz fixed clock (gmac0_rmii_refin)
[     1.000000] fclock10 at simplebus0: 125000000 Hz fixed clock (gmac0_rgmii_rxin)
[     1.000000] fclock11 at simplebus0: 32768 Hz fixed clock (rtc_osc)
[     1.000000] fclock12 at simplebus0: 74250000 Hz fixed clock (dvp_clk)
[     1.000000] fclock13 at simplebus0: 297000000 Hz fixed clock (hdmitx0_pixelclk)
[     1.000000] jh7110clkc0 at simplebus1: JH7110 System Clock and Reset Generator
[     1.000000] plic0 at simplebus1
[     1.000000] plic0: inactive HART for PLIC for context 0
[     1.000000] plic0: interrupt cpu0 (hart 1)/Machine external #11 handler installed
[     1.000000] plic0: interrupt cpu0 (hart 1)/Supervisor external #9 handler installed
[     1.000000] plic0: hart 1 context 2
[     1.000000] plic0: interrupt cpu1 (hart 2)/Machine external #11 handler installed
[     1.000000] plic0: interrupt cpu1 (hart 2)/Supervisor external #9 handler installed
[     1.000000] plic0: hart 2 context 4
[     1.000000] plic0: interrupt cpu2 (hart 3)/Machine external #11 handler installed
[     1.000000] plic0: interrupt cpu2 (hart 3)/Supervisor external #9 handler installed
[     1.000000] plic0: hart 3 context 6
[     1.000000] plic0: interrupt cpu3 (hart 4)/Machine external #11 handler installed
[     1.000000] plic0: interrupt cpu3 (hart 4)/Supervisor external #9 handler installed
[     1.000000] plic0: hart 4 context 8
[     1.000000] plic0: attaching: RISC-V PLIC (136 IRQs)
[     1.000000] jh7110pinctrl0 at simplebus1: Pin Controller
[     1.000000] jh7110clkc1 at simplebus1: JH7110 System-Top-Group Clock and Reset Generator
[     1.000000] jh7110clkc2 at simplebus1: JH7110 Always-On Clock and Reset Generator
[     1.000000] jh7110pinctrl1 at simplebus1: Pin Controller
[     1.000000] jh7110clkc3 at simplebus1: JH7110 Video Output Clock and Reset Generator
[     1.000000] jh7110pciephy0 at simplebus1: JH7110 PCIe PHY
[     1.000000] jh7110pciephy1 at simplebus1: JH7110 PCIe PHY
[     1.000000] ccache0 at simplebus1: L2 cache controller. 2048 KiB/64B 16-way (2048 set).
[     1.000000] ccache0: No. of banks             8
[     1.000000] ccache0: No. of ways per bank    16
[     1.000000] ccache0: Sets per bank          256
[     1.000000] ccache0: Bytes per cache block   64
[     1.000000] ccache0: Largest way enabled     15
[     1.000000] /opp-table-0 at simplebus0 not configured
[     1.000000] dwiic0 at simplebus1: I2C controller
[     1.000000] dwiic0: interrupting on plic0 irq 50
[     1.000000] iic0 at dwiic0: I2C bus
[     1.000000] pmic (x-powers,axp15060) at iic0 addr 0x36 not configured
[     1.000000] jh71x0temp0 at simplebus1: JH71x0 temperature sensor
[     1.000000] /soc/dma-controller@16050000 at simplebus1 not configured
[     1.000000] /soc/phy@10200000 at simplebus1 not configured
[     1.000000] /soc/syscon@13030000/clock-controller at syscon0 not configured
[     1.000000] /soc/dma-controller@16008000 at simplebus1 not configured
[     1.000000] eqos0 at simplebus1: DesignWare EQOS ver 0x52 (0x41)
[     1.000000] eqos0: hw features 1a2173f7 09845904 01000000 00000000
[     1.000000] eqos0: Ethernet address f2:00:1b:33:82:be
[     1.000000] mcommphy0 at eqos0 phy 0: Motorcomm YT8531 GbE PHY
[     1.000000] mcommphy0: motorcomm,tx-clk-adj-enabled is true
[     1.000000] mcommphy0: motorcomm,tx-clk-100-inverted is true
[     1.000000] mcommphy0: motorcomm,tx-clk-1000-inverted is true
[     1.000000] mcommphy0: rx-clk-drv-microamp is 3970
[     1.000000] mcommphy0: motorcomm,rx-data-drv-microamp is 2910
[     1.000000] mcommphy0: rx-internal-delay-ps is 1500
[     1.000000] mcommphy0: tx-internal-delay-ps is 1500
[     1.000000] mcommphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, auto
[     1.000000] eqos0: interrupting on plic0 irq 7
[     1.000000] /soc/clock-controller@19810000 at simplebus1 not configured
[     1.000000] /soc/phy@19820000 at simplebus1 not configured
[     1.000000] /soc/power-controller@17030000 at simplebus1 not configured
[     1.000000] jh7110pcie0 at simplebus1: PCIe
[     1.000000] jh7110pcie0: interrupting on plic0 irq 56
[     1.000000] jh7110pcie0: MMIO (32-bit non-prefetchable): 0x0000000030000000 + 0x0000000008000000 @ 0x0000000030000000
[     1.000000] jh7110pcie0: MMIO (64-bit prefetchable)    : 0x0000000900000000 + 0x0000000040000000 @ 0x0000000900000000
[     1.000000] pci0 at jh7110pcie0 bus 0
[     1.000000] pci0: memory space enabled, rd/line, rd/mult, wr/inv ok
[     1.000000] ppb0 at pci0 dev 0 function 0: vendor 1556 product 1111 (rev. 0x02)
[     1.000000] ppb0: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x1 @ 5.0GT/s
[     1.000000] pci1 at ppb0 bus 1
[     1.000000] pci1: memory space enabled, rd/line, wr/inv ok
[     1.000000] xhci0 at pci1 dev 0 function 0: vendor 1106 product 3483 (rev. 0x01)
[     1.000000] xhci0: 64-bit DMA
[     1.000000] xhci0: interrupting at plic0 irq 56
[     1.000000] xhci0: xHCI version 1.0
[     1.000000] usb0 at xhci0: USB revision 3.0
[     1.000000] usb1 at xhci0: USB revision 2.0
[     1.000000] jh7110pcie1 at simplebus1: PCIe
[     1.000000] jh7110pcie1: autoconfiguration error: link not up
[     1.000000] /sound at simplebus0 not configured
[     1.000000] /soc/pwmdac@100b0000 at simplebus1 not configured
[     1.000000] /audio-codec at simplebus0 not configured
[     1.000000] /soc/timer@2000000 at simplebus1 not configured
[     1.000000] com0 at simplebus1: DesignWare APB UART, no fifo
[     1.000000] com0: console
[     1.000000] com0: interrupting on plic0 irq 32
[     1.000000] dwiic1 at simplebus1: I2C controller
[     1.000000] dwiic1: interrupting on plic0 irq 35
[     1.000000] iic1 at dwiic1: I2C bus
[     1.000000] dwiic2 at simplebus1: I2C controller
[     1.000000] dwiic2: interrupting on plic0 irq 37
[     1.000000] iic2 at dwiic2: I2C bus
[     1.000000] /soc/spi@10060000 at simplebus1 not configured
[     1.000000] jh71x0usb0 at simplebus1: USB
[     1.000000] dwiic3 at simplebus1: I2C controller
[     1.000000] dwiic3: interrupting on plic0 irq 51
[     1.000000] iic3 at dwiic3: I2C bus
[     1.000000] /soc/pwm@120d0000 at simplebus1 not configured
[     1.000000] /soc/spi@13010000 at simplebus1 not configured
[     1.000000] /soc/crypto@16000000 at simplebus1 not configured
[     1.000000] jh7110trng0 at simplebus1: JH7110 TRNG
[     1.000000] jh7110trng0: Features    : 0xd<mode reset state=0x1=mission mode,ring oscillator=0x1=present,max rand length=0x1=256-bit>
[     1.000000] jh7110trng0: Build config: 0x3d<auto reseed loopback=0x1=present,mode after reset=0x1=mission mode,PRNG after reset=0x1=present,max PRNG length=0x1=256-bit>
[     1.000000] jh7110trng0: interrupting on plic0 irq 30
[     1.000000] dwcmmc0 at simplebus1: DesignWare SD/MMC
[     1.000000] dwcmmc0: interrupting on plic0 irq 74
[     1.000000] dwcmmc1 at simplebus1: DesignWare SD/MMC
[     1.000000] dwcmmc1: interrupting on plic0 irq 75
[     1.000000] /gpio-restart at simplebus0 not configured
[     1.000000] /soc/watchdog@13070000 at simplebus1 not configured
[     1.000000] xhci1 at simplebus1autoconfiguration error: : cannot switch 'otg' mode to host
[     1.000000] timecounter: Timecounter "riscv" frequency 4000000 Hz quality 100
[     1.000003] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
[     1.799984] cpu_boot_secondary_processors: starting secondary processors
[     1.799984] cpu_boot_secondary_processors: secondary processors hatched. 4 running
[     1.815242] uhub0 at usb0: NetBSD (0x0000) xHCI root hub (0x0000), class 9/0, rev 3.00/1.00, addr 0
[     1.819981] uhub0: 4 ports with 4 removable, self powered
[     1.819981] uhub1 at usb1: NetBSD (0x0000) xHCI root hub (0x0000), class 9/0, rev 2.00/1.00, addr 0
[     1.829977] uhub1: 1 port with 1 removable, self powered
[     1.829977] sdmmc0 at dwcmmc0
[     1.829977] sdmmc1 at dwcmmc1
[     1.849982] IPsec: Initialized Security Association Processing.
[     1.959977] sdmmc1: SD card status: 4-bit, C10, U1, V10, A1
[     1.959977] ld1 at sdmmc1: <0x27:0x5048:SD32G:0x60:0x00000307:0x156>
[     1.969977] ld1: 29508 MB, 7494 cyl, 128 head, 63 sec, 512 bytes/sect x 60432384 sectors
[     1.979977] ld1: GPT GUID: e27a702f-df79-4d5a-be5f-1cda14fa33e4
[     1.979977] dk0 at ld1: "spl", 4096 blocks at 2048, type: <unknown>
[     1.989977] dk1 at ld1: "uboot", 8192 blocks at 8192, type: <unknown>
[     1.999977] dk2 at ld1: "EFI", 163840 blocks at 16384, type: msdos
[     1.999977] dk3 at ld1: "netbsd-root", 60252127 blocks at 180224, type: ffs
[     2.009977] ld1: 4-bit width, High-Speed/SDR25, 50.000 MHz
[     2.269976] uhub2 at uhub1 port 1: vendor 2109 (0x2109) USB2.0 Hub (0x3431), class 9/0, rev 2.10/4.20, addr 1
[     2.279977] uhub2: single transaction translator
[     2.289976] uhub2: 4 ports with 4 removable, self powered
[     2.629983] swwdog0: software watchdog initialized
[     2.629983] WARNING: 2 errors while detecting hardware; check system log.
[     2.641566] boot device: dk3
[     2.641566] root on dk3
[     2.659978] root file system type: ffs
[     2.669979] kern.module.path=/stand/riscv/11.99.7/modules
[     2.669979] WARNING: no TOD clock present
[     2.681527] WARNING: using filesystem time
[     2.685595] WARNING: CHECK AND RESET THE DATE!

Nothing Was Using the Cores: Three Idle Arm64 Boards, One HA k3s Cluster, and a BSD Build Farm

A.C. Jokela — Sun, 05 Jul 2026 01:30:00 GMT

Update (August 2026): This cluster has since grown from three nodes to eight, and it is no longer all Arm — two x86-64 boxes have joined the quorum's company, along with the NanoPC-T6N I reviewed a few weeks later. One of the two projects those BSD builds exist for — the ballistics engine — now has a post of its own: what it actually computes, how you test physics when there is no oracle to check it against, and why it ships to thirteen platforms.

The plan, at first, was to buy my way there. I wanted a three-node cluster, and the tidy way to get one is three identical machines — so I priced out two more Orange Pi 5 Max boards to stand beside the one I already had: same chip, same memory, same clean symmetry. Then I looked at the total.

We are, as I write this, deep in a memory and storage shortage, and the prices show it. DRAM and NAND have both climbed to where two more maxed-out single-board computers — sixteen gigabytes of LPDDR and a fast terabyte of NVMe apiece — added up to real money, for hardware that would spend most of its life, like everything else on that shelf, doing nothing. Buying idle capacity to set it next to idle capacity is a special kind of foolish, and I could feel that before I could quite justify it.

So before spending anything, I did the cheaper thing first — the thing I should have done to begin with. I took stock of what I already owned.

I had three small computers on a shelf, and on any given afternoon they were doing almost nothing.

That is not a complaint about the hardware. The hardware is lovely. There was an Orange Pi 5 Max — an RK3588 board, eight cores split four-and-four between big Cortex-A76s and little A55s, sixteen gigabytes of RAM, a neural processing unit, and a one-terabyte NVMe drive bolted underneath. Next to it, a Raspberry Pi 5, four cores and eight gigabytes and its own terabyte of WD_BLACK NVMe, quietly serving a web page. And a Compute Module 5 on a carrier, four cores, another sixteen gigabytes, two terabytes of NVMe — the fastest storage of the three and the least to do.

Add it up and it's sixteen cores, forty gigabytes of memory, four terabytes of solid-state storage. And the honest utilization graph, most of the time, was a flat line near the floor. The Orange Pi woke up when I asked it to run a local language model on its NPU. The Pi answered HTTP. The CM5 held its breath. Three machines, each doing one modest thing, each idle in the gaps — which is to say, idle almost always.

I have written before on this blog about hauling systems software onto hardware it was never supposed to run on, and there's a particular feeling that precedes those projects: a gap that's infuriating rather than merely inconvenient, a mismatch between what a machine plainly could do and what it was actually doing. This was that feeling, pointed inward at my own shelf. The boards were not underpowered. They were under-asked. And three of anything, on a network together, with fast local disks and nothing to prove — that is not three appliances. That is a cluster that hasn't been told yet.

So I told it.

Three boxes, one quorum

The goal was a real high-availability Kubernetes cluster, not a toy. For three small Arm boards that means k3s — Rancher's single-binary Kubernetes, which folds the API server, scheduler, controller manager, a container runtime, and networking into one process that starts in seconds and sips memory. On a fleet of SBCs it is the difference between "Kubernetes runs here" and "Kubernetes runs here and also leaves you room to run anything else."

High availability, specifically, is the reason there are three boards in this story and not two. Kubernetes keeps its entire state in etcd, and etcd is a consensus system: it stays available only as long as a strict majority of its members can see each other. With two nodes, a majority is two — lose either and the cluster's brain goes read-only. With three, a majority is two out of three, and you can lose any single node — reboot it, unplug it, drop its switch port — and the cluster keeps making decisions. Three is the smallest number where "highly available" stops being marketing. k3s can run etcd embedded across the server nodes, so all three boards became servers, all three became voting members of the quorum, and all three stayed schedulable for real work. No dedicated control plane sitting idle; the control plane is the workers.

Getting there was mostly smooth, with one wrinkle that everyone who has tried this on Raspberry Pi hardware eventually meets. Kubernetes needs the kernel's memory cgroup controller to enforce per-container memory limits, and the Raspberry Pi OS kernel ships with it switched off by default — a small performance optimization for people who aren't running containers, and a hard wall for people who are. The kubelet simply refuses to start. The fix is one line appended to /boot/firmware/cmdline.txt — cgroup_enable=memory cgroup_memory=1 — and a reboot, which I automated across the Pis with Ansible so the change was recorded rather than remembered. The Orange Pi's Armbian kernel already had the controller enabled, so it needed nothing. That asymmetry — same architecture, same job, different kernel defaults — is the texture of a homelab built from whatever excellent boards happened to be on sale.

I pointed k3s's data directory at the NVMe on each node instead of the default location on the boot media. etcd is unusually sensitive to storage latency — it fsyncs every write it commits — and the difference between a consensus log on an SD card and one on an NVMe drive is the difference between a cluster that feels sluggish and one that feels like a server. It also spares the SD and eMMC cards the write amplification that eventually kills them. Fast local disk is the one thing all three boards had in surplus, and the control plane was the workload most grateful to use it.

A short while later, kubectl get nodes returned three lines, all Ready, all control-plane,etcd, and the flat utilization line finally had somewhere to go. But an empty cluster is just a more elaborate way of being idle. The point was never Kubernetes. The point was the thing the Orange Pi had quietly been doing all along, badly.

NAME              STATUS   ROLES                AGE   VERSION
orangepi5-max     Ready    control-plane,etcd   25h   v1.36.2+k3s1
raspberrypi-cm5   Ready    control-plane,etcd   25h   v1.36.2+k3s1
rpi5              Ready    control-plane,etcd   25h   v1.36.2+k3s1

The build servers that never slept

Here is what had actually been running on that Orange Pi, underneath the occasional language model: three virtual machines, on all the time, existing only to compile software.

I maintain two projects that care a great deal about running natively on the BSDs. It is not because there is a huge BSD following for these projects. In fact, there are probably no dedicated BSD users. I do, however, have a soft spot for the BSDs. For nearly two decades, FreeBSD was my server operating system of choice. When I was into vintage hardware, like Sun Microsystems workstations from the 1990s, or the MicroVAX II I once owned, NetBSD was my preferred operating system.

One of the projects is a small programming language and its compiler, written in C, that I like to ship as real binaries for FreeBSD, OpenBSD, and NetBSD rather than telling people to build it themselves. The other is a ballistics engine written in Rust with the same ambition. Cross-compiling to the BSDs from Linux is possible in theory and miserable in practice — you need each target's system libraries, its linker quirks, its idea of where things live — so the honest way to get a native FreeBSD Arm64 binary is to build it on FreeBSD Arm64. The three VMs were exactly that: a FreeBSD guest, an OpenBSD guest, and a NetBSD guest, each an aarch64 machine running under QEMU with KVM acceleration, each holding a full toolchain, each reachable over SSH so a build script could push source in and pull a binary out.

They worked. And they were awake twenty-four hours a day to be useful for the few minutes a week I actually cut a release. Three virtual machines, two gigabytes of RAM apiece, pinned CPU, humming along through the night so that occasionally a compiler could run. It's the same waste as the idle cluster, just wearing a more respectable coat: capacity committed to a workload that is almost never present.

Once there was a cluster underneath them, the reshape was obvious. A build is a job — it starts, it runs, it produces an artifact, it ends. That is precisely the shape Kubernetes was built around. The always-on VMs wanted to become on-demand Jobs: nothing running when there's nothing to build, and when a build is needed, a pod springs up, does the work on real cores, and disappears.

The catch is the part that makes this a story instead of a paragraph. A Kubernetes Job runs a Linux container — a process sharing the host's Linux kernel. You cannot boot a FreeBSD kernel, or an OpenBSD one, or a NetBSD one, inside an ordinary container; there is no BSD kernel there to boot. The real BSD environment lives in those VM disk images, and the only honest way to build on a BSD is to run the BSD. So the design that survived contact with reality was a VM inside the Job: a privileged pod that mounts the host's /dev/kvm, boots the golden BSD image with QEMU at native speed, waits for the guest's SSH to come up, runs the project's own build script over that SSH connection exactly as before, copies the finished binary out to a persistent volume, and powers the guest down. The VM didn't go away. It stopped being a thing that's always on and became a thing that exists for the length of one build.

The pile of small, specific lies

The plan was clean. The path from plan to working was paved with the kind of bugs that only appear when real hardware stops being polite, and I want to walk through them, because each one is a small lesson and together they are the whole texture of the job.

The first was the most alarming and the most educational. I booted the FreeBSD image inside a pod, and QEMU came up spinning at four hundred percent CPU while the guest's console filled with synchronous exception panics — the Arm equivalent of a program tripping over an instruction the processor won't execute. The image booted fine on the host directly. It crashed reliably inside the pod. The difference was the RK3588's defining feature and, here, its trap: it is a big.LITTLE chip, four fast A76 cores and four efficient A55 cores, and they do not implement exactly the same instructions. QEMU was passing the host CPU straight through to the guest, and the Linux scheduler, seeing an unpinned QEMU thread, was free to migrate the virtual CPU from a big core to a little core mid-execution. The guest would start an instruction the A76 supports, get moved to an A55 that doesn't, and fall over. The host's own VM launch scripts had known this all along — they pinned QEMU with taskset -c 4-7, confining it to the four A76 cores — and my pod hadn't. One taskset and the synchronous exceptions vanished. The cluster's cleverest hardware feature was, for this workload, a thing to carefully switch off.

The second bug was a quiet philosophical one. My plan referred throughout to the BSD disk images as "golden images" — static, pristine templates you overlay a scratch copy on top of and never mutate. When I went to make that scratch overlay, QEMU refused: the image was locked, held open read-write by another process. The other process was the always-on VM. The "golden image" was the live disk of the running build server — not a template at all, but the working drive of a machine that was, at that moment, mid-thought. Overlaying a copy-on-write layer on a disk that's being written underneath you is not a snapshot; it's a data-corruption generator. And when I did force a copy, it failed its filesystem check on boot, because the source had been running for thirteen days without a clean shutdown and its dirty bit had never been cleared. The fix was to do the boring, correct thing I'd been skipping: cleanly shut each VM down, take a genuine point-in-time snapshot of the now-consistent disk into a real golden image, and restart the VM. The builds overlay those — static, clean, never open by anything else. It cost a couple of minutes of downtime per guest and bought a foundation that isn't quietly rotting.

Then the small, specific lies of virtual hardware. The first pod boot that got past the CPU pinning died instantly with QEMU complaining it couldn't find efi-virtio.rom — a network-card boot ROM that isn't packaged in a slim container image. The host scripts, again, already knew: they set romfile= empty on the virtual NIC, disabling the ROM the build doesn't need. Add that, and the guest booted further — and then OpenBSD, specifically, hung partway through, wedged on device probing. FreeBSD had sailed through the same configuration. The difference was vectors=0, another flag the host scripts carried and mine didn't, which disables MSI-X interrupts on the virtual devices; OpenBSD's virtio drivers, under emulation, needed the older interrupt style to come up at all. FreeBSD tolerated the modern path. OpenBSD did not. The host's launch scripts, it turned out, were a compressed archive of exactly these lessons, learned once and encoded as flags I had to re-learn by leaving them out.

The BSDs then taught me about their own disks. My build script staged source into the guest's home directory and immediately hit "No space left on device." OpenBSD, by long tradition, slices its disk into many small partitions, and its root filesystem — where a naive home directory lives — was four hundred megabytes and already full. NetBSD does the same, with a root barely over a gigabyte. FreeBSD hands you one big root and doesn't care. So the builder learned to build where the space actually is: on OpenBSD, in /usr/obj, the partition the system reserves for build output; on NetBSD, in /usr/pkg, its roomy second disk. Rust made this worse before it made it better, because Cargo unpacks its entire crate registry into a cache under the home directory, and on NetBSD that cache alone overflowed the tiny root the instant a real dependency graph arrived. Pointing CARGO_HOME at the big partition too, per guest, finally let a Rust project build on a NetBSD machine whose root filesystem couldn't have held its own dependencies.

The last of them wasn't a build bug at all; it was a networking cleanup that turned into an accidental repair. NetBSD's VM, unlike its siblings, had been bridged directly onto my LAN, while FreeBSD and OpenBSD used QEMU's simpler user-mode networking. I wanted them consistent, so I converted NetBSD to user-mode too — and in restarting it discovered that the bridged setup would have failed on its very next boot regardless, because a QEMU package update had silently stripped the CAP_NET_ADMIN capability off the bridge helper, the small privileged binary that lets an unprivileged VM attach to a bridge. The VM had been running for so long it hadn't been restarted since the update landed. The cleanup I wanted to do had been sitting on top of a latent failure I didn't know about, and converting away from bridging both simplified the design and defused a bomb.

None of these were deep. Every one was a mundane mistake or a mundane environmental fact, wearing for an afternoon the costume of something profound. That is the entire pattern of this kind of work, and I've made peace with it: the exotic theory that flatters your understanding almost always sits on top of a boring cause that flatters nothing. taskset. romfile=. A partition that's full. A capability an update took away.

When it was done, the payoff was concrete. A single command — make bsd-build PROJECT=lattice OS=freebsd — now fetches the latest source straight from GitHub, boots the golden FreeBSD image in a pod on the Orange Pi, compiles the project inside it, and drops a genuine aarch64 FreeBSD executable back onto my workstation, stripped and ready. The same command builds for OpenBSD and NetBSD, for both projects, and a release variant does all of it and publishes. Six native BSD binaries, produced by boards that used to keep three virtual machines awake all night to do the same thing on demand for no one.

Naming the tenants

With the builds turned into jobs, the always-on VMs had no reason to exist. I stopped and disabled them — reversibly; the golden images remain, and a systemctl away is the old world back — and the Orange Pi handed back roughly six gigabytes of RAM and its four A76 cores. Which returns us, neatly, to the theme this whole exercise keeps circling: reclaimed idle capacity is only worth reclaiming if you then ask something of it.

So here is the honest census of what actually runs on the cluster now — the jobs, named plainly.

On-demand BSD binary builds are the anchor tenant, the workload the whole thing was justified by: FreeBSD, OpenBSD, and NetBSD Arm64 binaries for the language compiler and the ballistics engine, each build a pod that boots a real BSD, compiles, and exits. Zero footprint between releases; real cores during them.

A full observability stack is the second tenant, and it exists partly because a cluster you can't see is a cluster you can't trust. It's the standard Prometheus-and-Grafana pairing, deployed the k3s-native way, with node exporters on every board reporting not just CPU and memory and disk but the thing that actually matters on fanless Arm boards crammed with NVMe: temperature. SBCs throttle when they cook, and a build farm that silently halves its own clock speed under thermal load is a build farm that lies to you about why it's slow. Grafana lives at a name on my LAN — grafana.localnet, resolved by an Unbound server on a little FreeBSD box that also runs my DNS — behind an ingress controller carefully arranged not to collide with the web page one of the Pis was already serving on port eighty. The stack pins its stateful pieces to the CM5, the node with the two-terabyte drive and the least else to do. The board that was holding its breath now holds the metrics.

Continuous fuzzing of the compiler is the newest tenant, and the one that most makes me glad the cores were sitting there. The language project ships nine libFuzzer harnesses — for the compiler front-end, three different bytecode VMs, the lexer, the formatter, and the JSON, TOML, and YAML parsers — and a fuzzer is the most gluttonous, most embarrassingly-parallel workload I own. It will eat every idle cycle you're willing to feed it and ask for more, forever, and every cycle it eats is a cycle spent trying to crash your code before a user does. On a cluster whose defining problem was too many idle cores, that is a nearly perfect fit. Each target gets its own continuous pod, spread across the three boards, building itself from the latest source and grinding against a corpus that persists and compounds on the NVMe — the first of the nine are coming online as I write this. And the results go somewhere I'll actually see them: when a fuzzer finds a crash, a small watcher reproduces it, fingerprints it so the same bug doesn't file itself twice, and opens a Jira ticket — the real bug tracker the project already uses — with the sanitizer backtrace and a base64 reproducer attached, tagged to the right component. Live throughput and coverage stream into the same Grafana the rest of the cluster reports to; a nightly job renders coverage as HTML behind that same ingress. Idle cores in; deduplicated, actionable bug reports out.

And underneath all of it, still, the Orange Pi's NPU runs a local language model when I ask it to — the original tenant, never evicted, just no longer the only one. Kubernetes shares the boards with it rather than owning them outright, which is exactly the relationship a homelab wants: the cluster is a tenant of the hardware, not its landlord.

The capacity was always there

There's a version of this post that's a shopping list — buy these boards, run these commands, get a cluster. I don't think that's the true version. The commands were the easy part; they always are. The hard part was the pile of small specific lies the hardware tells when you finally push it, and the honest bookkeeping of what's really running versus what's merely installed.

But the part I keep turning over isn't technical at all. It's that none of this required new hardware. Every core, every gigabyte, every terabyte of fast disk in this cluster was already sitting on my shelf, already powered, already mostly doing nothing. The Orange Pi was awake all night to compile software a few minutes a week. The CM5 was holding its breath. The utilization graph was a flat line near the floor, and I'd looked at it a hundred times without seeing it as a to-do item.

What changed wasn't the machines. It was the question I asked them. Three boards that each did one small thing became a single system that builds native binaries for three operating systems, watches its own temperature, and fuzzes a compiler around the clock — and the only thing I added was the willingness to treat idle capacity as a debt worth collecting.

The cluster was already there. It just hadn't been told yet.

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

A.C. Jokela — Thu, 18 Jun 2026 19:45:00 GMT

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

I have a Rust project — a ballistics engine — that ships binaries for an embarrassing number of platforms. Linux on x86 and ARM, Windows, FreeBSD, NetBSD, and, because someone always asks and because I like the discipline of it, OpenBSD. Most of those targets are easy: a build matrix, a cross-compiler, a Docker image with the right musl toolchain, and the artifacts fall out the other end. The BSDs are where the matrix stops being polite.

For FreeBSD and NetBSD on arm64 I already had a clean answer: spin up a real Graviton instance, build natively, collect the binary, terminate the instance. A build costs a few cents and finishes in minutes, because the hardware is real ARM and there's no emulation tax. I'd written a little orchestrator for exactly this — launch, build, tear down — and it worked beautifully for two of the three BSDs.

OpenBSD was the one that wouldn't fall in line. And the reason is so clean, so completely a consequence of what OpenBSD is, that it took me a while to stop treating it as a bug to be worked around and start treating it as a fact to be designed around.

This is the story of building native OpenBSD/arm64 binaries on a cloud that OpenBSD, in the most literal sense, cannot see.

Why native, and why that's suddenly hard

The honest first question is whether native builds matter at all. Couldn't I cross-compile OpenBSD/arm64 binaries from Linux and skip the whole circus?

For a trivial program, maybe. For anything real, no. Cross-compiling to OpenBSD means reproducing its libc, its linker behavior, its system call conventions, and its packaging quirks in a toolchain that lives somewhere else — and then trusting that the binary you produced behaves the way a binary built on OpenBSD would. The entire value of shipping an OpenBSD artifact is the claim "this was built and linked on the platform it runs on." A cross-compiled binary that mostly works is worse than no binary at all, because it fails in ways your users discover and you can't reproduce.

So: native. Build it on OpenBSD/arm64, on real ARM silicon, the way the platform's own users would.

The old way to get there was emulation — QEMU on an x86 host, software-emulating an ARM CPU. It works, and it's miserably slow: with no hardware virtualization to lean on, every guest instruction gets translated, and a four-minute build stretches to twenty-five. I did not want to go back.

The new way was supposed to be the Graviton instance: real ARM cores, native speed, terminate when done. It works for FreeBSD. It works for NetBSD. And then you try it with OpenBSD and walk straight into the wall.

The wall: OpenBSD has no driver for AWS's network card

Here is the entire problem, and it is almost funny once you see it.

Modern EC2 instances present their network to the guest through the Elastic Network Adapter — ena, a custom Amazon device with a custom driver. Every cloud-friendly OS ships that driver: Linux has it, FreeBSD has it, NetBSD's community AMIs have it. It is the thing that lets an instance have a network at all.

OpenBSD's arm64 kernel does not have an ena driver.

This isn't an oversight, and it isn't a slight. OpenBSD is a small, fiercely curated tree. Drivers get in because someone runs the hardware, writes clean code, and maintains it — and effectively nobody runs OpenBSD as a first-class EC2 guest, so the driver for Amazon's proprietary NIC simply isn't there. The project doesn't chase cloud hardware the way a commercial OS does, and that restraint is a feature, not a failing. But it has a consequence: an OpenBSD/arm64 instance on EC2 boots, comes up, and finds no network interface it understands. The NIC is right there on the PCI bus, and the kernel has nothing to bind to it.

An EC2 instance you cannot reach over the network is not an instance. It's a billable void. There's no SSH, no console you can usefully drive, no way to hand it a build and get a binary back. The OS runs perfectly; it just can't talk to the one piece of hardware that makes a cloud instance a thing you can use.

I sat with that for a bit. You can't add the driver from outside. You can't give OpenBSD a NIC it understands by configuring the instance differently, because AWS only offers ena. The platform and the OS disagree about reality at the level of the network card, and neither one is going to move.

So I stopped trying to make them agree.

Stop asking OpenBSD to talk to AWS

The move, once it arrived, was obvious in the way these things always are in retrospect: don't let OpenBSD touch the cloud's hardware at all.

Put a Linux host on the metal. Linux has the ena driver; it brings the network up without complaint. Then run OpenBSD as a virtual machine inside that Linux host, under QEMU with KVM acceleration — and hand the guest the kind of hardware OpenBSD does understand: virtio. OpenBSD has had solid virtio drivers for years. It sees a virtio network card and a virtio disk and is perfectly happy, never knowing or caring that the physical NIC underneath is an Amazon device it couldn't have spoken to directly.

The Linux host owns the ena and the public IP. The OpenBSD guest lives behind it on a private virtual network, reachable through a port forward. The two layers each get hardware they recognize, and the impedance mismatch — the missing driver — simply stops mattering, because the only thing that ever talks to the ena is Linux.

It's the same shape as a hundred other infrastructure problems: when two layers can't agree, you don't force them, you put a translator between them. Here the translator is an entire hypervisor, and the thing being translated is the basic question of what a network card is.

This also quietly solves the speed problem that made x86 emulation unbearable. Because the host is itself ARM, KVM can run the ARM guest at near-native speed — no instruction translation, just virtualization. OpenBSD/arm64 on an ARM host under KVM boots in seconds and builds at real-hardware pace. The thing that was slow about the old emulation approach was never "virtualization"; it was cross-architecture emulation. Match the architectures and the tax disappears.

Why it has to be bare metal

There's a catch, and it's the reason this costs more than a few cents.

KVM needs hardware virtualization extensions, and on AWS those are only exposed to the guest OS on bare-metal instances. On an ordinary virtualized Graviton instance, the Nitro hypervisor already owns the CPU's virtualization layer; your kernel can't open /dev/kvm because something else is already standing where KVM would stand. Try to accelerate QEMU on a normal instance and it silently falls back to the slow software path — right back to the emulation tax I was trying to escape.

So the host has to be .metal. The cheapest bare-metal Graviton AWS sells is a1.metal — first-generation Graviton, sixteen Cortex-A72 cores, and, crucially, a real /dev/kvm exposed to the host kernel. It's about forty-one cents an hour on demand. That's more than the few-cents-per-build the native FreeBSD path costs, but a build spins the instance up, runs, and terminates in fifteen or twenty minutes, so a release still costs well under a dollar.

I picked Ubuntu for the Linux host — its QEMU and UEFI firmware packages are exactly what the recipe wants — pointed it at the right ARM AMI, and gave it a roomy root volume. The host comes up, apt installs QEMU and the AAVMF UEFI firmware, and now I have an ARM machine with KVM, ready to boot an OpenBSD guest. That part was easy.

The part that ate an afternoon was getting OpenBSD to actually attach its own disks.

Two device-tree gotchas that ate an afternoon

QEMU's generic ARM machine type, virt, gives you a clean, modern, device-tree-described board. You hand it virtio devices and it wires them up. I did the obvious thing — a virtio block device for the installer image, a virtio block device for the target disk, a virtio network device — booted, and watched the OpenBSD installer come up, reach the disk-selection step, and announce:

Available disks are: none.

No disks. Also, quietly, no network interface. Both virtio devices were there — UEFI had even booted the installer kernel off one of them — and yet the running OpenBSD kernel saw neither.

The cause is a transport mismatch that's easy to miss. On the virt machine, virtio-blk-device and virtio-net-device use the virtio-MMIO transport — devices described in the device tree, sitting at fixed memory addresses. UEFI can boot from those. OpenBSD's arm64 kernel, it turns out, does not attach them. What it does attach is virtio-PCI — the same virtio devices presented as PCI devices on the PCIe controller the kernel already probes. Swap virtio-blk-device for virtio-blk-pci and virtio-net-device for virtio-net-pci, and suddenly the disks appear as sd0 and sd1 and the NIC as vio0. One word per device, and the installer can see the world.

That fixed the disks. The very next boot hung.

Not crashed — hung. The kernel probed all the devices, printed them, and then stopped cold, right before mounting its root filesystem. The host's load average sat at zero: the guest CPU wasn't spinning, it was blocked, waiting for something that was never going to come. The something was an interrupt.

virtio-PCI devices default to MSI-X — message-signaled interrupts, delivered on ARM through the GIC's Interrupt Translation Service. On this first-generation Graviton host, under KVM, that MSI path doesn't deliver. The guest arms its interrupts, waits for the first one, and waits forever. The fix is to tell QEMU to give each virtio device zero MSI-X vectors, forcing it back to plain old pin-based (INTx) interrupts, which ride the GIC's distributor directly and work fine:

-device virtio-blk-pci,drive=hd0,bootindex=1,vectors=0
-device virtio-net-pci,netdev=net0,vectors=0

With vectors=0 on every virtio device, the device line in the boot log changes from msix to a plain irq, the kernel gets its interrupt, mounts root, and reaches the installer prompt. Two one-line changes — the PCI transport and the disabled MSI-X — separated by an afternoon of staring at a boot that stopped in exactly the same place every time.

Neither of these is in the friendly "run OpenBSD under QEMU" recipes you'll find, because those recipes assume a host where MSI-X works and where someone already knew to use the PCI transport. The combination of this OS on this generation of silicon under this hypervisor is niche enough that you find the edges yourself.

Installing an OS through a serial straw

The virt machine has no graphics — no VGA, no framebuffer worth the name. OpenBSD's installer comes up on the serial console, which QEMU helpfully wires to standard out. That's fine for a human at a keyboard. It's less fine when you're trying to make the whole thing repeatable, because the OpenBSD installer is a long, interactive conversation: hostname, network, disk, sets, passwords, two dozen prompts.

OpenBSD has an answer for this — autoinstall, where a response file answers every prompt and the install runs unattended. I tried it first, serving the response file over HTTP from the Linux host. It failed in an instructive way: QEMU's built-in user-mode network doesn't advertise an autoinstall server in its DHCP lease, and when the installer falls back to asking where the response file is, it tries to fetch it before the guest's network is actually up. The fetch fails, autoinstall aborts, and you're dropped to a shell. The unattended path assumes a network environment QEMU's toy DHCP doesn't provide.

So the install came down to driving the interactive program by hand — except I didn't drive it. I let Claude drive, programmatically: QEMU running inside a tmux session, the agent feeding the serial console one answer at a time and reading the screen back between each step. It's a strange way to install an operating system — an AI typing into a virtual serial port through a terminal multiplexer over SSH — and it fumbled in telling ways. It stuffed answers ahead of the prompts and desynced, at one point confidently typing vio0 into the DNS domain name field because it had guessed the question order and the installer disagreed; a mismatched password pair followed, and I watched it back out and resynchronize. Once it was reading each prompt instead of predicting it, the rest was deterministic, and it pulled the install sets straight off the installer image so the guest never needed working DNS at all. Whole-disk GPT, auto layout, sets from the local media, and a few minutes later: CONGRATULATIONS! Your OpenBSD install has been successfully completed!

Drop the installer image, reboot from the disk alone, and the installed system comes up on its own — generates its host keys, starts sshd, reaches a login prompt. uname confirms it: OpenBSD 7.9, arm64, on a "QEMU KVM Virtual Machine." A real, persistent OpenBSD/arm64 system, running on a cloud that can't host it directly.

Making it ephemeral: the golden image and the one-button build

A hand-installed VM is a proof of concept, not a build server. The point was always to make this disappear into a command — spin the whole stack up, build, collect the artifact, spin it down, and pay pennies.

So I did the slow part exactly once. I took that freshly installed guest, added the toolchain a build needs — git, the Rust compiler — and a build key whose public half lives in the image's authorized_keys. Then I shut it down cleanly and compressed the disk: a 20 GB virtual volume holding maybe two gigs of real data squeezed down to an 832 MB qcow2. That golden image went into S3, in the same region as the metal hosts, so pulling it back down is fast and free.

Around it I wrote an orchestrator — the OpenBSD sibling of the FreeBSD/NetBSD tool I already had. One command does the whole dance: launch an a1.metal, wait for it, install QEMU, pull the golden image from S3 through a presigned URL (so the host needs no AWS credentials of its own), boot the OpenBSD guest with the exact virtio-PCI-and-vectors=0 incantation the afternoon taught me, and wait for the guest's SSH to answer. The guest is reachable only through a port forward on the host, so the orchestrator jumps through the host to reach it — the build key never leaves my laptop, and no extra port is ever opened to the world. Then it pipes the build into the guest, runs it natively, copies the binary back out, checksums it, and — in a finally block, so it happens whether the build succeeds, fails, or throws — terminates the metal host.

Spin up, build, collect, spin down. The instance exists for the length of one build and then it's gone, and the bill is loose change.

The last wall: the build that wouldn't build

I ran the orchestrator end to end, and it was beautiful right up until the part that mattered. It launched the host, pulled the image, booted OpenBSD, SSH'd in, started the build — and the build died compiling a TLS library, deep in my engine's dependency tree.

I knew exactly what it was. Or I thought I did. My ballistics engine has an optional "online" feature, on by default, that pulls in an HTTP client and with it a whole TLS stack — rustls, the ring crypto backend, the works. Of course that was the thing that wouldn't build on an OS this far off the beaten path. I turned the feature off, the build went green, and I jotted down a tidy little lesson about exotic platforms and the liabilities of convenient defaults.

The lesson was wrong, and I found out a few hours later. The release binary that already ships for this exact platform has that entire TLS stack compiled into it — rustls builds on OpenBSD/arm64 perfectly well. So that was never the wall. The real wall was disk. OpenBSD's installer auto-layout splits the disk into a fistful of small partitions, and the build was running in /tmp — about a gigabyte. A Rust release target/ directory does not fit in a gigabyte, and rustls happens to be a large crate, so the partition filled mid-compile and the compiler fell over with an error that looked like a code problem. Turning off the feature "fixed" it only because a leaner build squeaked under the limit. The actual fix was one line — build in /usr/obj, the roomy partition OpenBSD hands you for exactly this — and with room to work, the full build, TLS and all, finishes fine.

Here's the part that stings, and the reason I'm telling it on myself. I am not new to this. I've dabbled in the BSDs for a long time — FreeBSD was my daily driver in the early 2000s — and I know OpenBSD well enough that its installer holds no surprises; it has barely changed in twenty years. "OpenBSD carves the disk into small partitions and you do your building in /usr/obj" is not arcana to me, it's reflex. A version of me at the keyboard would have read compiler died, OpenBSD, /tmp and smelled out of disk before finishing the line. But I wasn't at the keyboard — I'd handed it to the same agent that, a few screens earlier, had typed vio0 into the DNS field. An agent has read about OpenBSD without ever once running out of disk on it, so it had no instinct for the boring failure. It reached instead for the exotic one sitting right there in the dependency graph — the explanation that was plausible, pre-loaded, and a little flattering, because if the crypto stack won't compile you're wrestling something genuinely hard, rather than something you'd have caught in 2003.

And that is the trap worth naming, sharper than "niche platforms are hard." On an unfamiliar system every failure presents as exotic; the strange explanation announces itself and the mundane one doesn't. The further off the beaten path you are, the more deliberately you have to rule out the boring thing first — because out here everything wears the platform's costume, and an out-of-disk error in a one-gigabyte /tmp looks exactly like a dependency that can't be built, right up until someone who's been burned before tells you it's just disk.

What it actually was about

Step back and the whole project rhymes with a pattern I keep meeting. The headline problem — "build OpenBSD/arm64 binaries on AWS" — has a one-sentence answer once you have it: run OpenBSD as a KVM guest on a bare-metal Graviton host. That sentence is the easy part. Everything real lived in the gap between two systems that were each, on their own terms, completely correct.

OpenBSD is correct to omit a driver for hardware almost none of its users run; its restraint is exactly why it's OpenBSD. AWS is correct to present its network through a custom adapter built for the scale it operates at. Neither is wrong. They simply grew up with no expectation of meeting, and the moment you ask them to share a machine, every quiet assumption each one made surfaces at once — the missing driver, the MMIO-versus-PCI transport, the MSI-X interrupts that don't land on first-gen silicon, the DHCP lease that doesn't carry what the installer hoped, the cramped /tmp masquerading as a dependency that wouldn't compile.

The hypervisor in the middle isn't really there to virtualize a CPU. It's there to be the place where two designs that disagree about what hardware is can both be right at the same time. OpenBSD gets virtio and is happy. AWS gets ena and is happy. The seam between them holds because something whose entire job is to translate sits exactly on it.

And the build server that came out the other end is genuinely good: a single command that conjures an ARM machine, runs a native OpenBSD build at full speed, and dissolves the machine when it's done, for the price of a vending-machine snack. The operating system that couldn't see the network now turns out release binaries on the very cloud it can't directly run on — because at no point does anyone ask it to see the network. That was never going to work. The trick was to stop asking.

JokelaOS: Writing a Bare-Metal x86 Kernel from Scratch

A.C. Jokela — Tue, 10 Mar 2026 15:00:00 GMT

There's a moment early in any OS project where the serial port prints its first character and you realize that nothing you've written has a safety net. No libc. No kernel underneath. No syscall to fall back on. If the byte appears on the terminal, it's because you programmed the UART divisor latch, polled the line status register, and wrote to the data port. If it doesn't appear, you stare at register dumps until you find the mistake. There's no debugger; you haven't written one yet.

The closest thing I can compare it to is the first time I got a RetroShield Z80 talking over serial, that moment where a processor you wired up yourself pushes a character out of an emulated ACIA and it appears on your screen. The Z80 version involves physical hardware and solder. The x86 version is virtual (QEMU, a cross-compiler, and a Multiboot header), but the feeling is the same. You built the entire path from CPU to character. Nothing was given to you.

JokelaOS started there: a Multiboot header, a stack, and a call kmain. Everything that followed (GDT (Global Descriptor Table), IDT (Interrupt Descriptor Table), memory management, a network stack, preemptive multitasking, paging, user mode, a shell) was built one subsystem at a time, tested after every change, with no external code. No forks of existing kernels. No libc.

To be clear about what this is: JokelaOS is a toy. It's a learning project. The memory allocator is a linear scan. The scheduler has no concept of priority. The file system can't delete files. The user authentication stores passwords in plaintext in a static array. Nothing here is production-grade, and none of it is intended to be. The value is in the building: understanding what each subsystem actually does by writing it from scratch, making the mistakes, and fixing them with nothing between you and the hardware.

This is the story of what it takes to go from twenty lines of NASM to a kernel that boots, manages memory, runs user programs in Ring 3, handles syscalls, responds to pings, and gives you a command prompt.

The Target

JokelaOS targets 32-bit x86 (i686) and runs under QEMU. The toolchain is a cross-compiler (i686-elf-gcc, i686-elf-ld) with NASM for the assembly files. The C standard is gnu11; GNU extensions are required for inline assembly. There are no external libraries whatsoever, not even a freestanding string.h. Every memcpy, every memset, every printf-like function is written from scratch.

The only console is the serial port. COM1 at 0x3F8, 115200 baud, 8N1 (8 data bits, no parity, 1 stop bit). All kernel output goes through serial_printf(). This is a deliberate choice: serial is simpler than VGA text mode, works perfectly with QEMU's -serial stdio, and means the kernel's output appears directly in the host terminal. No framebuffer driver needed, no font rendering, no cursor management. Just bytes on a wire.

$ make run
qemu-system-i386 -kernel build/jokelaos.bin -serial stdio \
    -display none -device rtl8139,netdev=net0 \
    -netdev user,id=net0 -no-reboot

Kernel Architecture

JokelaOS is monolithic: everything runs in Ring 0, in one address space. When the network stack needs a page, it calls pmm_alloc_frame() directly. When the shell loads a program, the call chain goes through the loader, the PMM, and the paging subsystem without ever crossing an address space boundary. The trade-off is that a bug in the RTL8139 driver can corrupt the process table, and a buffer overrun in the serial handler can overwrite page tables. In a toy kernel written by one person, bugs are spectacular.

A microkernel would isolate those failures, but it would also triple the code before you could print a single character. You'd need working IPC before the serial driver could talk to anything. JokelaOS is monolithic because it's the simplest architecture to build and the easiest to debug: serial_printf() anywhere can see everything.

Booting: The First 33 Lines

The entire boot sequence fits in boot.asm. Multiboot v1 requires a magic number (0x1BADB002), flags, and a checksum in a specific header format. GRUB or QEMU's -kernel loader scans for this header, loads the binary, and jumps to _start in protected mode with paging disabled.

section .multiboot
align 4
    dd 0x1BADB002                           ; Multiboot magic
    dd 0x00000003                           ; Flags: page-align + memory map
    dd -(0x1BADB002 + 0x00000003)           ; Checksum

section .text
global _start
extern kmain

_start:
    mov esp, stack_top
    push 0
    popf                    ; Clear EFLAGS
    push ebx                ; Multiboot info struct pointer
    push eax                ; Multiboot magic number
    call kmain
    cli
.hang:
    hlt
    jmp .hang

That's it. Set up a stack, clear the flags register, push the two values the Multiboot spec guarantees (magic number in EAX, info struct pointer in EBX), and call C. If kmain ever returns, disable interrupts and halt forever.

The 16 KB stack is allocated in the BSS section, zeroed at load time. The linker script places the kernel at 1 MB (the standard x86 protected-mode load address), with .multiboot first so the bootloader can find the header within the first 8 KB of the binary.

Protection Rings: Hardware-Enforced Privilege

x86 protected mode provides four privilege levels, numbered 0 through 3, called rings. Ring 0 is the most privileged: the kernel runs here. Ring 3 is the least privileged: user programs run here. Rings 1 and 2 exist in the hardware but almost nobody uses them. Linux doesn't. Windows doesn't. JokelaOS doesn't. The practical x86 privilege model is two rings: kernel and user.

The ring system isn't a software convention. It's enforced by the CPU itself, in silicon. The processor tracks the Current Privilege Level (CPL), the ring the currently executing code belongs to, and checks it against every sensitive operation. A Ring 3 process that executes cli (disable interrupts), hlt (halt the CPU), lgdt (load a new GDT), or mov cr3 (change the page directory) triggers a General Protection Fault. The CPU literally refuses to execute the instruction. A Ring 3 process can't touch I/O ports unless the kernel has explicitly granted access through the I/O Permission Bitmap in the TSS. It can't modify its own segment registers to escalate privilege, because the CPU validates every segment load against the descriptor's DPL (Descriptor Privilege Level).

The only way for Ring 3 code to enter Ring 0 is through a gate: an interrupt gate, a trap gate, or a call gate. Gates are entries in the IDT or GDT that the kernel sets up in advance. They define the exact entry points where Ring 3 code can cross into Ring 0, what the new code and stack segments will be, and what privilege level is required to use them. There's no way for user code to jump to an arbitrary kernel address. It can only enter the kernel through the doors the kernel has built.

This is what makes an operating system an operating system rather than a library. Without ring separation, a buggy user program can corrupt kernel memory, disable interrupts, reprogram the PIC, or overwrite the page tables. With ring separation, the worst it can do is crash itself.

The mechanism that implements all of this is the Global Descriptor Table.

The GDT: Defining the World

The GDT defines memory segments: their base addresses, sizes, privilege levels, and whether they hold code or data. Each segment descriptor is an 8-byte structure with fields packed into non-obvious bit positions (a consequence of backward compatibility with the 286, which had a different descriptor format that the 386 had to extend without breaking).

JokelaOS uses a flat memory model: every segment covers the full 4 GB address space with base 0 and limit 0xFFFFFFFF. The segmentation hardware is effectively nullified, which is what you want on modern x86 where paging handles memory protection. But the GDT is still mandatory; the CPU requires it for the ring system to function. Even with flat segments, the DPL field in each descriptor is what tells the CPU "code using this segment is Ring 0" or "code using this segment is Ring 3."

The GDT has six entries:

Index	Selector	Purpose
0	0x00	Null descriptor (required by x86)
1	0x08	Kernel code (Ring 0, execute/read)
2	0x10	Kernel data (Ring 0, read/write)
3	0x18	User code (Ring 3, execute/read)
4	0x20	User data (Ring 3, read/write)
5	0x28	Task State Segment

Entries 1 and 2 are identical to entries 3 and 4 in every way except the DPL field: two bits in the access byte that say 00 (Ring 0) versus 11 (Ring 3). That two-bit difference is the entire kernel/user boundary.

When a user process runs, the CPU's CS register is loaded with 0x1B; that's selector 0x18 (pointing to GDT entry 3, the user code segment) OR'd with RPL 3 (the bottom two bits of the selector). The data segment registers get 0x23 (GDT entry 4, user data, RPL 3). The CPU sets CPL to match, and from that point on, every instruction is checked against Ring 3 privileges. The kernel runs with CS=0x08 (GDT entry 1, RPL 0) and DS=0x10 (GDT entry 2, RPL 0).

The TSS (Task State Segment) is the bridge between rings. When the CPU takes an interrupt while running Ring 3 code, it needs to switch to a Ring 0 stack, because you can't trust the user's stack pointer to be valid, and you certainly can't run kernel interrupt handlers on a user-controlled stack. The TSS holds the Ring 0 stack pointer (esp0). Every context switch updates the TSS with the current process's kernel stack, so the CPU always knows where to land when transitioning from user mode to kernel mode.

void gdt_init(void) {
    gdt_set_entry(0, 0, 0, 0, 0);                     // Null
    gdt_set_entry(1, 0, 0xFFFFFFFF, 0x9A, 0xCF);      // Kernel code
    gdt_set_entry(2, 0, 0xFFFFFFFF, 0x92, 0xCF);      // Kernel data
    gdt_set_entry(3, 0, 0xFFFFFFFF, 0xFA, 0xCF);      // User code
    gdt_set_entry(4, 0, 0xFFFFFFFF, 0xF2, 0xCF);      // User data
    // TSS entry built separately
    ...
}

The access byte 0x9A means: present, Ring 0, code segment, executable, readable. 0xFA means the same thing but Ring 3. These magic numbers come straight from the Intel manuals and they're the kind of thing you get wrong three times before you get right once.

Interrupts: Exceptions, IRQs, and the PIC

The IDT maps interrupt vectors to handler functions. JokelaOS sets up 256 entries: CPU exceptions (0-31), hardware IRQs (32-47), and the syscall gate (0x80).

The x86 PIC needs remapping. By default, the master PIC maps IRQs 0-7 to interrupt vectors 8-15, which collide with CPU exceptions (double fault is vector 8, for instance). The standard fix is to remap the master PIC to vectors 32-39 and the slave to 40-47. This requires sending four Initialization Command Words to each PIC in the correct sequence, the kind of hardware protocol that hasn't changed since the IBM PC/AT in 1984.

ISR stubs are written in NASM. Each one pushes an error code (or a dummy zero for exceptions that don't push one), pushes the interrupt number, saves all general-purpose registers, calls the C handler, restores registers, and does an iret. The stubs are generated with macros:

%macro ISR_NOERRCODE 1
global isr%1
isr%1:
    push dword 0        ; dummy error code
    push dword %1       ; interrupt number
    jmp isr_common
%endmacro

The C-side dispatcher checks the interrupt number. For exceptions (0-31), it prints the register state and halts, since there's no recovery from a page fault when you don't have a page fault handler yet. For IRQs (32-47), it calls the registered handler function and sends an EOI command to the PIC. For interrupt 0x80, it dispatches to the syscall handler.

One critical detail: interrupt 0x80 is set as a trap gate with DPL 3, not an interrupt gate. This means Ring 3 code can trigger it with int 0x80. All other interrupt gates are DPL 0, so a user program that tries to execute int 0x00 gets a General Protection Fault instead. This is the mechanism that makes syscalls work while keeping everything else protected.

Memory: Three Allocators

JokelaOS has three layers of memory management, each built on top of the previous one.

The Bump Allocator

The simplest possible allocator. A pointer starts at the first page boundary after the kernel image (_kernel_end from the linker script) and only moves forward. kmalloc(size) aligns the pointer to 16 bytes, returns it, and advances by size. There is no kfree(). Memory allocated with the bump allocator is permanent.

This sounds primitive, and it is. But it's also exactly right for kernel initialization. The GDT, IDT, page tables, file system metadata, user table; these are allocated once and never freed. The bump allocator handles all of them with zero fragmentation and zero overhead.

The Physical Memory Manager

Once the kernel needs to allocate and free pages dynamically (for process stacks, program code, page tables), it needs a real allocator. The PMM uses a bitmap: one bit per 4 KB physical frame, supporting up to 256 MB of RAM (65,536 frames, 8 KB bitmap).

Initialization parses the Multiboot memory map to find usable RAM regions, then marks everything from frame 0 through the end of the bump heap as reserved. This protects the IVT, BIOS data area, kernel image, and all bump-allocated structures from being handed out as free pages.

uint32_t pmm_alloc_frame(void) {
    for (uint32_t i = 0; i < total_frames; i++) {
        if (!(bitmap[i / 8] & (1 << (i % 8)))) {
            bitmap[i / 8] |= (1 << (i % 8));
            free_count--;
            return i * PAGE_SIZE;
        }
    }
    return 0;  // out of memory
}

Linear scan, no free lists, no buddy system. It's O(n) per allocation, which is fine when n is measured in thousands and allocations are infrequent. A production kernel would use something smarter. This kernel allocates a few dozen pages total.

Paging

With physical frames available, the kernel can enable paging. paging_init() builds a page directory and 32 page tables, identity-mapping the first 128 MB of physical memory (virtual address = physical address). The page directory goes into CR3, and setting the PG bit in CR0 turns the MMU on.

Identity mapping means the kernel doesn't need to worry about virtual-to-physical translation for its own code and data. Kernel pointers just work. When user processes need memory, the loader allocates physical frames and maps them into the process's address space with the PG_USER flag set, allowing Ring 3 access.

void paging_map_page(uint32_t virt, uint32_t phys, uint32_t flags) {
    uint32_t dir_idx = virt >> 22;
    uint32_t tbl_idx = (virt >> 12) & 0x3FF;

    if (!(page_directory[dir_idx] & PG_PRESENT)) {
        uint32_t tbl_frame = pmm_alloc_frame();
        memset((void *)tbl_frame, 0, PAGE_SIZE);
        page_directory[dir_idx] = tbl_frame | PG_PRESENT | PG_WRITE | flags;
    }

    uint32_t *table = (uint32_t *)(page_directory[dir_idx] & 0xFFFFF000);
    table[tbl_idx] = (phys & 0xFFFFF000) | flags;

    asm volatile("invlpg (%0)" : : "r"(virt) : "memory");
}

The invlpg instruction flushes the TLB entry for the mapped virtual address, which is critical. Without it, the CPU might use a stale translation from its cache and access the wrong physical page.

The Network Stack

JokelaOS has a working network stack, the one subsystem where "toy" undersells it slightly. It resolves ARP, constructs IPv4 packets with correct checksums, and handles ICMP echo request/reply with measured round-trip times. There's no TCP, no UDP, no sockets. But the packets that leave this kernel are real packets that traverse real networks.

The NIC is an emulated RTL8139, the simplest PCI Ethernet controller that QEMU supports. The driver initializes the chip by writing to its configuration registers: reset, enable transmitter and receiver, set up a receive ring buffer, configure the interrupt mask, and unmask IRQ 11. Packet transmission uses a four-descriptor TX ring; reception is interrupt-driven through the RTL8139's ring buffer.

PCI enumeration scans the configuration space to find the RTL8139 by vendor/device ID (0x10EC:0x8139), reads the I/O base address from BAR0, and enables bus mastering. This is the only driver in the system; there's no USB, no disk, no display. One NIC, one network.

The stack is layered:

Layer	Module	Purpose
Link	`ethernet.c`	Frame demux by EtherType
ARP	`arp.c`	Table + request/reply
Network	`ipv4.c`	Routing, header checksum
Transport	`icmp.c`	Echo reply + outgoing ping

On boot, the kernel sends an ARP request for the gateway (10.0.2.2, QEMU's default) and waits for the reply. Once the gateway's MAC address is resolved, the kernel can ping arbitrary hosts through QEMU's SLIRP NAT. A ping 10.1.1.1 from the shell constructs an ICMP echo request, wraps it in an IPv4 packet, wraps that in an Ethernet frame, and pushes it out through the RTL8139's TX ring. When the reply comes back, the receive ISR fires, the Ethernet layer demuxes by EtherType, the IP layer validates the checksum, and the ICMP handler matches the echo reply to the outstanding request and computes the RTT.

root$ ping 10.1.1.1
[ping] Pinging 10.1.1.1...
  64 bytes from 10.1.1.1: seq=1 time=4 ms
  64 bytes from 10.1.1.1: seq=2 time=2 ms
  64 bytes from 10.1.1.1: seq=3 time=3 ms
  64 bytes from 10.1.1.1: seq=4 time=2 ms

Getting here required writing every byte-order conversion (htons, htonl), every checksum computation (the IP header checksum is a one's complement sum of 16-bit words), every packet layout (Ethernet header is 14 bytes, IP header is 20, ICMP is 8 plus payload). None of this is hard individually. Together, it's a thousand places to put a byte in the wrong order.

Processes and Preemptive Multitasking

The process subsystem manages up to 16 processes in a static table. Each process has a state (UNUSED, READY, RUNNING, DEAD), a kernel stack pointer, and a user-mode entry point and stack.

Process creation doesn't follow the UNIX fork()/exec() model. There's no cloning of address spaces, no copy-on-write, no replacing the current process image. Instead, the loader allocates fresh physical frames for the program's code and stack, copies the flat binary into the code pages, and calls proc_create(), which allocates a 4 KB kernel stack and builds a fake stack frame on it. This stack frame is what context_switch() will "return" into on the process's first schedule; it contains saved registers and a return address pointing to proc_entry_user().

proc_entry_user() is a small assembly sequence that performs the Ring 0 to Ring 3 transition. It sets the data segment registers to the user data selector (0x23), pushes a fake interrupt frame (SS, ESP, EFLAGS with IF=1, CS, EIP), and executes iret. The CPU pops the frame, switches to Ring 3, and starts executing the user program. From the hardware's perspective, this looks identical to returning from an interrupt that happened to interrupt a user-mode program, which is exactly the trick.

static void proc_entry_user(void) {
    process_t *p = proc_current();
    asm volatile(
        "mov $0x23, %%ax \n"
        "mov %%ax, %%ds  \n"
        "mov %%ax, %%es  \n"
        "mov %%ax, %%fs  \n"
        "mov %%ax, %%gs  \n"
        "push $0x23      \n"   // SS
        "push %0         \n"   // ESP
        "pushf           \n"
        "pop %%eax       \n"
        "or $0x200, %%eax\n"   // Set IF
        "push %%eax      \n"   // EFLAGS
        "push $0x1B      \n"   // CS (user code)
        "push %1         \n"   // EIP
        "iret"
        : : "r"(p->user_esp), "r"(p->user_eip)
        : "eax", "memory"
    );
}

Context switching uses a simple assembly stub in switch.asm. It saves the callee-saved registers (EBP, EBX, ESI, EDI), stores ESP into the old process's slot, loads the new process's ESP, restores registers, and returns. The ret instruction pops the return address from the new stack and resumes where that process left off.

Scheduling is preemptive round-robin. The PIT fires at 1000 Hz. Every 10 ticks (10 ms), the IRQ handler calls proc_schedule(), which finds the next READY process and switches to it. If no user processes are ready, control stays with PID 0 (the kernel/shell). This is the minimum viable scheduler: no priorities, no time slices, no fairness guarantees. But it works: two user programs printing characters to serial run concurrently, interleaved by the timer.

Syscalls

User programs communicate with the kernel through int 0x80. The mechanism, a software interrupt that transitions from Ring 3 to Ring 0, is the same one Linux used on i386 before sysenter replaced it. The register convention is borrowed too: syscall number in EAX, arguments in EBX/ECX/EDX/ESI/EDI, return value in EAX. But that's where the resemblance ends.

JokelaOS is not a UNIX. The syscall numbers are custom (exit is 0, write is 1, getpid is 2, read is 3), not Linux's i386 table (where exit is 1, read is 3, write is 4, getpid is 20). There's no fork(), no exec(), no open(), no close(), no signals, no pipes. File descriptors 0 and 1 exist as concepts (stdin maps to the keyboard buffer, stdout maps to the serial port) but there's no file descriptor table behind them. The syscall handler just checks if (fd == 1) and calls serial_putchar(). The process model isn't UNIX either; there's no parent/child relationship, no wait(), no process groups. Processes are created by the loader and scheduled round-robin until they exit. It's closer to a microcontroller RTOS than to anything in the UNIX lineage.

Four syscalls are implemented:

Number	Name	Arguments	Description
0	SYS_EXIT	ebx=status	Terminate process
1	SYS_WRITE	ebx=fd, ecx=buf, edx=len	Write to serial (fd=1)
2	SYS_GETPID	-	Return current PID
3	SYS_READ	ebx=fd, ecx=buf, edx=len	Read from keyboard (fd=0)

This is enough to write programs that print output, read input, identify themselves, and exit cleanly. The syscall dispatcher validates file descriptors (only 0 and 1 are legal) and bounds-checks lengths. SYS_WRITE sends bytes to the serial port; SYS_READ drains the keyboard buffer non-blocking.

User programs are flat binaries: raw machine code with no headers, no relocations, no ELF parsing. The loader copies the binary to freshly allocated pages and jumps to byte zero. Programs that need to reference their own data use position-independent tricks:

call next       ; push EIP
next:
pop ebp         ; EBP = address of this instruction
lea ecx, [ebp + offset_to_data]

This is the same technique used by shellcode and position-independent code on x86. It works because call pushes the address of the next instruction, which gives you a known reference point relative to the code's actual load address.

The Shell

With all the subsystems in place, the shell ties them together into something interactive. shell_run() is the kernel's main loop after initialization. It presents a login prompt, authenticates against the user table, and drops into a command interpreter.

==============================
  JokelaOS v0.1
==============================

[ok] GDT loaded (ring 0 + ring 3 + TSS)
[ok] IDT loaded, PIC remapped
[ok] Multiboot confirmed
[ok] Multiboot info at 0x9500
[ok] Bump allocator ready

PCI 00:03.0 - vendor 10EC device 8139 (RTL8139)
[ok] RTL8139 ready, MAC=52:54:00:12:34:56, IRQ=11

[ok] ramfs: 2 files
[ok] Users: root, guest
[ok] PMM: 31269 free frames (122 MB)
[ok] Paging enabled (128 MB identity-mapped)
[ok] PIT timer @ 1000 Hz
[ok] Keyboard + serial input ready

JokelaOS is alive.

login: root
password: ****
root$

The shell supports: help, ls, run <program>, ps, mem, ping <ip>, uptime, whoami, and logout. The line editor handles backspace. Password input echoes asterisks. The run command loads a flat binary from ramfs, creates a process, and the scheduler picks it up on the next timer tick.

ps shows the process table:

root$ ps
  PID  STATE
    0  RUNNING
    1  READY
    2  DEAD

mem shows memory usage:

root$ mem
Heap used: 8832 bytes
PMM free:  31267 frames (122 MB)

The keyboard input path is worth noting. The PS/2 keyboard controller fires IRQ 1. The handler reads the scancode from port 0x60, converts it to ASCII using a US QWERTY lookup table (with shift modifier tracking), and drops it into a 256-byte circular buffer. Serial input takes the same path; the UART's receive interrupt (IRQ 4) reads the incoming byte and injects it into the keyboard buffer. This means the shell works identically whether you're typing on a PS/2 keyboard or through the QEMU serial console.

The RAM File System

User programs need to live somewhere. With no disk driver, the file system is purely in-memory. ramfs stores up to 32 files, each with a name (28 bytes), a data pointer, and a size. ramfs_create() allocates space with the bump allocator and copies the binary in. ramfs_find() does a linear search by name.

During boot, two test programs are embedded directly in kmain.c as byte arrays of hand-assembled x86 machine code. One prints the character '1' ten times; the other prints '2' ten times. Both use SYS_WRITE to output through the serial port and SYS_EXIT to terminate cleanly. They're loaded into ramfs, and run print1 from the shell executes them in user mode.

This is about as minimal as a file system gets. No directories, no permissions, no deletion. But it demonstrates the complete path from "bytes in kernel memory" to "user-mode process executing with its own address space."

What I Learned

The boot process is the hardest part. Not because the code is complex (boot.asm is 33 lines), but because when something goes wrong, you have zero diagnostic capability. The serial port isn't initialized yet. The IDT isn't loaded. If your Multiboot header checksum is wrong by one bit, QEMU silently fails. You're debugging with QEMU's -d int flag and reading hex dumps of interrupt frames.

x86 protected mode is an archaeology project. The PIC remapping sequence dates from the IBM PC/AT (1984). The GDT access bytes encode information in bit patterns designed for hardware that predates flat memory models. The TSS exists because Intel's original vision for the 286 involved hardware task switching that nobody ended up using. You're programming against forty years of backward compatibility, and every one of those layers is still there, still mandatory, still silently breaking things if you get it wrong.

The gap between "works in Ring 0" and "works in Ring 3" is enormous. A kernel that runs entirely in supervisor mode can be surprisingly simple. The moment you add user mode, you need: the TSS (so the CPU knows where the kernel stack is), Ring 3 GDT segments, trap gates for syscalls, a mechanism to build fake interrupt frames for the initial iret into user mode, and careful validation of every pointer that crosses the kernel boundary. Each of these is individually straightforward. Getting them all correct simultaneously is not.

Preemptive scheduling is simpler than it sounds. The concept (save state, pick next process, restore state) translates almost directly into code. The context switch is twelve instructions of assembly. The scheduler is a for loop. What makes it tricky is the interaction with everything else: the TSS must be updated, the interrupt must send EOI before switching, the process's kernel stack must be set up so that restoring registers and returning lands in the right place. The scheduler itself is trivial. The invariants it depends on are not.

Writing a network stack is an exercise in byte ordering. Ethernet is big-endian. x86 is little-endian. IP addresses, port numbers, checksums, packet lengths: every multi-byte field requires explicit conversion. Miss one htons() and your packets are valid-looking garbage. The RTL8139 driver, the ARP implementation, the IP checksum; each is maybe fifty lines. The debugging when a byte is swapped is hours.

The Numbers

JokelaOS in its current form:

Component	Files	Approximate LOC
Boot (ASM)	3	~120
Kernel core	16	~1,200
Drivers	2	~250
Network stack	5	~450
Total	26	~2,000

Two thousand lines for a kernel that boots, manages memory with paging, runs preemptive multitasking with Ring 3 isolation, handles interrupts, implements syscalls, has a working network stack, and provides an interactive shell. No line is borrowed from another project. Every byte is accounted for.

The entire thing builds in under a second and the binary is around 40 KB. make run goes from source to a running kernel in QEMU in about two seconds. This fast iteration cycle is what made the project possible; every subsystem was tested immediately after being written, and bugs were caught before they could compound.

What's Next

The point of JokelaOS was never to build a production operating system. The point was to understand what an operating system actually does: not in the abstract, not from a textbook diagram, but in the specific, concrete sense of "these bytes go into these ports in this order and then the hardware does this thing." Every subsystem in JokelaOS exists because I wanted to understand it, and the only way to truly understand a piece of systems software is to write it yourself.

The source code is on GitHub.

Building Cross-Platform Rust Binaries: A Multi-Architecture Build Orchestration System

A.C. Jokela — Fri, 21 Nov 2025 21:19:35 GMT

When developing ballistics-engine, a high-performance ballistics calculation library written in Rust, I faced a challenge: how do I efficiently build and distribute binaries for multiple operating systems and architectures? The answer led to the creation of an automated build orchestration system that leverages diverse hardware (from single-board computers to powerful x86_64 servers) to build native binaries for macOS, Linux, FreeBSD, NetBSD, and OpenBSD across both ARM64 and x86_64 architectures. Now, you are probably wondering why I am bothering to show love for the BSD Trilogy; the answer is simple: because I want to. Sure they are a bit esoteric, but I ran FreeBSD for years as my mail server. I still like the BSDs.

This article explores the architecture, implementation, and lessons learned from building a production-grade multi-platform build system that powers https://ballistics.zip, where users can download pre-built binaries for their platform with a simple curl command.

curl --proto '=https' --tlsv1.2 -sSf https://ballistics.zip/install.sh | sh

The Problem: Cross-Platform Distribution

Rust's cross-compilation capabilities are impressive, but they have limitations:

Cross-compilation complexity: While Rust supports cross-compilation, getting it working reliably for BSD systems (especially with system dependencies) is challenging
Native testing: You need to test on actual hardware to ensure binaries work correctly
Binary compatibility: Different BSD versions and configurations require native builds
Performance verification: Emulated builds may behave differently than native ones

The solution? Build natively on each target platform using actual hardware or high-performance emulation.

Architecture Overview

The build orchestration system consists of three main components:

1. Build Nodes (Physical and Virtual Machines)

macOS systems (x86_64 and aarch64) - Local builds
Linux x86_64 server - Remote build via SSH
FreeBSD ARM64 - Single-board computer (Raspberry Pi 4)
OpenBSD ARM64 - QEMU VM emulated on x86_64 (rig.localnet)
NetBSD x86_64 and ARM64 - QEMU VMs

2. Orchestrator (Python-based coordinator)

Reads build node configuration from build-nodes.yaml
Executes builds in parallel across all nodes
Collects artifacts via SSH/SCP
Generates SHA256 checksums
Uploads to Google Cloud Storage
Updates version metadata

3. Distribution (ballistics.zip website)

Serves install script at https://ballistics.zip
Hosts binaries in GCS bucket (gs://ballistics-releases/)
Provides version detection and automatic downloads
Supports version fallback for platforms with delayed releases

Hardware Infrastructure

Single-Board Computers

Orange Pi 5 Max (ARM64)

Role: Host for NetBSD ARM64 VM
CPU: Rockchip RK3588 (8-core ARM Cortex-A76/A55)
RAM: 16GB
Why: Native ARM64 hardware for running QEMU VMs
Host IP: 10.1.1.10
VM IPs:
NetBSD ARM64: 10.1.1.15
OpenBSD ARM64 (native, disabled): 10.1.1.11

Raspberry Pi 4 (ARM64)

Role: FreeBSD ARM64 native builds
CPU: Broadcom BCM2711 (quad-core Cortex-A72)
RAM: 8GB
Why: Stable FreeBSD support, reliable ARM64 platform
IP: 10.1.1.7

x86_64 ("rig.localnet")

Role: Linux builds, BSD VM host, emulated ARM64 builds
CPU: Intel i9
RAM: 96GB
IP: 10.1.1.27 (Linux host), 10.1.1.17 (KVM host)
VMs Hosted:
FreeBSD x86_64: 10.1.1.21
OpenBSD x86_64: 10.1.1.20
OpenBSD ARM64 (emulated): 10.1.1.23
NetBSD x86_64: 10.1.1.19

Local macOS Development Machine

Role: macOS binary builds (both architectures)
Build Method: Local cargo builds with target flags
Architectures:
aarch64-apple-darwin (Apple Silicon)
x86_64-apple-darwin (Intel Macs)

A Surprising Discovery: Emulated ARM64 Performance

One of the most interesting findings during development was discovering that emulated ARM64 builds on powerful x86_64 hardware are significantly faster than emulated ARM64 on native ARM64 builds on single-board computers.

Performance Comparison

Emulated ARM64 on ARM64: ~99+ minutes per build
Emulated ARM64 on x86_64: 15m 37s ⚡

The emulated build on rig.localnet (running QEMU with KVM acceleration) completed in about 6x less time than the native ARM64 hardware. This is because:

The x86_64 server has significantly more powerful CPU cores
QEMU with KVM provides near-native performance for many workloads
Rust compilation is primarily CPU-bound and benefits from faster single-core performance
The x86_64 server has faster storage (NVMe vs eMMC/SD card)

As a result, the native OpenBSD ARM64 node on the Orange Pi is now disabled in favor of the emulated version.

Prerequisites

SSH Key-Based Authentication

Critical: The orchestration system requires passwordless SSH access to all remote build nodes. Here's how to set it up:

Generate SSH key (if you don't have one):

ssh-keygen -t ed25519 -C "build-orchestrator"

Copy public key to each build node:

# For each build node
ssh-copy-id user@build-node-ip

# Examples:
ssh-copy-id alex@10.1.1.27      # Linux x86_64
ssh-copy-id freebsd@10.1.1.7     # FreeBSD ARM64
ssh-copy-id root@10.1.1.20       # OpenBSD x86_64
ssh-copy-id root@10.1.1.23       # OpenBSD ARM64 emulated
ssh-copy-id root@10.1.1.19       # NetBSD x86_64
ssh-copy-id root@10.1.1.15       # NetBSD ARM64

Test SSH access:

ssh user@build-node-ip "uname -a"

Software Requirements

On Build Orchestrator Machine:

Python 3.8+
pyyaml (pip install pyyaml)
Google Cloud SDK (gcloud command) for GCS uploads
SSH client

On Each Build Node:

Rust toolchain (cargo, rustc)
Build essentials (compiler, linker)
curl, wget, or ftp (for downloading source)
Sufficient disk space (~2GB for build artifacts)

BSD-Specific Requirements

NetBSD: Install curl via pkgsrc (native ftp doesn't support HTTPS)

# Bootstrap pkgsrc
cd /usr && ftp -o pkgsrc.tar.gz http://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz
tar -xzf pkgsrc.tar.gz
cd /usr/pkgsrc/bootstrap && ./bootstrap --prefix=/usr/pkg

# Install curl
/usr/pkg/bin/pkgin -y update
/usr/pkg/bin/pkgin -y install curl

OpenBSD: Native ftp supports HTTPS

pkg_add rust git

FreeBSD: Use pkg for everything

pkg install -y rust git curl

The ballistics.zip Website and Install Script

How It Works

https://ballistics.zip serves as the primary distribution point for pre-built ballistics-engine binaries. The system uses:

GCS Bucket:
gs://ballistics-releases/ - Binary artifacts
CDN: Google Cloud CDN provides global distribution
Install Script: Universal installer that:
Detects OS and architecture
Downloads appropriate binary
Verifies SHA256 checksum
Installs to /usr/local/bin

Usage

Basic installation:

curl -sSL https://ballistics.zip/install.sh | bash

Specific version:

curl -sSL https://ballistics.zip/install.sh | bash -s -- --version 0.13.3

Different install location:

curl -sSL https://ballistics.zip/install.sh | bash -s -- --prefix ~/.local

Install Script Architecture

The install.sh script intelligently handles:

Platform Detection:

OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)

case "$ARCH" in
  x86_64|amd64) ARCH="x86_64" ;;
  aarch64|arm64) ARCH="aarch64" ;;
  *) echo "Unsupported architecture: $ARCH"; exit 1 ;;
esac

PLATFORM="${OS}-${ARCH}"  # e.g., "openbsd-aarch64"

Version Fallback: If a requested version isn't available for a platform, the script automatically finds the latest available version:

# If openbsd-aarch64 0.13.3 doesn't exist, fall back to 0.13.2
AVAILABLE_VERSION=$(curl -sL $BASE_URL/versions.txt | grep "^$PLATFORM:" | cut -d: -f2)

Checksum Verification:

EXPECTED_SHA=$(cat "$BINARY.sha256")
ACTUAL_SHA=$(sha256sum "$BINARY" | awk '{print $1}')

if [ "$EXPECTED_SHA" != "$ACTUAL_SHA" ]; then
  echo "Checksum verification failed!"
  exit 1
fi

Build Orchestration System Deep Dive

Configuration: build-nodes.yaml

The heart of the system is build-nodes.yaml, which defines all build targets:

nodes:
  # macOS builds (local machine)
  - name: macos-aarch64
    host: local
    target: aarch64-apple-darwin
    build_command: |
      cd /tmp && rm -rf ballistics-engine-{version}
      curl -L -o v{version}.tar.gz https://github.com/ajokela/ballistics-engine/archive/refs/tags/v{version}.tar.gz
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      cargo build --release --target {target}
    binary_path: /tmp/ballistics-engine-{version}/target/{target}/release/ballistics
    enabled: true

  # Linux x86_64 (remote via SSH)
  - name: linux-x86_64
    host: alex@10.1.1.27
    target: x86_64-unknown-linux-gnu
    build_command: |
      cd /tmp && rm -rf ballistics-engine-{version}
      wget -q https://github.com/ajokela/ballistics-engine/archive/refs/tags/v{version}.tar.gz
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      ~/.cargo/bin/cargo build --release --target {target}
    binary_path: /tmp/ballistics-engine-{version}/target/{target}/release/ballistics
    enabled: true

  # OpenBSD ARM64 emulated (FASTEST ARM64 BUILD!)
  - name: openbsd-aarch64-emulated
    host: root@10.1.1.23
    target: aarch64-unknown-openbsd
    build_command: |
      cd /tmp && rm -rf ballistics-engine-{version}
      ftp -o v{version}.tar.gz https://github.com/ajokela/ballistics-engine/archive/refs/tags/v{version}.tar.gz
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      cargo build --release
    binary_path: /tmp/ballistics-engine-{version}/target/release/ballistics
    enabled: true

  # NetBSD x86_64 (HTTPS support via pkgsrc curl)
  - name: netbsd-x86_64
    host: root@10.1.1.19
    target: x86_64-unknown-netbsd
    build_command: |
      cd /tmp && rm -rf ballistics-engine-{version}
      /usr/pkg/bin/curl -L -o v{version}.tar.gz https://github.com/ajokela/ballistics-engine/archive/refs/tags/v{version}.tar.gz
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      /usr/pkg/bin/cargo build --release
    binary_path: /tmp/ballistics-engine-{version}/target/release/ballistics
    enabled: true

Orchestrator Workflow

The orchestrator.py script coordinates the entire build process:

Step 1: Parallel Build Execution

def build_on_node(node, version):
    if node['host'] == 'local':
        # Local build
        subprocess.run(build_command, shell=True, check=True)
    else:
        # Remote build via SSH
        ssh_command = f"ssh {node['host']} '{build_command}'"
        subprocess.run(ssh_command, shell=True, check=True)

Step 2: Artifact Collection

def collect_artifacts(node, version):
    binary_name = f"ballistics-{version}-{node['name']}"

    if node['host'] == 'local':
        shutil.copy(node['binary_path'], f"./{binary_name}")
    else:
        # Download via SCP
        scp_command = f"scp {node['host']}:{node['binary_path']} ./{binary_name}"
        subprocess.run(scp_command, shell=True, check=True)

Step 3: Checksum Generation

def generate_checksum(binary_path):
    with open(binary_path, 'rb') as f:
        sha256 = hashlib.sha256(f.read()).hexdigest()

    with open(f"{binary_path}.sha256", 'w') as f:
        f.write(sha256)

Step 4: Upload to GCS

def upload_to_gcs(version):
    bucket_path = f"gs://ballistics-releases/{version}/"

    # Upload binaries and checksums
    subprocess.run(f"gsutil -m cp ballistics-* {bucket_path}", shell=True)

    # Set public read permissions
    subprocess.run(f"gsutil -m acl ch -u AllUsers:R {bucket_path}*", shell=True)

    # Update latest-version.txt
    with open('latest-version.txt', 'w') as f:
        f.write(version)
    subprocess.run("gsutil cp latest-version.txt gs://ballistics-releases/", shell=True)

Running a Build

Dry-run (test without uploading):

cd build-orchestrator
./build.sh --version 0.13.4 --dry-run

Production build:

./build.sh --version 0.13.4

Output:

Building ballistics-engine v0.13.4
===========================================

Enabled build nodes: 7
- macos-aarch64 (local)
- macos-x86_64 (local)
- linux-x86_64 (alex@10.1.1.27)
- freebsd-aarch64 (freebsd@10.1.1.7)
- openbsd-aarch64-emulated (root@10.1.1.23)
- netbsd-x86_64 (root@10.1.1.19)
- netbsd-aarch64 (root@10.1.1.15)

Starting parallel builds...
[macos-aarch64] Building... (PID: 12345)
[linux-x86_64] Building... (PID: 12346)
...

Build results:
✓ macos-aarch64 (45s)
✓ linux-x86_64 (28s)
✓ freebsd-aarch64 (6m 32s)
✓ openbsd-aarch64-emulated (15m 37s)  ⚡ FASTEST ARM64!
...

Uploading to gs://ballistics-releases/0.13.4/
✓ Uploaded 7 binaries
✓ Uploaded 7 checksums
✓ Updated latest-version.txt

Build complete! 🎉
Total time: 16m 12s

Adding New Build Nodes

Interactive Script

The easiest way to add a new node is using the interactive script:

cd build-orchestrator
./add-node.sh

This will prompt you for: - Node name (e.g., openbsd-aarch64-emulated) - SSH host (e.g., root@10.1.1.23 or local) - Rust target triple (e.g., aarch64-unknown-openbsd) - Build commands (how to download and build) - Binary location (where the compiled binary is located)

Manual Configuration

Alternatively, edit build-nodes.yaml directly:

  - name: your-new-platform
    host: user@ip-address  # or 'local' for local builds
    target: rust-target-triple
    build_command: |
      # Commands to download source and build
      cd /tmp && rm -rf ballistics-engine-{version}
      curl -L -o v{version}.tar.gz https://github.com/...
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      cargo build --release
    binary_path: /path/to/compiled/binary
    enabled: true

Variables: - {version}: Replaced with target version (e.g., 0.13.4) - {target}: Replaced with Rust target triple

Setting Up a New VM

Example: OpenBSD ARM64 Emulated

Create VM on host:

ssh alex@rig.localnet
cd /opt/bsd-vms/openbsd-arm64-emulated

Create boot script:

cat > boot.sh << 'EOF'
#!/bin/bash
exec qemu-system-aarch64 \
  -M virt,highmem=off \
  -cpu cortex-a57 \
  -smp 4 \
  -m 2G \
  -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd \
  -drive file=openbsd.qcow2,if=virtio,format=qcow2 \
  -netdev bridge,id=net0,br=br0 \
  -device virtio-net-pci,netdev=net0,romfile=,mac=52:54:00:12:34:99 \
  -nographic
EOF
chmod +x boot.sh

Create systemd service:

sudo cat > /etc/systemd/system/openbsd-arm64-emulated-vm.service << 'EOF'
[Unit]
Description=OpenBSD ARM64 VM (Emulated on x86_64)
After=network.target

[Service]
Type=simple
User=alex
WorkingDirectory=/opt/bsd-vms/openbsd-arm64-emulated
ExecStart=/opt/bsd-vms/openbsd-arm64-emulated/boot.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable openbsd-arm64-emulated-vm.service
sudo systemctl start openbsd-arm64-emulated-vm.service

Configure networking (assign static IP 10.1.1.23)
Install build tools inside VM:

ssh root@10.1.1.23
pkg_add rust git

Test SSH access:

ssh root@10.1.1.23 "cargo --version"

Add to build-nodes.yaml and test:

./build.sh --version 0.13.3 --dry-run

GitHub Webhook Integration (Optional)

For fully automated builds triggered by GitHub releases:

1. Deploy Webhook Receiver to Cloud Run

cd build-orchestrator
gcloud run deploy ballistics-build-webhook \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --set-env-vars GITHUB_WEBHOOK_SECRET=your-secret-here

2. Configure GitHub Webhook

Go to: https://github.com/yourusername/your-repo/settings/hooks
Add webhook:
Payload URL: https://ballistics-build-webhook-xxx.run.app/webhook
Content type: application/json
Secret: Your webhook secret
Events: Select "Releases" only

3. Test

Create a new release on GitHub, and the webhook will automatically trigger builds for all platforms!

Performance Metrics and Insights

From real-world builds of ballistics-engine v0.13.3:

Platform	Hardware	Build Time	Notes
macOS aarch64	Apple M1/M2	45s	Native Apple Silicon
macOS x86_64	Intel i7/i9	30s	Cross-compile on Apple Silicon
Linux x86_64	Xeon/EPYC	25s	Fastest overall ⚡
FreeBSD aarch64	Raspberry Pi 4	6m 32s	Native ARM64 hardware
OpenBSD aarch64 (emulated)	x86_64 QEMU	15m 37s	⚡ FASTEST ARM64
OpenBSD aarch64 (native)	Orange Pi 5 Max	99+ min	Disabled due to slower speed
NetBSD x86_64	x86_64 VM	3m 45s	KVM acceleration
NetBSD aarch64	Orange Pi VM	8m 12s	QEMU on ARM64 host

Key Insights:

x86_64 is fastest: Modern x86_64 CPUs dominate for single-threaded compilation
Emulation wins for ARM64: x86_64 emulating ARM64 beats native ARM64 SBCs
SBCs are viable: Raspberry Pi and Orange Pi work well for native builds, but slower
Parallel execution: Running all 7 builds in parallel takes only ~16 minutes (longest pole is FreeBSD ARM64)

Conclusion

Building a custom multi-platform build orchestration system may seem daunting, but the benefits are substantial:

→ Full control: Own your build infrastructure

→ Native builds: Real hardware ensures compatibility

→ Cost-effective: Low operational costs after initial hardware investment

→ Fast iteration: Parallel builds complete in ~16 minutes

→ Flexibility: Easy to add new platforms

→ Learning: Deep understanding of cross-platform development

The surprising discovery that emulated ARM64 on powerful x86_64 hardware outperforms native ARM64 single-board computers has practical implications: you don't always need native hardware for every architecture. Strategic use of emulation can provide better performance while maintaining compatibility.

For projects requiring broad platform support (especially BSD systems not well-served by traditional CI/CD), this approach offers a reliable, maintainable, and cost-effective solution.

Architecture Diagram

graph TB subgraph "Trigger Sources" GH[GitHub Release
v0.13.x] MANUAL[Manual Execution
./build.sh] end subgraph "Build Orchestrator" ORCH[Python Orchestrator
orchestrator.py] CONFIG[Build Configuration
build-nodes.yaml] end subgraph "Build Nodes - Local" MAC_ARM[macOS ARM64
Apple Silicon
~45s] MAC_X86[macOS x86_64
Rosetta 2
~30s] end subgraph "Build Nodes - Remote x86_64" LINUX_X86[Linux x86_64
alex@10.1.1.27
~25s] FREEBSD_X86[FreeBSD x86_64
root@10.1.1.21
~4m] OPENBSD_X86[OpenBSD x86_64
root@10.1.1.20
~12m] NETBSD_X86[NetBSD x86_64
root@10.1.1.19
~3m 45s] end subgraph "Build Nodes - Remote ARM64" FREEBSD_ARM[FreeBSD ARM64
freebsd@10.1.1.7
~6m 32s] OPENBSD_ARM_EMU[OpenBSD ARM64
root@10.1.1.23
Emulated on x86_64
~15m 37s ⚡] NETBSD_ARM[NetBSD ARM64
root@10.1.1.15
~8m 12s] end subgraph "Artifact Collection" COLLECT[SCP Collection
Pull binaries from nodes] CHECKSUM[Generate SHA256
checksums] end subgraph "Distribution" GCS[Google Cloud Storage
gs://ballistics-releases/] WEBSITE[ballistics.zip
Install Script] end GH -->|webhook| ORCH MANUAL -->|CLI| ORCH CONFIG -->|reads| ORCH ORCH -->|SSH parallel builds| MAC_ARM ORCH -->|SSH parallel builds| MAC_X86 ORCH -->|SSH parallel builds| LINUX_X86 ORCH -->|SSH parallel builds| FREEBSD_X86 ORCH -->|SSH parallel builds| OPENBSD_X86 ORCH -->|SSH parallel builds| NETBSD_X86 ORCH -->|SSH parallel builds| FREEBSD_ARM ORCH -->|SSH parallel builds| OPENBSD_ARM_EMU ORCH -->|SSH parallel builds| NETBSD_ARM MAC_ARM -->|binary| COLLECT MAC_X86 -->|binary| COLLECT LINUX_X86 -->|binary| COLLECT FREEBSD_X86 -->|binary| COLLECT OPENBSD_X86 -->|binary| COLLECT NETBSD_X86 -->|binary| COLLECT FREEBSD_ARM -->|binary| COLLECT OPENBSD_ARM_EMU -->|binary| COLLECT NETBSD_ARM -->|binary| COLLECT COLLECT --> CHECKSUM CHECKSUM --> GCS GCS --> WEBSITE style OPENBSD_ARM_EMU fill:#90EE90 style LINUX_X86 fill:#87CEEB style GCS fill:#FFD700 style WEBSITE fill:#FFD700

Diagram Legend

Green: Fastest ARM64 build (emulated on powerful x86_64)
Blue: Fastest overall build (native Linux x86_64)
Yellow: Distribution endpoints

Build Flow

Trigger: GitHub release webhook or manual execution
Parallel Execution: All enabled build nodes start simultaneously
Collection: Orchestrator collects binaries via SCP
Verification: SHA256 checksums generated for integrity
Upload: Binaries and checksums uploaded to GCS
Availability: Install script immediately serves new version

The History, Use and Technical Details of QEMU

A.C. Jokela — Sun, 23 Jul 2023 03:43:01 GMT

Introduction

In today's interconnected world, the need for virtualization technologies is more pronounced than ever. These tools help developers simulate different computing environments, facilitating easy testing and debugging across platforms. QEMU, an abbreviation for Quick Emulator, is one such powerful technology. It's a free and open-source hypervisor that performs hardware virtualization. Since its inception, it has become a cornerstone in the world of emulation, offering support for various hardware platforms and operating systems. This article will delve into the history of QEMU, its uses, and technical aspects.

To be honest, I had shyed away from experimenting with QEMU; I had used VMWare and VirtualBox for some time. I first utlitized VMWare in 2007 when a, at the time, friend and I started a software development and consultancy. Amazon Web Services had been launched just one year prior and it was not an established name is computing. Cloud Computing was not called that; I recall a reading about Sun Microsystems developing "Utility Computing"; pay as you go. The idea seemed crazy to me at the time. When you needed compute power, you bought physical hardware, in our case, a Dell PowerEdge 1950. We did an initial installation of Debian Linux and installed VMWare and then shipped the server to a colocation facility in Miami, FL. There, we would create, manage, and ultimately destroy countless virtual machines over the lifetime of the server as well as the halcyon days of the consultancy.

In an odd chain-reaction of thoughts, I wanted to experiment with getting MacOS 9 running on modern hardware. As it turns out, QEMU comes in quite handy.

History of QEMU

QEMU was initially released in 2003, a creation of French programmer Fabrice Bellard, who is also known for creating other groundbreaking tools like FFmpeg and LZEXE. Bellard's idea was to develop a fast, portable dynamic translator that could make software developed for one machine run on another.

While the initial version of QEMU only supported emulation for x86, ARM, and SPARC architectures, it gradually expanded to cater to various others like PowerPC, MIPS, and more. Throughout its history, QEMU has continually evolved and improved, integrating with other projects like KVM (Kernel-based Virtual Machine) and libvirt, and extending its support to system and user-mode emulation.

Use of QEMU

QEMU has a wide array of applications, ranging from cross-platform development, virtualization, sandboxing, to hardware design and testing.

Cross-Platform Development and Testing: As QEMU can emulate different architectures, it has become an invaluable tool for developers, allowing them to compile and test their code across various platforms without needing physical access to them. This functionality significantly accelerates the software development process, enabling more efficient multi-platform software creation.

Virtualization: In conjunction with KVM, QEMU can run multiple operating systems concurrently on a single hardware host. This feature has driven the rapid growth of cloud computing, where multiple virtual machines are hosted on powerful servers, offering flexibility and scalability to businesses.

Sandboxing: QEMU's ability to emulate an entire system makes it ideal for creating secure sandboxes. This is particularly useful for testing potentially harmful code or software without risking the host system's integrity.

Hardware Design and Testing: In hardware design and testing, QEMU is used to emulate the behavior of different hardware components. It enables hardware designers to simulate and validate their designs before manufacturing the physical components.

Technical Details of QEMU

QEMU can function in two primary modes: system emulation mode and user-mode emulation.

System Emulation Mode: In this mode, QEMU can emulate a full computer system, including a processor and various peripherals. It can boot and run different operating systems and applications compiled for a different CPU, providing complete system isolation. The system mode is beneficial for debugging system code and running unmodified guest operating systems.

User-mode Emulation: In user mode, QEMU can launch individual Linux processes compiled for one CPU on another CPU. This approach is highly beneficial for running binary files from different architectures and is often used to build and test software in a cross-compilation environment.

The technical prowess of QEMU lies in its dynamic binary translation. It translates the binary code of a guest system into the binary code of a host one instruction at a time, creating a cache of translated code to optimize performance. QEMU supports numerous architectures, including x86, ARM, MIPS, SPARC, and PowerPC, among others, either as the host or the guest architecture.

When paired with KVM, QEMU's capabilities expand even further. KVM, a feature of the Linux kernel, facilitates hardware-assisted virtualization. When QEMU is combined with KVM, it uses the CPU's virtualization extensions, offering near-native performance for the emulated guest system.

To manage QEMU and KVM virtual machines, libvirt, a toolkit, is often used. It provides a common, consistent API, simplifying the management of different virtualization technologies.

Quick HOWTO

Installation: sudo apt-get install qemu-system-ppc
- You may or may not need to install other packages. Google will be your friend.
Download Required Files:
- Head over to here to grab an ISO of macOS 9.2.2
Prepare a QEMU Disk Image:
- Create a blank disk image that will serve as the virtual hard drive for macOS 9.2. You can do this using the qemu-img command: qemu-img create -f qcow2 macos9.qcow2 10G
Install macOS 9.2 on QEMU:
- Launch the QEMU virtual machine with the following command: qemu-system-ppc -M mac99 -m 512 -boot d -cdrom path/to/macos9.iso -hda macos9.qcow2 -netdev user,id=net0 -device sungem,netdev=net0 Note: Replace path/to/macos9.iso with the path to your macOS 9.2 installation ISO or CD image.
Restore:
- You cannot simply install macOS, you will need to restore. Locate Apple Software Restore on the CD drive. Run the program and follow the instructions.
Launching macOS 9.2:
Once the restore is complete, you can start macOS 9.2 using a similar command as before: qemu-system-ppc -M mac99 -m 512 -hda macos9.qcow2 -netdev user,id=net0 -device sungem,netdev=net0 -g 1024x768x32

Keep in mind that macOS 9.2 may not work perfectly on QEMU due to limited hardware emulation and driver support. If you encounter any issues, there might not be straightforward solutions. Use Google

Final Thoughts

Here are instructions on running MacOS 9 on a couple different flavors of Mac architecture. I personally got MacOS 9 running under Linux and specifically Armbian running on a Pine64 ROCKPro64; this meant that audio and a couple other things did not work.