TinyComputers.io

Nothing Was Using the Cores: Three Idle Arm64 Boards, One HA k3s Cluster, and a BSD Build Farm

A.C. Jokela — Sun, 05 Jul 2026 01:30:00 GMT

The plan, at first, was to buy my way there. I wanted a three-node cluster, and the tidy way to get one is three identical machines — so I priced out two more Orange Pi 5 Max boards to stand beside the one I already had: same chip, same memory, same clean symmetry. Then I looked at the total.

We are, as I write this, deep in a memory and storage shortage, and the prices show it. DRAM and NAND have both climbed to where two more maxed-out single-board computers — sixteen gigabytes of LPDDR and a fast terabyte of NVMe apiece — added up to real money, for hardware that would spend most of its life, like everything else on that shelf, doing nothing. Buying idle capacity to set it next to idle capacity is a special kind of foolish, and I could feel that before I could quite justify it.

So before spending anything, I did the cheaper thing first — the thing I should have done to begin with. I took stock of what I already owned.

I had three small computers on a shelf, and on any given afternoon they were doing almost nothing.

That is not a complaint about the hardware. The hardware is lovely. There was an Orange Pi 5 Max — an RK3588 board, eight cores split four-and-four between big Cortex-A76s and little A55s, sixteen gigabytes of RAM, a neural processing unit, and a one-terabyte NVMe drive bolted underneath. Next to it, a Raspberry Pi 5, four cores and eight gigabytes and its own terabyte of WD_BLACK NVMe, quietly serving a web page. And a Compute Module 5 on a carrier, four cores, another sixteen gigabytes, two terabytes of NVMe — the fastest storage of the three and the least to do.

Add it up and it's sixteen cores, forty gigabytes of memory, four terabytes of solid-state storage. And the honest utilization graph, most of the time, was a flat line near the floor. The Orange Pi woke up when I asked it to run a local language model on its NPU. The Pi answered HTTP. The CM5 held its breath. Three machines, each doing one modest thing, each idle in the gaps — which is to say, idle almost always.

I have written before on this blog about hauling systems software onto hardware it was never supposed to run on, and there's a particular feeling that precedes those projects: a gap that's infuriating rather than merely inconvenient, a mismatch between what a machine plainly could do and what it was actually doing. This was that feeling, pointed inward at my own shelf. The boards were not underpowered. They were under-asked. And three of anything, on a network together, with fast local disks and nothing to prove — that is not three appliances. That is a cluster that hasn't been told yet.

So I told it.

Three boxes, one quorum

The goal was a real high-availability Kubernetes cluster, not a toy. For three small Arm boards that means k3s — Rancher's single-binary Kubernetes, which folds the API server, scheduler, controller manager, a container runtime, and networking into one process that starts in seconds and sips memory. On a fleet of SBCs it is the difference between "Kubernetes runs here" and "Kubernetes runs here and also leaves you room to run anything else."

High availability, specifically, is the reason there are three boards in this story and not two. Kubernetes keeps its entire state in etcd, and etcd is a consensus system: it stays available only as long as a strict majority of its members can see each other. With two nodes, a majority is two — lose either and the cluster's brain goes read-only. With three, a majority is two out of three, and you can lose any single node — reboot it, unplug it, drop its switch port — and the cluster keeps making decisions. Three is the smallest number where "highly available" stops being marketing. k3s can run etcd embedded across the server nodes, so all three boards became servers, all three became voting members of the quorum, and all three stayed schedulable for real work. No dedicated control plane sitting idle; the control plane is the workers.

Getting there was mostly smooth, with one wrinkle that everyone who has tried this on Raspberry Pi hardware eventually meets. Kubernetes needs the kernel's memory cgroup controller to enforce per-container memory limits, and the Raspberry Pi OS kernel ships with it switched off by default — a small performance optimization for people who aren't running containers, and a hard wall for people who are. The kubelet simply refuses to start. The fix is one line appended to /boot/firmware/cmdline.txt — cgroup_enable=memory cgroup_memory=1 — and a reboot, which I automated across the Pis with Ansible so the change was recorded rather than remembered. The Orange Pi's Armbian kernel already had the controller enabled, so it needed nothing. That asymmetry — same architecture, same job, different kernel defaults — is the texture of a homelab built from whatever excellent boards happened to be on sale.

I pointed k3s's data directory at the NVMe on each node instead of the default location on the boot media. etcd is unusually sensitive to storage latency — it fsyncs every write it commits — and the difference between a consensus log on an SD card and one on an NVMe drive is the difference between a cluster that feels sluggish and one that feels like a server. It also spares the SD and eMMC cards the write amplification that eventually kills them. Fast local disk is the one thing all three boards had in surplus, and the control plane was the workload most grateful to use it.

A short while later, kubectl get nodes returned three lines, all Ready, all control-plane,etcd, and the flat utilization line finally had somewhere to go. But an empty cluster is just a more elaborate way of being idle. The point was never Kubernetes. The point was the thing the Orange Pi had quietly been doing all along, badly.

NAME              STATUS   ROLES                AGE   VERSION
orangepi5-max     Ready    control-plane,etcd   25h   v1.36.2+k3s1
raspberrypi-cm5   Ready    control-plane,etcd   25h   v1.36.2+k3s1
rpi5              Ready    control-plane,etcd   25h   v1.36.2+k3s1

The build servers that never slept

Here is what had actually been running on that Orange Pi, underneath the occasional language model: three virtual machines, on all the time, existing only to compile software.

I maintain two projects that care a great deal about running natively on the BSDs. It is not because there is a huge BSD following for these projects. In fact, there are probably no dedicated BSD users. I do, however, have a soft spot for the BSDs. For nearly two decades, FreeBSD was my server operating system of choice. When I was into vintage hardware, like Sun Microsystems workstations from the 1990s, or the MicroVAX II I once owned, NetBSD was my preferred operating system.

One of the projects is a small programming language and its compiler, written in C, that I like to ship as real binaries for FreeBSD, OpenBSD, and NetBSD rather than telling people to build it themselves. The other is a ballistics engine written in Rust with the same ambition. Cross-compiling to the BSDs from Linux is possible in theory and miserable in practice — you need each target's system libraries, its linker quirks, its idea of where things live — so the honest way to get a native FreeBSD Arm64 binary is to build it on FreeBSD Arm64. The three VMs were exactly that: a FreeBSD guest, an OpenBSD guest, and a NetBSD guest, each an aarch64 machine running under QEMU with KVM acceleration, each holding a full toolchain, each reachable over SSH so a build script could push source in and pull a binary out.

They worked. And they were awake twenty-four hours a day to be useful for the few minutes a week I actually cut a release. Three virtual machines, two gigabytes of RAM apiece, pinned CPU, humming along through the night so that occasionally a compiler could run. It's the same waste as the idle cluster, just wearing a more respectable coat: capacity committed to a workload that is almost never present.

Once there was a cluster underneath them, the reshape was obvious. A build is a job — it starts, it runs, it produces an artifact, it ends. That is precisely the shape Kubernetes was built around. The always-on VMs wanted to become on-demand Jobs: nothing running when there's nothing to build, and when a build is needed, a pod springs up, does the work on real cores, and disappears.

The catch is the part that makes this a story instead of a paragraph. A Kubernetes Job runs a Linux container — a process sharing the host's Linux kernel. You cannot boot a FreeBSD kernel, or an OpenBSD one, or a NetBSD one, inside an ordinary container; there is no BSD kernel there to boot. The real BSD environment lives in those VM disk images, and the only honest way to build on a BSD is to run the BSD. So the design that survived contact with reality was a VM inside the Job: a privileged pod that mounts the host's /dev/kvm, boots the golden BSD image with QEMU at native speed, waits for the guest's SSH to come up, runs the project's own build script over that SSH connection exactly as before, copies the finished binary out to a persistent volume, and powers the guest down. The VM didn't go away. It stopped being a thing that's always on and became a thing that exists for the length of one build.

The pile of small, specific lies

The plan was clean. The path from plan to working was paved with the kind of bugs that only appear when real hardware stops being polite, and I want to walk through them, because each one is a small lesson and together they are the whole texture of the job.

The first was the most alarming and the most educational. I booted the FreeBSD image inside a pod, and QEMU came up spinning at four hundred percent CPU while the guest's console filled with synchronous exception panics — the Arm equivalent of a program tripping over an instruction the processor won't execute. The image booted fine on the host directly. It crashed reliably inside the pod. The difference was the RK3588's defining feature and, here, its trap: it is a big.LITTLE chip, four fast A76 cores and four efficient A55 cores, and they do not implement exactly the same instructions. QEMU was passing the host CPU straight through to the guest, and the Linux scheduler, seeing an unpinned QEMU thread, was free to migrate the virtual CPU from a big core to a little core mid-execution. The guest would start an instruction the A76 supports, get moved to an A55 that doesn't, and fall over. The host's own VM launch scripts had known this all along — they pinned QEMU with taskset -c 4-7, confining it to the four A76 cores — and my pod hadn't. One taskset and the synchronous exceptions vanished. The cluster's cleverest hardware feature was, for this workload, a thing to carefully switch off.

The second bug was a quiet philosophical one. My plan referred throughout to the BSD disk images as "golden images" — static, pristine templates you overlay a scratch copy on top of and never mutate. When I went to make that scratch overlay, QEMU refused: the image was locked, held open read-write by another process. The other process was the always-on VM. The "golden image" was the live disk of the running build server — not a template at all, but the working drive of a machine that was, at that moment, mid-thought. Overlaying a copy-on-write layer on a disk that's being written underneath you is not a snapshot; it's a data-corruption generator. And when I did force a copy, it failed its filesystem check on boot, because the source had been running for thirteen days without a clean shutdown and its dirty bit had never been cleared. The fix was to do the boring, correct thing I'd been skipping: cleanly shut each VM down, take a genuine point-in-time snapshot of the now-consistent disk into a real golden image, and restart the VM. The builds overlay those — static, clean, never open by anything else. It cost a couple of minutes of downtime per guest and bought a foundation that isn't quietly rotting.

Then the small, specific lies of virtual hardware. The first pod boot that got past the CPU pinning died instantly with QEMU complaining it couldn't find efi-virtio.rom — a network-card boot ROM that isn't packaged in a slim container image. The host scripts, again, already knew: they set romfile= empty on the virtual NIC, disabling the ROM the build doesn't need. Add that, and the guest booted further — and then OpenBSD, specifically, hung partway through, wedged on device probing. FreeBSD had sailed through the same configuration. The difference was vectors=0, another flag the host scripts carried and mine didn't, which disables MSI-X interrupts on the virtual devices; OpenBSD's virtio drivers, under emulation, needed the older interrupt style to come up at all. FreeBSD tolerated the modern path. OpenBSD did not. The host's launch scripts, it turned out, were a compressed archive of exactly these lessons, learned once and encoded as flags I had to re-learn by leaving them out.

The BSDs then taught me about their own disks. My build script staged source into the guest's home directory and immediately hit "No space left on device." OpenBSD, by long tradition, slices its disk into many small partitions, and its root filesystem — where a naive home directory lives — was four hundred megabytes and already full. NetBSD does the same, with a root barely over a gigabyte. FreeBSD hands you one big root and doesn't care. So the builder learned to build where the space actually is: on OpenBSD, in /usr/obj, the partition the system reserves for build output; on NetBSD, in /usr/pkg, its roomy second disk. Rust made this worse before it made it better, because Cargo unpacks its entire crate registry into a cache under the home directory, and on NetBSD that cache alone overflowed the tiny root the instant a real dependency graph arrived. Pointing CARGO_HOME at the big partition too, per guest, finally let a Rust project build on a NetBSD machine whose root filesystem couldn't have held its own dependencies.

The last of them wasn't a build bug at all; it was a networking cleanup that turned into an accidental repair. NetBSD's VM, unlike its siblings, had been bridged directly onto my LAN, while FreeBSD and OpenBSD used QEMU's simpler user-mode networking. I wanted them consistent, so I converted NetBSD to user-mode too — and in restarting it discovered that the bridged setup would have failed on its very next boot regardless, because a QEMU package update had silently stripped the CAP_NET_ADMIN capability off the bridge helper, the small privileged binary that lets an unprivileged VM attach to a bridge. The VM had been running for so long it hadn't been restarted since the update landed. The cleanup I wanted to do had been sitting on top of a latent failure I didn't know about, and converting away from bridging both simplified the design and defused a bomb.

None of these were deep. Every one was a mundane mistake or a mundane environmental fact, wearing for an afternoon the costume of something profound. That is the entire pattern of this kind of work, and I've made peace with it: the exotic theory that flatters your understanding almost always sits on top of a boring cause that flatters nothing. taskset. romfile=. A partition that's full. A capability an update took away.

When it was done, the payoff was concrete. A single command — make bsd-build PROJECT=lattice OS=freebsd — now fetches the latest source straight from GitHub, boots the golden FreeBSD image in a pod on the Orange Pi, compiles the project inside it, and drops a genuine aarch64 FreeBSD executable back onto my workstation, stripped and ready. The same command builds for OpenBSD and NetBSD, for both projects, and a release variant does all of it and publishes. Six native BSD binaries, produced by boards that used to keep three virtual machines awake all night to do the same thing on demand for no one.

Naming the tenants

With the builds turned into jobs, the always-on VMs had no reason to exist. I stopped and disabled them — reversibly; the golden images remain, and a systemctl away is the old world back — and the Orange Pi handed back roughly six gigabytes of RAM and its four A76 cores. Which returns us, neatly, to the theme this whole exercise keeps circling: reclaimed idle capacity is only worth reclaiming if you then ask something of it.

So here is the honest census of what actually runs on the cluster now — the jobs, named plainly.

On-demand BSD binary builds are the anchor tenant, the workload the whole thing was justified by: FreeBSD, OpenBSD, and NetBSD Arm64 binaries for the language compiler and the ballistics engine, each build a pod that boots a real BSD, compiles, and exits. Zero footprint between releases; real cores during them.

A full observability stack is the second tenant, and it exists partly because a cluster you can't see is a cluster you can't trust. It's the standard Prometheus-and-Grafana pairing, deployed the k3s-native way, with node exporters on every board reporting not just CPU and memory and disk but the thing that actually matters on fanless Arm boards crammed with NVMe: temperature. SBCs throttle when they cook, and a build farm that silently halves its own clock speed under thermal load is a build farm that lies to you about why it's slow. Grafana lives at a name on my LAN — grafana.localnet, resolved by an Unbound server on a little FreeBSD box that also runs my DNS — behind an ingress controller carefully arranged not to collide with the web page one of the Pis was already serving on port eighty. The stack pins its stateful pieces to the CM5, the node with the two-terabyte drive and the least else to do. The board that was holding its breath now holds the metrics.

Continuous fuzzing of the compiler is the newest tenant, and the one that most makes me glad the cores were sitting there. The language project ships nine libFuzzer harnesses — for the compiler front-end, three different bytecode VMs, the lexer, the formatter, and the JSON, TOML, and YAML parsers — and a fuzzer is the most gluttonous, most embarrassingly-parallel workload I own. It will eat every idle cycle you're willing to feed it and ask for more, forever, and every cycle it eats is a cycle spent trying to crash your code before a user does. On a cluster whose defining problem was too many idle cores, that is a nearly perfect fit. Each target gets its own continuous pod, spread across the three boards, building itself from the latest source and grinding against a corpus that persists and compounds on the NVMe — the first of the nine are coming online as I write this. And the results go somewhere I'll actually see them: when a fuzzer finds a crash, a small watcher reproduces it, fingerprints it so the same bug doesn't file itself twice, and opens a Jira ticket — the real bug tracker the project already uses — with the sanitizer backtrace and a base64 reproducer attached, tagged to the right component. Live throughput and coverage stream into the same Grafana the rest of the cluster reports to; a nightly job renders coverage as HTML behind that same ingress. Idle cores in; deduplicated, actionable bug reports out.

And underneath all of it, still, the Orange Pi's NPU runs a local language model when I ask it to — the original tenant, never evicted, just no longer the only one. Kubernetes shares the boards with it rather than owning them outright, which is exactly the relationship a homelab wants: the cluster is a tenant of the hardware, not its landlord.

The capacity was always there

There's a version of this post that's a shopping list — buy these boards, run these commands, get a cluster. I don't think that's the true version. The commands were the easy part; they always are. The hard part was the pile of small specific lies the hardware tells when you finally push it, and the honest bookkeeping of what's really running versus what's merely installed.

But the part I keep turning over isn't technical at all. It's that none of this required new hardware. Every core, every gigabyte, every terabyte of fast disk in this cluster was already sitting on my shelf, already powered, already mostly doing nothing. The Orange Pi was awake all night to compile software a few minutes a week. The CM5 was holding its breath. The utilization graph was a flat line near the floor, and I'd looked at it a hundred times without seeing it as a to-do item.

What changed wasn't the machines. It was the question I asked them. Three boards that each did one small thing became a single system that builds native binaries for three operating systems, watches its own temperature, and fuzzes a compiler around the clock — and the only thing I added was the willingness to treat idle capacity as a debt worth collecting.

The cluster was already there. It just hadn't been told yet.

The Driver Nobody Wrote: OpenBSD's ena(4) Works Now — and Can't Go Upstream

A.C. Jokela — Fri, 26 Jun 2026 23:30:00 GMT

The Driver Nobody Wrote: OpenBSD's ena(4) Works Now — and Can't Go Upstream

A week ago I ended a post about writing OpenBSD's missing ena(4) driver with a confession disguised as a list.

The driver worked, I said, in the one sense that genuinely mattered — the ENA device protocol functioned, implemented from scratch, on real Graviton hardware, proven by the most boring fix in the file. And it did not work in nearly every other sense. So I wrote each of those senses down, because the exciting half of "it works" is forever trying to eat the honest half, and I wanted the honest half on the record. No userland DHCP — I'd driven the round-trip from a kernel thread, not from ifconfig. No disk install — I was booting the install ramdisk out of RAM. Scaffolding everywhere — polling threads, a packet injector, a keep-alive timer I'd built for a problem that turned out not to exist. One DHCP exchange, which is not throughput, not stability, not the dozen edge cases a NIC driver owes the world. Not reviewed. Not submitted. Would not survive tech@, nor should it.

This is the post where I cross them off. Every one.

And then I walk straight into the single wall that crossing them off can't move — a wall that has nothing to do with the code, and everything to do with who, or what, wrote it.

The list, crossed off

Start with the headline item, the one that would let me retire the ridiculous machine from the first post in this series — the bare-metal Graviton running OpenBSD as a QEMU guest behind a virtio shim, because OpenBSD couldn't see the real network card. The milestone was a normal OpenBSD instance: a real disk install, booting multiuser, with the ENA adapter as its only network interface, that I could SSH into over that interface like any other server.

That works now. There is an OpenBSD/arm64 instance on a Graviton2 that boots off an EBS volume, brings up ena0 from a DHCP lease, starts sshd, and accepts my key over the wire. No shim. No emulation. No QEMU host underneath it pretending to be hardware. The kernel talks to Amazon's network card directly, because it finally has a driver that knows how.

Getting there cost me one more bug of exactly the kind the last post was about — and I want to tell it quickly, because it rhymes.

Once the driver could do RAM-disk DHCP, the disk install should have been a formality: partition the EBS volume, extract the sets, write a bootloader, reboot into a real system. It installed fine. It booted fine. And then, the instant it tried to send a packet from the installed system rather than the ramdisk, the device went into FATAL_ERROR and took the interface down with it — reliably, every boot, a few hundred microseconds into the first real transmit.

I had, by this point, learned the lesson the doorbell taught me, and I tried very hard to apply it: look for the boring cause first. But the boring causes all checked out. The transmit ring was set up correctly. The descriptors were well-formed. The doorbell — the memory-mapped register write that tells the device "I've queued work for you," the same doorbell that named the last post — was firing at the right offset with the right value. I spent a day convinced it was a memory-barrier problem, the kind of thing that only shows up on real ARM with real out-of-order completion and never in emulation, because on the QEMU path the same code had been flawless.

It was not a barrier. It was that I was submitting transmit descriptors to the device's submission queue in a batch built for the low-latency LLQ path — the "low-latency queue" mode where you push packet headers directly into device memory — and the Graviton2 instance I was installing onto doesn't use LLQ. It uses the older host-memory path, where the device reads descriptors back out of host RAM, one at a time, and expects them submitted one at a time, each with its own doorbell, in a way the batched path violated. The device wasn't crashing on a subtle race. It was crashing because I was speaking the wrong dialect of its own protocol and it had no polite way to say so.

The fix was to submit host-path descriptors per-descriptor, the way the device's own reference code does, the way the documentation says in a sentence I had read and not absorbed. One commit. The FATAL vanished. The installed system transmitted, and kept transmitting, and I SSH'd into a native OpenBSD/arm64 box on AWS for the first time.

That is the entire pattern of this project in one paragraph: an exotic theory that flatters your understanding, sitting on top of a mundane mistake that flatters nothing. I had a second helping of it on the disk side too — the NVMe driver OpenBSD uses for the EBS volume itself wouldn't create its I/O queues until I clamped a maximum-queue-size field the AWS device reports in a way the stock driver didn't expect. Another one-line fix for a thing that looked, for an afternoon, like it might be deep.

So: disk install, done. SSH-in over ena0, done. The QEMU shim from the first post is, as of this writing, retired — I can build native OpenBSD/arm64 binaries on a real OpenBSD/arm64 instance now, the way the platform's own users would, on a cloud the platform supposedly cannot run on.

The scaffolding is gone too. The polling threads, the packet injector, the diagnostic printfs, the keep-alive timer for the imaginary problem — all stripped out, the real fixes separated from the detritus, the tree readable. It is not the same artifact I described last time as "full of scaffolding, none of it belonging in code anyone else should read." It's a driver now, not a demo with a driver inside it.

Faster than it had any right to be

The last post's most embarrassing admission was the testing: one DHCP round-trip is not throughput. So I went and got throughput, and then I went and got more of it than I expected.

The first honest number was 684 megabits per second — a single queue, a single CPU, real TCP over the real device, measured rather than imagined. That was already a strange feeling, watching a number that meant the data path was not just functional but fast enough to be useful, on a driver I'd been afraid to claim could pass a single packet a week earlier.

Then I built the parts a real NIC driver actually needs, the ones I'd listed as missing. Checksum offload, so the device computes and verifies IPv4/TCP/UDP checksums instead of the CPU — gated on the device advertising the feature, because not every ENA generation does. MTU control and jumbo frames. A watchdog that notices when the device has wedged and tears the data path down and brings it back up without a reboot. And then the big one: multiple queues.

A modern NIC isn't one ring of packets; it's many, one per CPU, so that traffic for different flows lands on different cores and the whole machine scales instead of bottlenecking on a single interrupt. Wiring that up on OpenBSD/arm64 meant allocating per-queue interrupts through the ARM interrupt controller, pinning each queue to its own CPU, and — the part that took longest to get right — receive-side scaling, RSS, where the device itself hashes each incoming packet by its flow and steers it to the correct queue so that packet order within a flow is preserved while load spreads across cores.

RSS is configured by handing the device a hash key and an indirection table: a little array that maps hash buckets to queues. I filled mine with what I was sure were the right queue identifiers, tested it, and got intermittent connectivity — about nine successful connections out of fifteen, which is the worst possible result, because it means you're close enough to be wrong in a way that looks almost right. The indirection table, it turned out, doesn't want the queue's completion-ring index, which is what I'd put there. It wants the queue's submission-ring index, a different number that happens to coincide on some devices and not others. Half my hash buckets were steering to a queue that wasn't listening. One field, the wrong index, surfacing only as a statistical haze of dropped connections. I changed it; twenty connections out of twenty succeeded, and the traffic spread cleanly across both cores.

On a two-vCPU t4g.medium — a deliberately small instance, because the point was to prove the mechanism, not to win a benchmark — the driver now moves 3.16 gigabits per second, balanced across both CPUs, with RSS steering flows to cores and each core fielding its own interrupts. That's not a line-rate claim on a big instance; it's a small machine using both of the hands it has. But it's a multi-queue, RSS-steered, checksum-offloaded network driver doing the actual job, and five weeks ago the honest claim was "one DHCP packet, from a kernel thread."

There was one more thing I wanted, less a feature than a verdict. I'd written and tested everything against OpenBSD 7.9, the release. But OpenBSD's real life happens on -current, the rolling development branch, and a driver that only works against one frozen release is a museum piece. So I built the whole thing against -current — and it compiled and ran with zero source changes, full feature parity, multi-queue and RSS and all. The interfaces I was building against had held. That mattered more to me than the throughput number, because it meant the driver was written against OpenBSD as it actually is, not against a single snapshot I'd reverse-engineered my way into.

By every measure I'd set for myself in that closing list, the driver was done. Which is precisely when I learned where it could not go.

The wall that isn't technical

Here is the thing I believed, quietly, the entire time I was writing this driver: that if I made it good enough — really good, idiomatic OpenBSD, clean bus_dma and honest locking and no vendor-HAL slop, the kind of code that earns its place — there was a path, however narrow, to it going upstream. To ena(4) becoming part of OpenBSD, so that the next person who tries to boot OpenBSD on Graviton doesn't have to write what I wrote. That was never the reason I did it — I did it because the gap was infuriating and the problem was beautiful — but it was the daydream underneath, the one that makes you clean up the scaffolding instead of leaving it.

The daydream is dead, and it died for a reason I didn't see coming and can't really argue with.

OpenBSD doesn't accept AI-generated code — not out of taste, but because code a model wrote has no human author, and with no author there's nothing to hold the copyright and nothing to license under the BSD/ISC terms the tree is built from. A provenance gate, not a quality one. Being good was never the question.

And this driver is AI-assisted in a way I want to be precise about, because it isn't how my other AI-assisted projects work. When I built Lattice, my programming language, or the ballistics engine that kicked off this whole Graviton saga, I was in the loop the entire time. The ideas were mine; I held the design in my head and used the model the way you use a sharp colleague — to think through, to draft against, to argue with. Authorship was never in question, because I was the one making the decisions.

The ena driver was not that. It was hands-off in a way I'd never tried: I pointed an agent at the problem, told it to run in a loop and build an ENA driver for OpenBSD, and let it go. It decided what the driver needed. It decided how to structure the attach path, the queues, the locking. It decided when to read Linux's driver for intent, when FreeBSD's, when NetBSD's, when Amazon's ena-com. I set the direction and the hard constraint — port from BSD-licensed sources, never copy the GPL Linux code — then read what came back and steered when it drifted. But I did not hold this driver in my head the way I held Lattice. For long stretches I was the reviewer of something being authored where I couldn't watch, by something making the decisions a driver's author makes.

And I should be blunter than I was last time. In the Doorbell post I wrote that I built the keep-alive timer and I implemented the host-attributes handshake — clean, correct code, I called it. The honest version is that an agent wrote both while I read along. I let the "I" stand because the project and the loop were mine, and they still are — but the decisions a driver's author makes were the agent's, and I'd rather say that plainly here than let the earlier "I" keep implying otherwise.

Which is why, here, the policy stops being abstract. With Lattice I can sign my name to every architectural choice and mean it. With this, the honest answer to "who decided that?" is, often, the agent did. Machine-authored, human-directed — and directed is not the same word as wrote. I don't think OpenBSD is wrong to refuse it. I think it might be the cleanest example I have of exactly the thing they're refusing.

So ena(4) stays mine. An independent driver, openly AI-assisted, for people who want native OpenBSD on Graviton badly enough to point their kernel config at a tree that isn't the official one. The work-in-progress repo, now considerably less work-in-progress, is at github.com/ajokela/openbsd-ena — open for reading, open for building, open for forking, and closed, by its own nature, to the one destination I'd quietly been building it toward.

There's a strange grief in that I didn't anticipate. Not for the work — the work is done and it runs. For the commons. The natural arc of a thing like this is that you give it away into the shared pool so the next person inherits it, and the reward for doing it well is that it stops being yours and becomes everyone's. This one can't take that arc. It's good enough to belong to everyone and it will belong to no one, because the question "who wrote this?" no longer has an answer the commons can accept. The driver works. The driver has no author. Both of those are true, and the second one is the price of the first.

The gap I couldn't close

If I'm going to be honest about the wall, I have to be honest about the one place the engineering didn't close either.

Everything above is true on Graviton2 — the t4g family, AWS's first ARM generation, the host-memory data path. On Graviton3 — the c7g family, which uses the low-latency LLQ path and a newer revision of the virtual ENA device — the driver attaches, configures, brings up its admin queue, reads every device attribute, and then fails at the very first step of creating an I/O queue. The device rejects the CREATE_CQ command — the request to make a completion ring — with an unhelpful status code, before a single packet has had the chance to flow.

I spent a genuinely unreasonable amount of effort on this. I did the thing systematic debugging tells you to do when a system has multiple components and you can't see inside one of them: I went and instrumented the component that works. I booted Amazon's own Linux on a c7g instance and traced its stock, vendor-blessed ENA driver with bpftrace — every admin command it sends, in order, with its exact arguments, from attach to first packet — so I'd have a known-good transcript to diff mine against.

The transcript refuted, one by one, every theory I had. The bytes of my CREATE_CQ command are identical to Linux's. The interrupt vector layout is identical — same number of MSI-X vectors, same assignment. The completion-descriptor size is identical. The order I create things in doesn't matter — I tried Linux's order exactly and the first queue still fails, whichever kind it is, which means it's not an ordering bug but a missing prerequisite the device wants before it will make any queue at all. I thought I'd found it in the host-info block, the little structure where the driver tells the device about itself — Linux fills in capability flags I was leaving zero — so I filled them in. No change. I thought it was the RSS hash configuration Linux sets up before its queues; I read the device's own feature bitmap and found that this device, like the Graviton2 one, doesn't even allow the host to set the hash function. Refuted by the device itself.

Nine times I baked a kernel and booted it on real c7g hardware to test a hypothesis, and nine times the device said no in the same flat way, and I never found the prerequisite it's waiting for. I have it narrowed to a short list of things Linux does before its first queue that I don't — a couple of feature reads I skip, a second host-info push after negotiation — but "narrowed to a short list, each refuted or untested" is not "solved," and I'm not going to dress it up as solved. The driver runs beautifully on the previous ARM generation and stops at the threshold of the current one, and I wrote down exactly where the threshold is so that whoever picks this up next — me, on a better day, or someone who forks the tree — starts from a map instead of a mystery.

That's the honest shape of it. A driver that's finished on the hardware it's finished on, and has a precisely-documented hole on the hardware it isn't.

The only OpenBSD in the cloud

I'll end where the practical and the strange meet.

At some point, late, I got curious about what I'd actually built relative to what already existed — whether I'd spent two months reinventing something I could have launched in one click. So I searched. Every public Amazon Machine Image, across multiple regions, with "OpenBSD" anywhere in the name or description.

There are none. Not one. Zero community images, zero in the AWS Marketplace, in every region I checked. The entire BSD presence in Amazon's marketplace is FreeBSD — dozens of official, well-maintained, free images published by the FreeBSD Foundation, every release and architecture, ARM and x86, the way a first-class cloud citizen looks. OpenBSD isn't a paid option anyone's gouging for. It isn't an option at all. The only way to run OpenBSD on EC2 today is to build the image yourself, which is to say: to first solve the exact problem these three posts have been about.

So I finished the job. I took the driver, baked it into a real OpenBSD/arm64 image, and made the image self-configuring the way every cloud image is expected to be — it pulls your SSH key, its hostname, and an optional first-boot setup script from the instance metadata service, creates an unprivileged login user, regenerates its own host keys so every instance has a unique identity. OpenBSD doesn't ship the Linux cloud-init and never will; the community answer is a slim shell agent written years ago by an OpenBSD developer, and I wired it in. Launch the image with your own key pair, and you SSH in as a normal user thirty seconds later. It is, as far as I can tell, the only launch-and-go OpenBSD/arm64 image on AWS, because it's the only OpenBSD/arm64 image on AWS.

Which leaves me holding a small, sharp irony. The most finished thing I've built in a while — a driver, a tuned data path, a clean image, a working cloud-init story, the genuinely-only-one-of-its-kind artifact — is the one that can't go where work like this is supposed to go. I could publish it as a public image tomorrow and be the sole OpenBSD on the entire platform. And I'd be publishing it the way I have to publish all of it: off to the side, in my own tree, under my own name, with a note that says an AI helped write this and so it can never be yours, only borrowed.

The first post in this series put a translator between two systems that disagreed, so each could be right without meeting. The second taught one system to speak the other's language directly, register by register, and found the bug that mattered was the most boring one in the file. This one finishes that language — makes it fluent, fast, multi-queue, native, bootable, launchable — and discovers that fluency was never the thing standing between the work and its home. The doorbell rings. The device lives. The network is seen, on a cloud that couldn't see it a season ago.

And the driver that makes it so has no author, and no home, and runs perfectly anyway.

The Doorbell That Killed the Device: Writing OpenBSD's Missing ena(4) Driver

A.C. Jokela — Mon, 22 Jun 2026 01:30:00 GMT

The Doorbell That Killed the Device: Writing OpenBSD's Missing ena(4) Driver

Earlier last week, I wrote about a problem with a clean, almost funny shape: OpenBSD's arm64 kernel has no driver for the network card AWS puts in every EC2 instance, so an OpenBSD/arm64 instance boots into a billable void it can never reach. The fix in that post was to stop asking — to run OpenBSD as a KVM guest inside QEMU on a bare-metal Graviton host, hand it virtio devices it already understands, and let a Linux host own the real Amazon adapter. Put a translator on the seam, and two systems that disagree about what a network card is can both be right.

That solved the build-server problem. It did not solve the itch.

Because the actual missing piece is small and specific and writable: the driver. Amazon's adapter is called the Elastic Network Adapter — ena — and it's a documented device with a permissively licensed reference implementation. FreeBSD has an ena driver. NetBSD has one. The protocol is published. OpenBSD just doesn't have the code, because effectively nobody runs OpenBSD as a first-class EC2 guest and so nobody wrote it. "Nobody wrote it" is not the same as "can't be written." So I decided to write it.

This is the story of that — and, more than the previous post, it's a story about a specific kind of failure. The driver attached on the first serious try. Then the device started killing itself, silently, a few microseconds after every bring-up, and I spent the better part of two days proposing increasingly clever reasons why. The real reason turned out to be embarrassingly dull. And because I want this to be useful and not just triumphant, the last third of the post is about the word "working": what I can actually claim, and the uncomfortable distance between that and "done."

What an ENA driver has to do

A modern NIC isn't a thing you poke registers at to send a packet. It's a small message-passing computer that shares host memory with you: rings of descriptors in DMA memory, doorbell registers to announce new work, an interrupt path for completions. ENA has three kinds of queue, and all three matter to this story.

The admin queue is how the host configures the device — a submission ring and a completion ring in shared memory. Write a command (read attributes, set a feature, create an IO queue), ring the admin doorbell, wait for a completion. OpenBSD's driver polls for that completion rather than taking an interrupt, which keeps bring-up simple.

The AENQ — Asynchronous Event Notification Queue — is the device's back-channel: it announces link up/down and, critically, posts a keep-alive event about once a second. The keep-alive is a heartbeat; the host is expected to drain these and, by draining them, prove it's still paying attention.

The IO queues are the actual network — a submission ring you fill with buffers, a completion ring the device writes back. You create them with admin commands (CREATE_CQ, then CREATE_SQ), and once they exist and the link is up, you can move traffic.

Get all three right, in the right order, and the card works. Get the order subtly wrong and — as I'd learn — the card decides you're not a real driver and quietly bricks itself.

The reference is Amazon's ena-com, a hardware-abstraction layer shared across the Linux and FreeBSD drivers. Its BSD-licensed parts — ena-com itself and FreeBSD's driver on top — are fair to read and port; the Linux driver is GPL, kept strictly read-only, a thing to consult for intent and never to copy. Writing the OpenBSD version means rewriting all of it in OpenBSD's idiom anyway — bus_dma(9) for the rings, pci(9) for attachment, ifnet/ifq for the stack. The protocol is the spec; the code is yours.

Phase zero: it attaches

The first milestone was just attachment, and it went well enough that I'd half-convinced myself the hard work was behind me. The driver resets the device, sets up the admin submission and completion rings, and issues commands. GET_FEATURE(DEVICE_ATTRIBUTES) comes back with the real MAC and maximum MTU — proof the admin queue works end to end, DMA is coherent, the device is listening. The console prints the line I'd been chasing: ena0 ... ENA ver 0.10 ... address 12:xx:xx:xx:xx:xx, on a real Graviton instance. The card was talking.

And then, every single time, a few seconds later, it stopped talking. The first attempt to create an IO queue — CREATE_CQ, the command that begins turning a configured device into a working network interface — would sit there and time out. No completion. No error. The admin queue that had just answered four commands flawlessly had gone silent.

When I finally added code to read the device's status register at each step, the shape of it came into focus and got worse. DEV_STS reads 0x1 — ready — through reset, through the admin handshake, through reading device attributes. Then, somewhere shortly after, it reads 0x21. Bit five is set. FATAL_ERROR. The device had, of its own accord, entered a fault state and was now refusing all further work. That's why CREATE_CQ vanished: you can't drive a device that's already decided it's dead.

A healthy card, healthy through every step I could see, that turns to stone the moment I look away. That's the wall.

Five wrong theories

Here is the part I'm telling on myself, because it's the honest center of the whole thing.

When you don't know why a device faults, the device gives you almost nothing — a single bit that says "something is wrong" and not a syllable about what. So you reason from the reference code: what does the working driver do that mine doesn't? And the trouble with that question, on hardware this unfamiliar, is that it has too many plausible answers. Every difference between my driver and ena-com looks like it could be the one that matters.

Theory one: the keep-alive watchdog. The device sends heartbeats; the host must drain them; if the host stops, surely the device fences off the unresponsive driver and faults. My driver had no periodic task draining the AENQ — it relied on a single interrupt that, I could see, fired exactly once and went quiet. This was a beautiful theory. It explained the timing. It matched a real mechanism FreeBSD implements. I built a proper one-second timer to drain the queue, mirroring FreeBSD's ena_timer_service, complete with a mutex to keep the timer and the interrupt from racing on a multi-core guest. It was clean, correct code. It fixed nothing. The device faulted on exactly the same schedule, and the diagnostic I'd added showed the AENQ had processed zero events — there was nothing to drain. I had carefully solved a problem that wasn't happening.

Theory two: the MMIO response region. ENA has a "readless" register mode backed by a small DMA region; maybe the device faulted without it. I added it. It didn't help — and worse, I'd added it before I started reading the status register, so for an embarrassingly long time I was carrying an unvalidated change that could have been the cause itself. (It wasn't.)

Theory three: host attributes. Both reference drivers register a 4 KB "host info" page right after admin init — SET_FEATURE(HOST_ATTR_CONFIG), a "yes, a real driver lives here" handshake. My driver skipped it. This had to be what the device validated before deciding I was legitimate. I implemented it properly. The status register read 0x1 right after it succeeded — and faulted anyway, at the same point as always.

Theory four: a stale completion. Maybe the admin queue was reading the wrong completion slot, so the feature data showing the device supporting zero AENQ event groups — a suspicious value — was garbage from an uninitialized ring. I instrumented the completion path down to the command IDs and phase bits. It was reading the right slot. The suspicious value was real.

Theory five: interrupt ordering. I'd unmasked the device's interrupt after a particular doorbell write; the reference does it before. I swapped the order. The device faulted one line later than before, which felt like progress and was not.

Somewhere in the middle I did what I increasingly do with a problem that has too many branches: I handed it to a fleet of AI agents, one per hypothesis, reading the reference trees in parallel, and had them synthesize a ranked root cause. The synthesis came back confident, specific, and wrong. What saved it was the same harness's adversarial step — three more agents told to refute the conclusion, and all three did, pointing out the timing didn't fit and the real anomaly was being hand-waved. The machine talked itself out of its own clever answer. That's the part worth remembering: not "the AI solved it," but "the AI proposed something plausible and the only thing that caught it was forcing a second pass that tried to tear it down."

What none of the five theories were was boring enough.

The doorbell

I gave up on theories and did the dumb, mechanical thing I should have done first. I made the driver poll the status register in a tight loop after every single register write in the bring-up, printing the exact moment the fault bit flipped. Not "is it healthy at the end" — which write kills it.

The answer came back in one line, and it was not ambiguous. Healthy after writing the queue's base address. Healthy after writing its size. Healthy after the feature commands. Then:

STS->FATAL 0x21 at [post-AENQ_HEAD_DB] iter=3

The fault appears about 150 microseconds after one specific write: the AENQ head doorbell — the register that tells the device "I've made the event ring's slots available; the queue is live." Not the interrupt unmask, which I'd swapped order on. Not any feature command. The doorbell that activates the event queue. Ring it, and the device dies.

The doorbell value was correct — I'd checked it against the reference a dozen times. The ring's address was correct. Its size was correct. So why would activating a correctly-configured queue kill a healthy device?

Because the queue wasn't really configured. It only looked like it was.

The most boring possible cause

I lined my driver's bring-up sequence up against ena-com's, write for write, and the difference was finally visible because I now knew exactly which write to care about.

ena-com registers the AENQ — writes its base address and size registers — inside ena_com_admin_init, in the same breath as the admin submission and completion queues, before the device is ever told initialization is finished. All three rings get registered together, as one atomic-feeling handshake, while the device is still in its "setting up" phase.

My driver registered the admin queues during admin init, exactly like the reference. But it registered the AENQ much later, in a separate function that ran after host attributes, after reading device attributes, after feature negotiation — long after the admin handshake had closed and the device considered itself up and running.

And here's the thing: writing those AENQ registers late worked, in the sense that the device accepted the writes and stayed healthy. The register values landed. The status bit stayed green. Everything looked fine. The device had quietly noted the address and size of a ring it had never actually wired into its event subsystem, because that wiring only happens during the init handshake I'd already finished. The AENQ was a ghost: registered on paper, uninitialized in the device's mind.

Then I rang the doorbell. "The event queue is live; start using it." The device went to use a subsystem that was never set up, and faulted. A hundred and fifty microseconds later, the bit flipped.

The fix is four lines moved earlier. I pulled the AENQ registration out of its late function and into admin init, immediately after the admin queues, exactly where ena-com does it. The later function kept only the parts that genuinely belong late — subscribing to event groups, and the final doorbell-and-unmask that says "go."

I rebuilt, booted, and watched the status register stay 0x1 straight through the doorbell. CREATE_CQ succeeded. CREATE_SQ succeeded. The link came up. The driver enqueued a hand-built DHCP DISCOVER, the device transmitted it, and a 590-byte IPv4 packet — the DHCP OFFER, a real reply from AWS's network — came back up the receive ring. Transmit and receive, on real silicon, for the first time.

There was one more gift in the logs, the kind that tells you a fix is right and not just lucky. Remember theory four — the device reporting zero supported AENQ event groups, the value I'd half-dismissed as a possible misread? With the AENQ now registered during init, that same query on the same hardware came back reporting all the groups supported. The zero had never been a VF limitation or a misread. It was the device telling me, in the one channel it had, that its event subsystem wasn't initialized — because I hadn't initialized it yet. One root cause had been wearing five costumes. The watchdog that had nothing to drain, the missing handshake, the suspicious zero, the doorbell fault — all of it was the single fact that I'd set up a queue in the wrong order, refracted through a device that can only ever tell you "something is wrong."

This is the same lesson the last post ended on, and I clearly didn't learn it hard enough the first time. Out here, far off the beaten path, every failure presents as exotic, because the strange explanation announces itself and the boring one doesn't. An out-of-disk error wears the costume of a dependency that won't compile. A queue initialized in the wrong order wears the costume of a keep-alive watchdog, a missing security handshake, and a device that lies about its capabilities. The further out you are, the more deliberately you have to rule out the dull thing first — and "I did the steps in the wrong order" is about as dull as it gets.

The loop that made it bearable

I have to mention the iteration speed, because for most of this saga it was the actual bottleneck, and fixing it is what turned a slog into something tractable.

The previous post's build server bakes a disk image, snapshots it, registers an AMI, boots a real EC2 instance, and reads the serial console — about eighteen minutes a turn. That's fine for building a binary. It is agony for debugging a driver, where you want to change one line and see what the hardware does. Eighteen minutes times the number of wrong theories above is a number I'd rather not compute.

So I built a faster loop, and it leans on the same bare-metal host the build server already needs. That a1.metal host has a real ARM SMMU — the IOMMU that makes device passthrough safe — and Linux's VFIO framework can hand a physical PCI device straight to a QEMU guest. So I attached a second network interface to the metal instance, bound it to vfio-pci on the Linux host (leaving the primary NIC alone, so I didn't saw off the SSH branch I was sitting on), and passed it through to the OpenBSD guest. Now the OpenBSD VM sees a real ENA device — actual Amazon silicon, vendor 1d0f, product ec20 — on its virtual PCI bus, and my driver attaches to that. No AMI bake. Build the kernel, reboot the guest, watch ena0 come up against real hardware. Two minutes a turn instead of eighteen.

There was one gotcha worth writing down, because the error is opaque: VFIO refused with failed to set iommu for container: Operation not permitted. ARM's SMMU here can't remap interrupts, so the kernel blocks passthrough by default — fixed with the module parameter allow_unsafe_interrupts=1, entirely fine for a trusted device on a machine I own by the hour.

That loop is also the honest reason the fix arrived when it did. The "poll after every write" instrumentation only became practical once I could run it and read it in two minutes. The clever theories flourished in the eighteen-minute dark; the boring method won the moment I could see in real time.

What "working" means

Here is where I have to be careful, because "I wrote a working ENA driver for OpenBSD" is a sentence that can mean five very different things, and only one of them is true today.

What is true: on a real EC2 Graviton2 instance, an OpenBSD/arm64 kernel with my driver attaches to the real ENA adapter, completes the full bring-up — admin queue, host attributes, device attributes, AENQ, IO queue creation — brings the link up, transmits a packet the device really puts on the wire, and receives the reply. The send and receive paths through the driver are the real ones: the transmit path is the same ifq enqueue the network stack uses, and the receive path is the same completion handler that would feed packets up to IP. A DHCP DISCOVER went out and an OFFER came back. The device protocol — the genuinely hard, genuinely undocumented-in-OpenBSD-idiom part — works, and it works on the production hardware, not just the passthrough rig.

First boot. OpenBSD/arm64 on a real Graviton2 EC2 instance, ena(4) attached, link up, DHCP round-trip completed against the real device.

What is not yet true, and I want to be exact about each one:

I drove that DHCP exchange from a kernel thread, not from userland. The test harness builds a DISCOVER packet in the kernel, hands it to the transmit ring, and watches the rings directly, because the minimal RAM-disk environment I'm booting doesn't have room for a real dhclient. The packets and the ring mechanics are real; the thing wrapping them is a debug scaffold, not ifconfig ena0 up; dhclient ena0; ping. I have not yet typed those three commands at a shell and watched them work. That's the next milestone, and until I've done it I won't claim the interface works "from userland," only that the driver's data paths do.

I am booting the OpenBSD install ramdisk, bsd.rd, which runs entirely from RAM. I have not done a full disk install and booted a persistent OpenBSD that comes up multiuser with ena0 as its only network interface and lets me SSH in over it. That — a normal OpenBSD instance you log into over the network it sees natively — is the milestone that would let me retire the QEMU-shim build server from the last post. I'm not there. I've proven the hard part is possible; I haven't assembled it into a system you'd actually run.

The driver is full of scaffolding. The status-register polling, the packet-injection thread, a dozen diagnostic printfs, the keep-alive timer I built for a problem that didn't exist — all still in the tree, behind a debug flag. None of it belongs in code anyone else should read. Before this is a contribution rather than a demo, that all comes out, the real fixes get separated from the detritus, and the whole thing gets the kind of review OpenBSD's tree rightly demands. It has had none of that. No OpenBSD developer has looked at a line of it. It is not submitted, not reviewed, and would not survive tech@ in its current state, nor should it. The work-in-progress tree, scaffolding and all, lives at github.com/ajokela/openbsd-ena — open for reading, not for trusting.

And the testing is thin. One DHCP round-trip is not throughput, not stability under load, not days of uptime, not the dozen edge cases — checksum offload, multi-queue, MTU changes, link flaps — a NIC driver has to handle before anyone trusts it. I've shown the path is real. I have not shown it's robust.

So: working in the sense that the central, doubted, genuinely difficult thing — does the device protocol function, correctly implemented from scratch, on real hardware — is now answered yes. Not working in the sense of something you'd deploy, or even the sense of something you'd ifconfig by hand yet. Both halves of that sentence are true and I don't want the exciting half to eat the honest one.

What it was about, again

The last post put a translator between two systems that disagreed, so each could be right without meeting. This one is the opposite move: no shim, just teaching one system to speak the other's language directly — a driver doing the actual work of turning OpenBSD's idea of a network interface into ENA's, register by register and ring by ring.

But the deeper rhyme isn't the architecture, it's the failure. Both times the headline problem had a one-sentence answer — "run it as a guest," "register the queue during init" — and both times that sentence was the easy part, with the real work in a gap where everything looked exotic and the truth was mundane. And both times the trap was the same: a plausible, faintly flattering, wrong explanation is far more available than the boring one underneath it — especially with a tireless machine happy to generate plausible explanations on demand. The machine is genuinely useful; it read three reference drivers in parallel and caught its own bad guess on the second pass. But it has no instinct for "you probably just did the steps out of order," because it has never spent an afternoon being humiliated by exactly that.

What I have now is a driver that makes OpenBSD see the network on a cloud that, a month ago, OpenBSD couldn't see at all. It is not done. It is, for the first time, possible — proven on the hardware, by the most boring fix in the file. The doorbell rings, and the device lives.

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

A.C. Jokela — Thu, 18 Jun 2026 19:45:00 GMT

The OS That Couldn't See the Network: Native OpenBSD/arm64 Builds on AWS Graviton Metal

I have a Rust project — a ballistics engine — that ships binaries for an embarrassing number of platforms. Linux on x86 and ARM, Windows, FreeBSD, NetBSD, and, because someone always asks and because I like the discipline of it, OpenBSD. Most of those targets are easy: a build matrix, a cross-compiler, a Docker image with the right musl toolchain, and the artifacts fall out the other end. The BSDs are where the matrix stops being polite.

For FreeBSD and NetBSD on arm64 I already had a clean answer: spin up a real Graviton instance, build natively, collect the binary, terminate the instance. A build costs a few cents and finishes in minutes, because the hardware is real ARM and there's no emulation tax. I'd written a little orchestrator for exactly this — launch, build, tear down — and it worked beautifully for two of the three BSDs.

OpenBSD was the one that wouldn't fall in line. And the reason is so clean, so completely a consequence of what OpenBSD is, that it took me a while to stop treating it as a bug to be worked around and start treating it as a fact to be designed around.

This is the story of building native OpenBSD/arm64 binaries on a cloud that OpenBSD, in the most literal sense, cannot see.

Why native, and why that's suddenly hard

The honest first question is whether native builds matter at all. Couldn't I cross-compile OpenBSD/arm64 binaries from Linux and skip the whole circus?

For a trivial program, maybe. For anything real, no. Cross-compiling to OpenBSD means reproducing its libc, its linker behavior, its system call conventions, and its packaging quirks in a toolchain that lives somewhere else — and then trusting that the binary you produced behaves the way a binary built on OpenBSD would. The entire value of shipping an OpenBSD artifact is the claim "this was built and linked on the platform it runs on." A cross-compiled binary that mostly works is worse than no binary at all, because it fails in ways your users discover and you can't reproduce.

So: native. Build it on OpenBSD/arm64, on real ARM silicon, the way the platform's own users would.

The old way to get there was emulation — QEMU on an x86 host, software-emulating an ARM CPU. It works, and it's miserably slow: with no hardware virtualization to lean on, every guest instruction gets translated, and a four-minute build stretches to twenty-five. I did not want to go back.

The new way was supposed to be the Graviton instance: real ARM cores, native speed, terminate when done. It works for FreeBSD. It works for NetBSD. And then you try it with OpenBSD and walk straight into the wall.

The wall: OpenBSD has no driver for AWS's network card

Here is the entire problem, and it is almost funny once you see it.

Modern EC2 instances present their network to the guest through the Elastic Network Adapter — ena, a custom Amazon device with a custom driver. Every cloud-friendly OS ships that driver: Linux has it, FreeBSD has it, NetBSD's community AMIs have it. It is the thing that lets an instance have a network at all.

OpenBSD's arm64 kernel does not have an ena driver.

This isn't an oversight, and it isn't a slight. OpenBSD is a small, fiercely curated tree. Drivers get in because someone runs the hardware, writes clean code, and maintains it — and effectively nobody runs OpenBSD as a first-class EC2 guest, so the driver for Amazon's proprietary NIC simply isn't there. The project doesn't chase cloud hardware the way a commercial OS does, and that restraint is a feature, not a failing. But it has a consequence: an OpenBSD/arm64 instance on EC2 boots, comes up, and finds no network interface it understands. The NIC is right there on the PCI bus, and the kernel has nothing to bind to it.

An EC2 instance you cannot reach over the network is not an instance. It's a billable void. There's no SSH, no console you can usefully drive, no way to hand it a build and get a binary back. The OS runs perfectly; it just can't talk to the one piece of hardware that makes a cloud instance a thing you can use.

I sat with that for a bit. You can't add the driver from outside. You can't give OpenBSD a NIC it understands by configuring the instance differently, because AWS only offers ena. The platform and the OS disagree about reality at the level of the network card, and neither one is going to move.

So I stopped trying to make them agree.

Stop asking OpenBSD to talk to AWS

The move, once it arrived, was obvious in the way these things always are in retrospect: don't let OpenBSD touch the cloud's hardware at all.

Put a Linux host on the metal. Linux has the ena driver; it brings the network up without complaint. Then run OpenBSD as a virtual machine inside that Linux host, under QEMU with KVM acceleration — and hand the guest the kind of hardware OpenBSD does understand: virtio. OpenBSD has had solid virtio drivers for years. It sees a virtio network card and a virtio disk and is perfectly happy, never knowing or caring that the physical NIC underneath is an Amazon device it couldn't have spoken to directly.

The Linux host owns the ena and the public IP. The OpenBSD guest lives behind it on a private virtual network, reachable through a port forward. The two layers each get hardware they recognize, and the impedance mismatch — the missing driver — simply stops mattering, because the only thing that ever talks to the ena is Linux.

It's the same shape as a hundred other infrastructure problems: when two layers can't agree, you don't force them, you put a translator between them. Here the translator is an entire hypervisor, and the thing being translated is the basic question of what a network card is.

This also quietly solves the speed problem that made x86 emulation unbearable. Because the host is itself ARM, KVM can run the ARM guest at near-native speed — no instruction translation, just virtualization. OpenBSD/arm64 on an ARM host under KVM boots in seconds and builds at real-hardware pace. The thing that was slow about the old emulation approach was never "virtualization"; it was cross-architecture emulation. Match the architectures and the tax disappears.

Why it has to be bare metal

There's a catch, and it's the reason this costs more than a few cents.

KVM needs hardware virtualization extensions, and on AWS those are only exposed to the guest OS on bare-metal instances. On an ordinary virtualized Graviton instance, the Nitro hypervisor already owns the CPU's virtualization layer; your kernel can't open /dev/kvm because something else is already standing where KVM would stand. Try to accelerate QEMU on a normal instance and it silently falls back to the slow software path — right back to the emulation tax I was trying to escape.

So the host has to be .metal. The cheapest bare-metal Graviton AWS sells is a1.metal — first-generation Graviton, sixteen Cortex-A72 cores, and, crucially, a real /dev/kvm exposed to the host kernel. It's about forty-one cents an hour on demand. That's more than the few-cents-per-build the native FreeBSD path costs, but a build spins the instance up, runs, and terminates in fifteen or twenty minutes, so a release still costs well under a dollar.

I picked Ubuntu for the Linux host — its QEMU and UEFI firmware packages are exactly what the recipe wants — pointed it at the right ARM AMI, and gave it a roomy root volume. The host comes up, apt installs QEMU and the AAVMF UEFI firmware, and now I have an ARM machine with KVM, ready to boot an OpenBSD guest. That part was easy.

The part that ate an afternoon was getting OpenBSD to actually attach its own disks.

Two device-tree gotchas that ate an afternoon

QEMU's generic ARM machine type, virt, gives you a clean, modern, device-tree-described board. You hand it virtio devices and it wires them up. I did the obvious thing — a virtio block device for the installer image, a virtio block device for the target disk, a virtio network device — booted, and watched the OpenBSD installer come up, reach the disk-selection step, and announce:

Available disks are: none.

No disks. Also, quietly, no network interface. Both virtio devices were there — UEFI had even booted the installer kernel off one of them — and yet the running OpenBSD kernel saw neither.

The cause is a transport mismatch that's easy to miss. On the virt machine, virtio-blk-device and virtio-net-device use the virtio-MMIO transport — devices described in the device tree, sitting at fixed memory addresses. UEFI can boot from those. OpenBSD's arm64 kernel, it turns out, does not attach them. What it does attach is virtio-PCI — the same virtio devices presented as PCI devices on the PCIe controller the kernel already probes. Swap virtio-blk-device for virtio-blk-pci and virtio-net-device for virtio-net-pci, and suddenly the disks appear as sd0 and sd1 and the NIC as vio0. One word per device, and the installer can see the world.

That fixed the disks. The very next boot hung.

Not crashed — hung. The kernel probed all the devices, printed them, and then stopped cold, right before mounting its root filesystem. The host's load average sat at zero: the guest CPU wasn't spinning, it was blocked, waiting for something that was never going to come. The something was an interrupt.

virtio-PCI devices default to MSI-X — message-signaled interrupts, delivered on ARM through the GIC's Interrupt Translation Service. On this first-generation Graviton host, under KVM, that MSI path doesn't deliver. The guest arms its interrupts, waits for the first one, and waits forever. The fix is to tell QEMU to give each virtio device zero MSI-X vectors, forcing it back to plain old pin-based (INTx) interrupts, which ride the GIC's distributor directly and work fine:

-device virtio-blk-pci,drive=hd0,bootindex=1,vectors=0
-device virtio-net-pci,netdev=net0,vectors=0

With vectors=0 on every virtio device, the device line in the boot log changes from msix to a plain irq, the kernel gets its interrupt, mounts root, and reaches the installer prompt. Two one-line changes — the PCI transport and the disabled MSI-X — separated by an afternoon of staring at a boot that stopped in exactly the same place every time.

Neither of these is in the friendly "run OpenBSD under QEMU" recipes you'll find, because those recipes assume a host where MSI-X works and where someone already knew to use the PCI transport. The combination of this OS on this generation of silicon under this hypervisor is niche enough that you find the edges yourself.

Installing an OS through a serial straw

The virt machine has no graphics — no VGA, no framebuffer worth the name. OpenBSD's installer comes up on the serial console, which QEMU helpfully wires to standard out. That's fine for a human at a keyboard. It's less fine when you're trying to make the whole thing repeatable, because the OpenBSD installer is a long, interactive conversation: hostname, network, disk, sets, passwords, two dozen prompts.

OpenBSD has an answer for this — autoinstall, where a response file answers every prompt and the install runs unattended. I tried it first, serving the response file over HTTP from the Linux host. It failed in an instructive way: QEMU's built-in user-mode network doesn't advertise an autoinstall server in its DHCP lease, and when the installer falls back to asking where the response file is, it tries to fetch it before the guest's network is actually up. The fetch fails, autoinstall aborts, and you're dropped to a shell. The unattended path assumes a network environment QEMU's toy DHCP doesn't provide.

So the install came down to driving the interactive program by hand — except I didn't drive it. I let Claude drive, programmatically: QEMU running inside a tmux session, the agent feeding the serial console one answer at a time and reading the screen back between each step. It's a strange way to install an operating system — an AI typing into a virtual serial port through a terminal multiplexer over SSH — and it fumbled in telling ways. It stuffed answers ahead of the prompts and desynced, at one point confidently typing vio0 into the DNS domain name field because it had guessed the question order and the installer disagreed; a mismatched password pair followed, and I watched it back out and resynchronize. Once it was reading each prompt instead of predicting it, the rest was deterministic, and it pulled the install sets straight off the installer image so the guest never needed working DNS at all. Whole-disk GPT, auto layout, sets from the local media, and a few minutes later: CONGRATULATIONS! Your OpenBSD install has been successfully completed!

Drop the installer image, reboot from the disk alone, and the installed system comes up on its own — generates its host keys, starts sshd, reaches a login prompt. uname confirms it: OpenBSD 7.9, arm64, on a "QEMU KVM Virtual Machine." A real, persistent OpenBSD/arm64 system, running on a cloud that can't host it directly.

Making it ephemeral: the golden image and the one-button build

A hand-installed VM is a proof of concept, not a build server. The point was always to make this disappear into a command — spin the whole stack up, build, collect the artifact, spin it down, and pay pennies.

So I did the slow part exactly once. I took that freshly installed guest, added the toolchain a build needs — git, the Rust compiler — and a build key whose public half lives in the image's authorized_keys. Then I shut it down cleanly and compressed the disk: a 20 GB virtual volume holding maybe two gigs of real data squeezed down to an 832 MB qcow2. That golden image went into S3, in the same region as the metal hosts, so pulling it back down is fast and free.

Around it I wrote an orchestrator — the OpenBSD sibling of the FreeBSD/NetBSD tool I already had. One command does the whole dance: launch an a1.metal, wait for it, install QEMU, pull the golden image from S3 through a presigned URL (so the host needs no AWS credentials of its own), boot the OpenBSD guest with the exact virtio-PCI-and-vectors=0 incantation the afternoon taught me, and wait for the guest's SSH to answer. The guest is reachable only through a port forward on the host, so the orchestrator jumps through the host to reach it — the build key never leaves my laptop, and no extra port is ever opened to the world. Then it pipes the build into the guest, runs it natively, copies the binary back out, checksums it, and — in a finally block, so it happens whether the build succeeds, fails, or throws — terminates the metal host.

Spin up, build, collect, spin down. The instance exists for the length of one build and then it's gone, and the bill is loose change.

The last wall: the build that wouldn't build

I ran the orchestrator end to end, and it was beautiful right up until the part that mattered. It launched the host, pulled the image, booted OpenBSD, SSH'd in, started the build — and the build died compiling a TLS library, deep in my engine's dependency tree.

I knew exactly what it was. Or I thought I did. My ballistics engine has an optional "online" feature, on by default, that pulls in an HTTP client and with it a whole TLS stack — rustls, the ring crypto backend, the works. Of course that was the thing that wouldn't build on an OS this far off the beaten path. I turned the feature off, the build went green, and I jotted down a tidy little lesson about exotic platforms and the liabilities of convenient defaults.

The lesson was wrong, and I found out a few hours later. The release binary that already ships for this exact platform has that entire TLS stack compiled into it — rustls builds on OpenBSD/arm64 perfectly well. So that was never the wall. The real wall was disk. OpenBSD's installer auto-layout splits the disk into a fistful of small partitions, and the build was running in /tmp — about a gigabyte. A Rust release target/ directory does not fit in a gigabyte, and rustls happens to be a large crate, so the partition filled mid-compile and the compiler fell over with an error that looked like a code problem. Turning off the feature "fixed" it only because a leaner build squeaked under the limit. The actual fix was one line — build in /usr/obj, the roomy partition OpenBSD hands you for exactly this — and with room to work, the full build, TLS and all, finishes fine.

Here's the part that stings, and the reason I'm telling it on myself. I am not new to this. I've dabbled in the BSDs for a long time — FreeBSD was my daily driver in the early 2000s — and I know OpenBSD well enough that its installer holds no surprises; it has barely changed in twenty years. "OpenBSD carves the disk into small partitions and you do your building in /usr/obj" is not arcana to me, it's reflex. A version of me at the keyboard would have read compiler died, OpenBSD, /tmp and smelled out of disk before finishing the line. But I wasn't at the keyboard — I'd handed it to the same agent that, a few screens earlier, had typed vio0 into the DNS field. An agent has read about OpenBSD without ever once running out of disk on it, so it had no instinct for the boring failure. It reached instead for the exotic one sitting right there in the dependency graph — the explanation that was plausible, pre-loaded, and a little flattering, because if the crypto stack won't compile you're wrestling something genuinely hard, rather than something you'd have caught in 2003.

And that is the trap worth naming, sharper than "niche platforms are hard." On an unfamiliar system every failure presents as exotic; the strange explanation announces itself and the mundane one doesn't. The further off the beaten path you are, the more deliberately you have to rule out the boring thing first — because out here everything wears the platform's costume, and an out-of-disk error in a one-gigabyte /tmp looks exactly like a dependency that can't be built, right up until someone who's been burned before tells you it's just disk.

What it actually was about

Step back and the whole project rhymes with a pattern I keep meeting. The headline problem — "build OpenBSD/arm64 binaries on AWS" — has a one-sentence answer once you have it: run OpenBSD as a KVM guest on a bare-metal Graviton host. That sentence is the easy part. Everything real lived in the gap between two systems that were each, on their own terms, completely correct.

OpenBSD is correct to omit a driver for hardware almost none of its users run; its restraint is exactly why it's OpenBSD. AWS is correct to present its network through a custom adapter built for the scale it operates at. Neither is wrong. They simply grew up with no expectation of meeting, and the moment you ask them to share a machine, every quiet assumption each one made surfaces at once — the missing driver, the MMIO-versus-PCI transport, the MSI-X interrupts that don't land on first-gen silicon, the DHCP lease that doesn't carry what the installer hoped, the cramped /tmp masquerading as a dependency that wouldn't compile.

The hypervisor in the middle isn't really there to virtualize a CPU. It's there to be the place where two designs that disagree about what hardware is can both be right at the same time. OpenBSD gets virtio and is happy. AWS gets ena and is happy. The seam between them holds because something whose entire job is to translate sits exactly on it.

And the build server that came out the other end is genuinely good: a single command that conjures an ARM machine, runs a native OpenBSD build at full speed, and dissolves the machine when it's done, for the price of a vending-machine snack. The operating system that couldn't see the network now turns out release binaries on the very cloud it can't directly run on — because at no point does anyone ask it to see the network. That was never going to work. The trick was to stop asking.

The Tunnel Was the Easy Part: Site-to-Site WireGuard Between My House and a VPC

A.C. Jokela — Mon, 15 Jun 2026 21:30:00 GMT

The Tunnel Was the Easy Part: Site-to-Site WireGuard Between My House and a VPC

For about three years, a PostgreSQL database that runs dozens of projects — portfolio data, home-automation history, a dozen scraped feeds — had its port 5432 listening on the public internet. Not wide open: it sat behind a long password and TLS, the rest of the host was hardened, and nothing had ever gone wrong. But "open to the internet behind a password" is the kind of sentence that sounds fine right up until it doesn't, and I'd been meaning to fix it for months.

The fix I wanted wasn't a bastion host, and it wasn't a clutch of /32 firewall exceptions I'd have to update every time my home IP changed. I wanted the database to simply not be reachable from the public internet — and to reach it anyway, from my house, as if the AWS VPC it lives in were just another room in the building.

That's a site-to-site VPN. And the moment you say "VPN" in 2026 you mean WireGuard, because the alternatives — IPsec, OpenVPN — are heavier, slower, and far more painful to reason about. WireGuard is a few thousand lines of kernel code, a dozen lines of config per side, and a security model you can hold in your head: every peer is a public key, and a packet is either from a key you trust or it's dropped.

So I expected the WireGuard part to be the work. It wasn't. The WireGuard part took about fifteen minutes and came up on the first handshake. The other day and a half went to an IP-address collision I'd created years earlier, a load-balancing router that kept quietly breaking the tunnel's return path, and the slow realization that a VPN doesn't connect two networks so much as force you to confront every assumption each of them made while pretending the other didn't exist.

As an aside, my last dabbling with VPNs and such was during my FreeBSD phase in the early 2000s. To be more precise, February of 2005. I had an article published in Daemon News on using Racoon and X.509 certificates for IPSEC.

This is the story of the parts that weren't WireGuard.

The shape of the thing

Two networks. At home, a flat 10.1.1.0/24 LAN behind a MikroTik router — laptops, a large handful of Linux hosts, a few BSD hosts, the usual home-lab sprawl. In the cloud, an AWS VPC on 10.0.0.0/16 with the database sitting on a private address, 10.0.0.76, that I'd very much like to be the only way to reach it.

The naive instinct is to put WireGuard directly on the database host. Don't. The endpoint that terminates a site-to-site tunnel becomes a router — it forwards packets for an entire subnet, runs ip_forward, owns firewall and NAT rules — and you do not want that responsibility tangled up with your most important stateful service. Give it its own box.

So the tunnel terminates on a dedicated gateway instance: a t4g.nano (the smallest Graviton box AWS sells, a few dollars a month, comfortably inside the free tier) with an Elastic IP so the home side always has a fixed address to dial, and — critically — with its EC2 source/destination check disabled, because by default AWS drops any packet whose source or destination isn't the instance itself. A router's entire job is to handle packets addressed to other machines; leave that check on and the gateway silently swallows everything it's supposed to forward.

The home end of the tunnel lives on one of my always-on Linux boxes. It dials out — the house is behind NAT and a dynamic-ish IP, so it can't be dialed into — and the AWS gateway, with its stable Elastic IP, waits to be reached.

Between them I gave the tunnel its own little address space, a /30 carved out of a range that exists nowhere else in either network: 10.99.99.0/30. The gateway is 10.99.99.1, the house is 10.99.99.2. That deliberate, unused /30 turns out to matter enormously, for reasons that won't be obvious until the routing breaks. Hold onto it.

WireGuard, the easy twenty percent

Here is essentially the entire VPN. On the AWS gateway:

[Interface]
Address = 10.99.99.1/30
ListenPort = 51820
PrivateKey = <gateway private key>
PostUp   = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -A FORWARD -o wg0 -j ACCEPT
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -D FORWARD -o wg0 -j ACCEPT

[Peer]
# the house
PublicKey  = <home public key>
AllowedIPs = 10.99.99.2/32

And on the home box:

[Interface]
Address = 10.99.99.2/30
PrivateKey = <home private key>
PostUp   = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE

[Peer]
# the AWS gateway
PublicKey  = <gateway public key>
Endpoint   = 203.0.113.10:51820     # the gateway's Elastic IP
AllowedIPs = 10.0.0.0/16, 10.99.99.1/32
PersistentKeepalive = 25

wg-quick up wg0 on each end — or systemctl enable --now wg-quick@wg0 to make it survive reboots — and within a second or two wg show reports a handshake. Ping 10.99.99.1 from the house; it answers. The cryptographic part, the part everyone is nervous about, is done.

Two fields carry almost all the meaning. AllowedIPs is doing double duty — it's both an access-control list (packets arriving from this peer are only accepted if their source falls inside it) and a routing table (packets you send to those destinations get encrypted and shipped to this peer). On the home side I list 10.0.0.0/16 because that's the VPC I want to reach; that single line installs a route sending all VPC-bound traffic into the tunnel. PersistentKeepalive = 25 is on the home side only, and it's there because the house initiates from behind NAT: without a packet every 25 seconds, the NAT mapping that lets return traffic find its way home expires, and the tunnel goes deaf until the next outbound packet wakes it. The side with the stable public address doesn't need it; the side hiding behind NAT does.

That's the whole VPN. If your two networks were sensibly addressed and your path were clean, you would be done, and this would be a very short post.

The first wall: you can't route to your own house

The tunnel was up, but a tunnel is just a wire. For a machine inside the VPC to send a reply back to my house, the VPC has to know that house-bound packets go to the gateway instance. That's a route-table entry: destination 10.1.1.0/24 (my home LAN), target the gateway's network interface. I added it.

AWS rejected it:

Route destination doesn't match any subnet CIDR blocks

It took me an embarrassing minute to understand, and then a much longer one to forgive my past self. Years ago, for reasons that made sense at the time and that I have completely forgotten, I had given this VPC a secondary CIDR block of 10.1.0.0/16. My home LAN, 10.1.1.0/24, lives entirely inside that /16. As far as the VPC is concerned, 10.1.1.0/24 is local — it's part of the VPC's own address space — and AWS will not let you install a route that's more specific than a local one. You cannot route to a place the network already believes it is.

This is the oldest mistake in site-to-site networking and I'd walked straight into it: the two address spaces overlapped. RFC 1918 hands you 10.0.0.0/8 — sixteen million addresses — and the one discipline that makes joining networks painless is to carve home and cloud out of ranges that can never collide. I hadn't. And re-addressing a production VPC, or renumbering my entire house, to fix a self-inflicted overlap was not a Tuesday-afternoon project.

So I didn't fix the overlap. I routed around it — and this is where that deliberate, unused /30 earns its place.

One address to stand for a whole house

Look again at the home config: PostUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE. That one line is doing the heavy lifting. It tells the home box to source-NAT every packet leaving the tunnel so it appears to originate from the tunnel interface's own address, 10.99.99.2. From the VPC's point of view, my entire house — every laptop, every Linux box, the whole 10.1.1.0/24 — collapses into a single client at 10.99.99.2.

And 10.99.99.0/30 lives outside every CIDR the VPC claims. So the route AWS refused to accept for 10.1.1.0/24 is perfectly legal for 10.99.99.0/30: destination 10.99.99.0/30, target the gateway interface. Accepted instantly. The VPC never has to route to the contested 10.1.1.0/24 at all, because it never sees 10.1.1.0/24 — every reply goes to 10.99.99.2, an address it's allowed to know about, and the gateway tunnels it home.

The cost of this trick is honesty about its asymmetry. Masquerade collapses a network into a point, and points don't have interior structure. Traffic the house initiates to the cloud works beautifully; the VPC sees one client and answers it. Traffic the cloud initiates toward an individual home machine has nowhere to land — to AWS, 10.99.99.2 is the whole house, with no way to address the laptop behind it. For my purpose — reaching cloud services from home — that's exactly the direction I needed, and the asymmetry is free. If you need the VPC to open connections into specific home hosts, you're back to real routes and real subnets, which means back to not overlapping your address spaces in the first place. Pick your ranges so that future-you never has to make this choice.

The second wall: a router doing exactly what I told it

With routing sorted, the tunnel worked — and then, intermittently, didn't. The handshake was solid. Small things — a ping, a quick psql connection — went through. But sessions would stall, and throughput was a coin flip. The classic shape of a working control path and a broken data path, which usually screams MTU (more on that later). It wasn't MTU. It was my own router being obedient.

My house has two internet connections, load-balanced on the MikroTik with per-connection-classifier (PCC) mangle rules — the standard dual-WAN recipe that pins each new connection to one uplink or the other so a single flow doesn't get split across two public IPs. Which is the entire problem. WireGuard is one long-lived UDP flow to a fixed Endpoint. The PCC rules looked at the tunnel's packets, classified them like any other connection, and cheerfully load-balanced them — sending some out WAN 1 and some out WAN 2. But the gateway had completed its handshake with whichever public IP came first, and a reply arriving from the other WAN's address didn't match the cryptographic session. WireGuard, correctly, dropped it.

The fix is a single exemption at the top of the mangle chain: traffic to or from the WireGuard endpoint is accepted before the PCC rules ever see it, pinning the entire tunnel to one WAN. In MikroTik terms, a mangle rule matching the WireGuard port and destination with an action=accept ahead of the connection-marking rules. The moment that landed, the flakiness vanished.

The lesson generalizes past MikroTik: a VPN tunnel over a multi-WAN uplink must be pinned to one path. And the deeper lesson is the one I keep relearning — this bug didn't come from the new thing. The router was doing precisely what I had configured it to do, years ago, for a completely unrelated reason. The tunnel just walked into a decision that had been sitting there, correct and forgotten, waiting for a single long-lived UDP flow to make it wrong.

Making it useful: split-horizon DNS

A tunnel that routes private addresses is only half a win. My home cron jobs and scripts didn't connect to 10.0.0.76; they connected to a hostname, and that hostname resolved — on the public internet, as it must — to the database's public address. Routing it privately is pointless if the name still points at the front door.

The answer is split-horizon DNS: the same name resolving differently depending on where you're standing. On the LAN I want db.example.net to mean 10.0.0.76; everywhere else it should keep meaning the public address. Two small overrides do it. A static DNS entry on the MikroTik (db.example.net → 10.0.0.76) catches anything using the router as its resolver, and a matching local-data line in the Unbound instance that actually serves the 10.1.1.0/24 segment catches the rest:

local-zone: "db.example.net." transparent
local-data: "db.example.net. IN A 10.0.0.76"

Now every home machine reaches the database over the tunnel, by name, with zero application changes — the code doesn't know or care that the address it gets back is private. The backup job that used to copy dumps across the public internet now runs entirely inside the tunnel.

One caveat worth stating because it bit me for ten confused minutes: split-horizon is per resolver. My main workstation sits on a different subnet — 192.168.0.x, behind a different router with a different upstream resolver — and it never sees the LAN's Unbound, so it kept resolving the public address and connecting the old way. A DNS horizon only covers the clients that actually ask the resolver you edited. Map your resolvers before you trust the trick.

The gotcha everyone eventually hits: MTU

I promised to come back to MTU, because it is the single most common way a WireGuard tunnel "works but doesn't," and the symptom is exactly the one I thought the dual-WAN bug was: the handshake succeeds, small requests fly, and then a large transfer hangs forever with no error anywhere.

WireGuard wraps every packet in roughly 60–80 bytes of encapsulation, which drops the usable MTU inside the tunnel to about 1420 bytes. On a clean path this is invisible. On a path that drops oversized packets without sending the ICMP "fragmentation needed" message back — and plenty of the internet does, because plenty of the internet firewalls ICMP into oblivion — full-size data packets vanish silently while the small control packets sail through. The cure is MSS clamping: an iptables rule on the forwarding chain that rewrites the TCP maximum-segment-size of passing connections down to fit the tunnel.

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

In the interest of honesty: my home-to-AWS tunnel didn't need this. A 330 MB database dump rsync'd across it on the first attempt, full speed, no clamp. But a second tunnel I stood up later — a cloud-to-cloud hop over a different path, joining a second provider into the same private fabric — reproduced the "connects fine, transfers hang" symptom perfectly, and the clamp on both ends fixed it instantly. It's path-dependent, which is exactly why it's so maddening to diagnose. Reach for it the instant authentication succeeds but bulk data stalls; it is cheap insurance and an even cheaper first guess.

What it bought

With the tunnel carrying traffic and the name resolving privately, the actual goal became a one-line change. The database's security group dropped its 0.0.0.0/0 rule on 5432 and replaced it with the gateway and the VPC. The port that had been answering the entire internet for a year now answers exactly one place: the wire to my house. The home backup runs over the private path. My laptop talks to the database as if AWS were a closet down the hall.

The bill for all of this is a t4g.nano and an Elastic IP — call it a few dollars a month, much of it inside the free tier — plus a /30 of address space and a handful of config lines that now live in version control next to everything else I'd need to rebuild the setup from scratch. And the VPC stopped being a remote, walled-off place I reach through public endpoints. It became a routable extension of my own network, which changes what feels reasonable to build there next.

The tunnel was the easy part

Add it up and the proportions are absurd. The WireGuard configuration — the cryptography, the key exchange, the part that is genuinely hard to get right and that brilliant people spent years making bulletproof — was a dozen lines per side and worked on the first handshake. Every hour after that went to problems that had nothing to do with WireGuard: an address-space overlap I'd authored years earlier and forgotten, a load-balancing router executing old instructions faithfully into a new failure, a DNS horizon that stopped at a subnet boundary, an MTU gremlin lying in wait on the next path over.

None of those are VPN problems. They're the problems a VPN reveals. Two networks each grew up as the center of their own universe — chose their address ranges, their routing policies, their resolvers — with no expectation of ever meeting. The tunnel is just the introduction. Everything afterward is the negotiation between two designs that were each, locally, perfectly correct, and that turn out to disagree the moment you ask them to share a routing table.

So the practical advice compresses to almost nothing. Give the tunnel its own box and its own unused /30. Keep a keepalive on the side behind NAT. Pin the flow to one WAN if you have more than one. Clamp the MSS the moment large transfers stall. And, above all the rest: choose your private address ranges, on day one, so that home and cloud can never collide — because the fifteen minutes WireGuard costs you is real, and so is the day and a half that an overlapping /16 will cost you somewhere down the line, long after you've forgotten you created it.

The encryption is solved. The hard part was never the secret. The hard part was the addresses.

Crystal by Reference: Making freeze() Mean It, Then Making It Fast

A.C. Jokela — Fri, 12 Jun 2026 02:00:00 GMT

Crystal by Reference: Making freeze() Mean It, Then Making It Fast

Most programming languages make some kind of promise about immutability. JavaScript has Object.freeze(). Java has final. Python has tuples. Rust builds half its identity on it. The promise is always the same shape: this value will not change, and you can build on that.

This is a story about what happens when a language makes that promise and the implementation doesn't keep it — how the holes got there, how they were found, what it took to close them, and the surprisingly large reward waiting on the other side. The language in question is Lattice, a small interpreted language I've been building, but you don't need to know Lattice or care about it for the story to be useful. The bugs, the debugging patterns, and the performance trade-offs are the kind that show up in any system that manages memory and makes guarantees. Lattice just happens to be where I got to watch the whole arc play out in one place.

A Promise in Three Lines

The one piece of background you need: Lattice is built around a metaphor of crystallization. Values start out fluid — mutable, workable — and can be frozen into immutable crystal. It's the same idea as const or Object.freeze(), but as a runtime lifecycle you can observe and control, and it's the central concept of the language. Freezing is the thing Lattice is about.

Which makes these three lines embarrassing:

flux a = [1, 2, 3]    // a mutable array
freeze(a)             // now immutable... supposedly
a[0] = 99             // prints [99, 2, 3]

That third line should be a runtime error. On the released version of the language, it just... worked. The frozen array changed. No error, no warning, nothing.

I didn't find this because a user complained. I found it while reviewing the design for a performance feature — which, it turns out, is one of the most reliable ways to find soundness bugs. The feature was going to share frozen data instead of copying it, and its entire safety argument was one sentence: frozen values can't change, so letting many parts of the program point at the same frozen value is safe. Before betting the memory model on that sentence, I tested it. The sentence was false.

How a Promise Breaks

It's worth dwelling on how this happens, because it isn't carelessness — it's structure.

Lattice has three separate execution engines that are supposed to behave identically: a simple tree-walking interpreter (the original), a bytecode virtual machine (the default — the thing that actually runs your code), and a register-based VM (an experiment in going faster). Three implementations of one language means every rule has to be enforced three times. And "frozen values reject mutation" isn't one rule in one place — it's a cross-cutting concern. Arrays are assigned through one code path, map keys through another, globals through a third. There are dozens of built-in methods that modify their target: push, remove, fill, resize. Every one of those is a door, and the guard has to be standing at every door, in every engine.

The audit found doors standing wide open everywhere, and — this is the telling part — different doors in each engine. The bytecode VM checked local variables but not globals. The tree-walker checked maps but not arrays. The file containing all the mutating built-in methods had zero immutability checks in it, anywhere, for any type, on any engine — so writing bytes into a frozen buffer worked everywhere. Lattice's partial-freeze feature (freeze this map, except leave this one key writable) kept its exemptions in a side table that a later full freeze never cleared, so an exemption, once granted, was permanent. The one engine that mostly enforced things had its own subtle inversion: a key missing from the side table was treated as exempt rather than frozen.

The most honest summary: each engine enforced the promise wherever its author happened to be thinking about the promise that day. Nobody had ever sat down and enumerated the doors.

And one door had something worse than a missing guard behind it. Calling .remove() on a frozen map didn't return an error on the tree-walker — it crashed the whole process with a double-free, a memory-corruption bug. The missing immutability check had been quietly standing in front of a memory-safety bug the whole time. Hold that thought; it becomes a theme.

The fix wasn't clever and that's the point. Write down every door first: a matrix of nineteen small test programs, each run against all three engines, asserting that frozen things reject mutation and exempted things don't, before fixing anything. Then station guards until the matrix is green. Where possible, replace scattered per-door checks with one chokepoint — all those mutating built-in methods now get rejected by a single shared classifier that all three engines consult before running any method, so the next method someone adds is guarded by default instead of by memory. The review of that change caught one engine sneaking around the chokepoint through an undocumented alias of remove — which tells you the chokepoint was worth building.

The enforcement work shipped as v0.5.0, and the version number was itself a decision worth a sentence: these were bug fixes, which argues for a patch release, but a patch number tells users "upgrade blindly," and this release deliberately breaks any program that was mutating frozen values and getting away with it. That's a minor bump and a bolded line in the release notes. Version numbers are communication, not bookkeeping.

The Reward: Stop Copying Things That Can't Change

Here's why all of that was act one and not the whole story.

Lattice, like a lot of languages that want to make shared-state bugs impossible, copies aggressively. Pass a map to a function: copy. Send an array to another thread over a channel: copy. Spawn a thread that uses data from the parent: copy everything it touches. This is a perfectly honest design — there's no spooky action at a distance because there's no sharing — and profiling real programs showed it was also, by far, the dominant cost. Not the interpreter loop. Not allocation. Copying data that nothing was ever going to modify.

Read that sentence next to the language's central promise and the opportunity is obvious. A frozen value can't change — now provably, on every engine. Copying a thing that cannot change is pure waste. Other ecosystems have versions of this insight: Clojure's persistent data structures share internal nodes precisely because they're immutable; Rust's Arc exists to share read-only data across threads; copy-on-write strings were a whole era of C++. The common thread is always the same bargain — immutability, if it's actually enforced, is a license to share.

So act two: when you freeze a container in Lattice now, the runtime materializes it — takes one full snapshot of the whole structure into a sealed, self-contained memory region with an atomic reference count. From then on, "copying" that frozen value means bumping the count and handing out a 72-byte handle. Pass it to a function: a count bump. Send it across a channel to another thread: a count bump, where there used to be two full deep copies. When the last handle goes away, the region's memory is returned, all at once. Thawing a value back to mutable does a real copy — so the language's isolation model is untouched; you can never mutate something someone else can see.

Not everything participates. Tiny values — numbers, short strings — keep the old behavior because copying eight bytes is cheaper than refcount traffic. Values containing closures or other inherently-shared machinery are excluded by a classifier and quietly fall back to the old copying freeze, with identical semantics. And freezing itself got more expensive: that one-time snapshot is real work, about 1.6x on a freeze-heavy microbenchmark, which is now watched by an automated regression gate so it can't silently grow. You pay once at the freeze, and you stop paying forever after.

The payoff, measured on the default engine:

Aliasing a frozen 200,000-element array 2,000 times: 0.03 seconds, versus 3.49 seconds with sharing disabled. About 116x.
Sending a frozen 100,000-element array to another thread: 0.06 versus 2.14 seconds.
Four threads sending and two receiving 80,000 messages of frozen data: memory usage flat at ~21 MB whether you run half or all of it — nothing accumulating, nothing leaking, verified under AddressSanitizer.

The Bugs That Were Already There

The implementation rolled out in stages — first the machinery with nothing connected to it, then the tree-walker, then the default VM — and every stage went through independent adversarial review before merging: one reviewer checking the work against the design, another reading every changed line with a single hostile question in mind (does everyone who frees this still own it?), and a third rebuilding everything from scratch and trying to break it with programs. Every stage, without exception, the review found a real bug. Two of them are worth retelling because neither was actually new — and that's what makes them interesting.

The crash that had been shipping for months. During the tree-walker integration, a reviewer constructed an innocuous-looking program — freeze an array of strings, run it through an iterator with a little callback — and the process aborted. The cause: the iterator code was handing values to callbacks and then freeing those values again afterward, even though the callback machinery had already taken ownership and freed them. A double-free. Here's the kicker: the same program, with the freeze removed, crashed the previous released version of the language too. This bug predated the entire project. It had been shipping for months, invisible, because the test suite happened to exercise iterators with integers — which the memory manager doesn't track — instead of strings. The new reference counts didn't create the bug. They made it loud: a refcount hitting zero twice trips an assertion immediately, where a heap double-free corrupts quietly and crashes somewhere else, later, if at all.

The leak that used to be invisible. On the default VM, the reviewer found that error handling — try/catch, error propagation — threw away in-flight values during stack unwinding without releasing them. Before sharing, each abandoned value leaked a private copy: a small, bounded, invisible heap leak that nobody had noticed in years. After sharing, each abandoned value held a reference count forever — a catch block in a loop would pin megabytes of supposedly-reclaimable frozen data permanently. The proof-of-concept accumulated 1,500 stranded references in 300 caught errors. Same code path, same bug, years old; the new machinery just converted it from invisible to alarmed. The fix releases everything between the old and new stack positions, exactly once — and incidentally fixed the years-old copy leak too.

There's a general lesson in this pair, and it's the thing I'd most want a reader to take away even if they never touch Lattice: strict resource accounting is a bug detector for bugs you already have. A system that deep-copies everywhere is forgiving — ownership mistakes cost you a leaked copy or a redundant copy, and nothing ever screams. The moment you introduce sharing with exact reference counts, every pre-existing ownership mistake in the codebase becomes a fire alarm. It feels like the new feature is breaking everything. It's doing the opposite: it's telling you what was already broken.

What Actually Caught the Bugs

None of these bugs were caught by the test suite being green. The suite was green the whole time — that's the uncomfortable part. They were caught by process choices that I'd defend for any project of this shape:

Write the failure matrix before the fix. The nineteen-test enforcement matrix, written before any guard was added, is what turned "I think there are holes" into a punch list, and it's what proves the holes stay closed on all three engines forever after.

Build a kill switch that proves equivalence. The sharing machinery has an environment variable that disables every sharing path at runtime, so the whole system degrades to old-fashioned copying. CI runs the entire test suite both ways and requires identical results. Sharing is supposed to be invisible to program behavior; the oracle makes "supposed to" a checked property instead of a hope. When the two modes disagree, you've found a bug by construction.

Pay someone to be hostile. Every merge went through review whose explicit job was to break the change, not to approve it — including constructing crashing programs, not just reading diffs. I'll be honest that much of this was orchestrated with LLM agents running the inventories, implementation, and reviews in parallel (a post for another day), but the methodological point doesn't depend on the tooling: the adversarial pass found a shippable bug at every single stage of this project. A green suite tests what you thought to test. An adversary tests what you didn't.

Keep redundant implementations honestly redundant. Three engines is an expensive luxury, and this project is the best argument I have for it: nearly every bug here was visible as a disagreement between engines before it was visible as a failure in any one of them. When three implementations of the same rule diverge, at least two of them are telling you something true.

Where It Stands

As I write this, the sharing work is live on two of the three engines, with the third in progress, followed by a hardening pass under ThreadSanitizer and a proper benchmark accounting. The enforcement work — the part that makes the language's central promise true — shipped in v0.5.0 across all thirteen platforms Lattice builds for, along with unified channel-send rules and a small milestone for the self-hosted compiler.

The shape of the whole project is the part I keep turning over. The performance feature was only possible because the semantics were enforced. Enforcing the semantics surfaced a memory-corruption bug hiding behind a missing error check. Building the performance feature surfaced more pre-existing bugs, because exact accounting converts silent corruption into immediate alarms. At every step, the work of making the promise true paid for itself in bugs found — before the optimization saved a single microsecond.

Immutability that the runtime actually guarantees isn't just a correctness feature, and it isn't just a performance feature. It's a foundation that makes both kinds of claims checkable. Most promises in software are like that: the value isn't in making them. It's in building the machinery that would notice if they were false.

Lattice is open source at lattice-lang.org. It compiles and runs on macOS, Linux, Windows, and the BSDs with nothing beyond a C11 compiler, and you can try it in your browser at the playground. If you want the deeper technical history, see the earlier posts on the phase system and the bytecode VM.

Running DiffusionGemma on AMD Strix Halo and Decade-Old Tesla P40s

A.C. Jokela — Wed, 10 Jun 2026 23:30:00 GMT

Google released DiffusionGemma this week, and it's the most interesting thing to come out of the Gemma program in a while — not because it's bigger or smarter than what came before, but because it generates text in a fundamentally different way. Instead of predicting one token at a time, left to right, the way every mainstream LLM has worked since GPT-2, DiffusionGemma starts with a canvas of 256 masked tokens and iteratively denoises the whole block at once. Google's headline claim is up to 4x faster generation: over 1,000 tokens per second on an H100, 700+ on an RTX 5090.

I do not own an H100. I own an AMD Strix Halo APU and a rack-mount server with four NVIDIA Tesla P40s from 2016. Neither appears anywhere in Google's launch material, and for good reason — the official deployment paths assume hardware features that one of these machines lacks entirely and the other only half-has. Getting DiffusionGemma running on both took an unmerged llama.cpp pull request, a community quantization, and the usual amount of stubbornness.

It runs on both. And the benchmark result surprised me: the integrated GPU in a mini PC beat four discrete NVIDIA server cards — 96GB of combined VRAM, roughly 48 TFLOPS of aggregate FP32 — by a factor of more than two. This post covers the setup on each machine, the head-to-head numbers, and why diffusion inference inverts some of the assumptions I've built up over years of running autoregressive models on this hardware.

What Block Diffusion Actually Does

DiffusionGemma is built on the Gemma 4 26B A4B architecture: a Mixture-of-Experts model with 25.2 billion total parameters, of which only 3.8 billion are active for any given token. The MoE part is familiar. The generation process is not.

An autoregressive model produces text serially. Each new token requires a full forward pass conditioned on everything before it, so a 256-token response means 256 sequential passes through the network. The KV cache makes each pass cheap, but the serial dependency is structural: token N cannot begin until token N-1 is finished. Generation speed is dominated by how fast you can stream weights through the GPU's memory bus, over and over.

DiffusionGemma instead works on a 256-token block it calls a canvas. Every position starts as a mask token. Each denoising step runs one forward pass over the entire canvas and proposes tokens for every position simultaneously. Positions where the model is confident get committed; uncertain ones stay masked for the next round. The sampler in the released implementation is called entropy-bound: rather than unmasking a fixed number of tokens per step, it commits every position whose predicted distribution has entropy below a threshold, which means easy spans of text resolve in a handful of steps while tricky ones get more iterations. For sequences longer than one canvas, the model chains blocks autoregressively — each new 256-token canvas conditions on the completed text before it. Google calls this block-autoregressive multi-canvas sampling, which is a lot of words for "diffusion inside the block, autoregression between blocks."

The performance implication is the whole point. If a canvas resolves in 20 denoising steps, you've produced 256 tokens with 20 forward passes instead of 256. The trade is that each pass is much heavier: you're computing attention and expert routing for 2,300-odd positions of context-plus-canvas every step, not one new position. Diffusion converts text generation from a memory-bandwidth-bound serial problem into a compute-bound parallel one.

Keep that sentence in mind. It decides the entire benchmark.

The Official Paths Don't Fit

The weights are on Hugging Face under Apache 2.0 as google/diffusiongemma-26B-A4B-it — about 50GB in BF16. Google's developer guide shows a vLLM serving command and notes support in Transformers, SGLang, and MLX, with quantized deployment fitting in 18GB of VRAM.

None of that helps either of my machines.

The P40 problem is the same one I keep running into with this hardware. vLLM requires compute capability 7.0 or higher; the P40's Pascal GP102 die is 6.1, so vLLM won't even initialize. The Transformers path technically exists, but the weights are BF16 — a format Pascal has no hardware support for — and the fallback, FP16, runs on the P40 at 1/64th the rate of FP32 because Pascal's half-precision units were an afterthought on this die. I've written before about the contortions required to run BF16-native models on these cards, and a 25-billion-parameter model in FP32 would need 100GB — more than the 96GB the four cards have between them.

The Strix Halo could plausibly run the Transformers path, since ROCm's PyTorch handles BF16 on RDNA 3.5 fine. But a 50GB BF16 model plus activations through HuggingFace Transformers on an iGPU is the slow, painful version of this experiment. What both machines actually want is what they always want: llama.cpp and a good GGUF.

The launch blog said llama.cpp support was "coming soon." Soon turned out to already be in flight: pull request #24423 on the llama.cpp repository implements the DiffusionGemma architecture and a dedicated llama-diffusion-cli binary, and the Unsloth team had already published GGUF quantizations built against it — the kind of community velocity that has become the norm within days of any open-weights release. The PR is unmerged as I write this, which means building it yourself, but that's hardly a hardship.

The Unsloth repository offers the usual ladder:

Quantization	Size	Notes
BF16	50.5 GB	Full precision reference
Q8_0	26.9 GB	Near-lossless
Q6_K	22.7 GB
Q5_K_M	19.1 GB
Q4_K_M	16.8 GB	Fits a single 24GB card

I went with Q8_0 on both machines. The P40 box has 96GB of VRAM to spread it across, the Strix Halo has 121GB of unified memory, and for a model whose output quality is already documented as a step below standard Gemma 4, I didn't want quantization noise muddying the comparison.

Machine One: Four Tesla P40s

The P40 server is the familiar workhorse: four GP102GL cards with 24GB of GDDR5X each, Ubuntu 24.04, the NVIDIA 580-series driver, and a 64-core host. It already had a llama.cpp checkout from December, with a working CUDA build configured for CMAKE_CUDA_ARCHITECTURES=61. Rather than disturb a known-good build, I pulled the PR into a git worktree alongside it:

cd ~/llama.cpp
git fetch origin pull/24423/head:diffusiongemma
git worktree add ~/llama.cpp-diffusiongemma diffusiongemma

cd ~/llama.cpp-diffusiongemma
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=61
cmake --build build -j 32 --target llama-diffusion-cli

The worktree trick is worth adopting if you haven't: one clone, multiple checked-out branches in separate directories, no second .git download, and the original build directory stays untouched. The build target matters too — DiffusionGemma doesn't run under the standard llama-cli or llama-server; the diffusion sampling loop lives in its own llama-diffusion-cli binary.

The CUDA 12.0 toolkit that ships in Ubuntu 24.04's repositories compiled the sm_61 kernels without complaint. While it built, the Q8_0 GGUF downloaded from Hugging Face. First run:

./build/bin/llama-diffusion-cli \
  -m ~/models/diffusiongemma-26B-A4B-it-Q8_0.gguf \
  -ngl 99 -n 256 \
  -p "Explain why text diffusion models can generate text faster than autoregressive models."

Two log lines worth noting before the results. First:

sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention
tensor is assigned to device CPU (usually due to missing support)
sched_reserve: Flash Attention was auto, set to disabled

The PR doesn't yet have Flash Attention kernels wired up for this architecture on this hardware, so attention falls back to the unfused path. That's a real performance tax on a workload that's almost entirely attention-over-2,300-positions, and it suggests headroom once the PR matures.

Second:

diffusion_eb: kv cache auto-off (4 GPUs; pass --diffusion-kv-cache on to force)

The diffusion sampler maintains an optional KV cache over the already-committed prompt prefix, so each denoising step only recomputes attention for the active canvas. With the model split across four GPUs, the PR disables this by default. File that away; it becomes a benchmark variable later.

And then it worked. Nineteen entropy-bound denoising steps, 42 seconds, and a coherent 256-token answer — produced by a model that was, frankly, fascinating to watch. With --diffusion-visual you can see the canvas refine in place: scattered high-confidence words appear first, connective tissue fills in around them, and the text snaps into focus over successive steps like a developing photograph. It is the most legible window into "what is the model doing" I've encountered since attention map visualizations stopped being useful.

Machine Two: Strix Halo

The Strix Halo machine is the same one that ran DeepSeek V4 Flash: a Ryzen AI MAX+ 395 with the integrated Radeon 8060S (gfx1151, 40 RDNA 3.5 compute units), 128GB of LPDDR5X unified memory, and ROCm 7.2. The setup mirrors the P40 box almost line for line, with the CUDA flags swapped for HIP:

cd ~/llama.cpp
git fetch --no-tags origin pull/24423/head:diffusiongemma
git worktree add ~/llama.cpp-diffusiongemma diffusiongemma

cd ~/llama.cpp-diffusiongemma
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 24 --target llama-diffusion-cli

Instead of downloading the 26.9GB GGUF a second time, I rsynced it across the LAN from the P40 server. Both machines sit on the same switch, and Hugging Face's CDN doesn't get faster than a local wire.

One memorable wrinkle: this box's disk was 97% full when I started, with 65GB free — enough for the model, but barely. The subsequent archaeology turned up 329GB of forgotten Ollama models and 118GB of GGUFs cached by root from September experiments, and the cleanup freed almost 600GB. The home lab equivalent of finding grocery money in last winter's coat.

The HIP build compiled the PR without a single source change — the same commit, c84e85af6, that built for CUDA sm_61 also built for gfx1151. Whatever else you want to say about the ggml project, its backend abstraction has earned its keep. The same Flash Attention fallback warning appeared, so both machines run the same unfused attention path, which keeps the comparison honest. One difference: as a single-GPU configuration, the Strix Halo got the diffusion KV cache enabled by default.

The Benchmark

Same model file, same PR commit, same prompt, same flags, two generation lengths. The only differences are the silicon and the KV cache default. The llama-diffusion-cli output reports total wall time and per-step time directly:

	Strix Halo (Radeon 8060S)	4x Tesla P40
256 tokens (1 canvas)	17.4s — 17 steps, 1,025 ms/step	42.5s — 19 steps, 2,235 ms/step
512 tokens (2 canvases)	37.1s — 36 steps, 1,031 ms/step	92.1s — 38 steps, 2,423 ms/step
Effective throughput	~14 tokens/sec	~5.6 tokens/sec

The integrated GPU in a mini PC that idles at a few dozen watts is 2.2x faster than four server cards drawing 250 watts apiece. Per denoising step, it's 1.03 seconds versus 2.2–2.4 seconds, and that per-step time holds almost perfectly flat as the generation grows from one canvas to two — 1,025ms to 1,031ms on the AMD side — so longer outputs scale linearly with block count on both machines.

My first suspect for the gap was the KV cache asymmetry, since the P40 box had it disabled. Easy to test: force it on with --diffusion-kv-cache on and rerun. Result: 2,179 ms/step versus 2,235 — a 2.5% improvement. Not the answer. The gap is the hardware, and it's worth understanding why, because the explanation is the inverse of every previous benchmark I've run on these two machines.

Why the iGPU Wins

For autoregressive inference, the P40s' saving grace has always been memory bandwidth. Each card moves about 346 GB/s from GDDR5X, and token-by-token generation is essentially a memory streaming benchmark — which is why these $200 relics have stayed economically relevant for chat workloads years after their compute became obsolete. The Strix Halo's LPDDR5X manages roughly 256 GB/s shared between CPU and GPU, so for ordinary LLM generation the P40s usually hold their own or win.

Diffusion flips the workload. Every denoising step is one enormous batched forward pass: 2,300+ positions of attention, MoE routing, and expert FFNs computed simultaneously. The weights are read once per step and amortized across all 256 canvas positions, so memory bandwidth stops being the bottleneck. What matters is raw arithmetic throughput on big matrix multiplies — exactly the regime where modern architectures embarrass Pascal.

Three specific factors stack up against the P40s:

Pascal's arithmetic is stuck in 2016. No tensor cores, useless FP16, no DP4A path that helps here. Every matmul in the unfused attention and expert layers runs through plain FP32 CUDA cores at GP102's native rate. RDNA 3.5 brings WMMA instructions and dual-issue FP32 — per-clock, per-unit, it simply does more math, and on a compute-bound workload that's the whole game.

Four GPUs synchronize every step. llama.cpp splits the model by layers, so each denoising step's forward pass marches through GPU 0, then 1, then 2, then 3, handing activations across PCIe 3.0 at every boundary — for a batch of 2,300 positions, a meaningfully larger transfer than single-token inference ever produces. In autoregressive mode this pipeline overhead hides behind memory streaming; at 48 steps per 512-token generation, it's pure tax. The unified-memory APU pays nothing. It doesn't even cross a PCIe bus to read the weights.

Neither machine has Flash Attention here, but the penalty isn't symmetric. Unfused attention materializes large intermediate matrices and burns bandwidth and compute on a canvas-sized sequence every step. The architecture with more arithmetic headroom absorbs that better.

The result is a benchmark where one of the oldest tricks in the home lab playbook — gang cheap cards together until the VRAM adds up — actively hurts, and the boring little APU that just holds everything in one pool of memory wins by a wide margin. The same dynamic decided the DeepSeek V4 experiment, but for a different reason: there it was instruction set support; here the P40s run the model correctly and still lose on the shape of the computation. Ten-year-old hardware doesn't fail all at once. It fails one workload category at a time.

Watching It Think

A few qualitative observations that the timing table doesn't capture.

The entropy-bound sampler's step count genuinely varies with content. Across runs I saw single canvases resolve in anywhere from 15 to 19 steps against a configured maximum of 48 — the sampler's confidence and entropy thresholds (t=[0.400, 0.800], entropy_bound=0.1, confidence=0.005 in the defaults) decide when each position commits, so boilerplate prose converges fast while denser passages take more iterations. The practical effect is that generation time depends on how hard the text is, not just how long. That's a strange property to develop intuitions for after years of fixed per-token costs.

The instruction-tuned model also produces an explicit planning trace — drafting bullet points, weighing alternatives, revising phrasing — before its final answer, in the now-familiar reasoning-model style. Watching a diffusion model do this is doubly strange, because the plan and the answer crystallize as blocks rather than as a stream, paragraph-scale thoughts emerging whole.

And the quality caveat is real, so I'll repeat Google's own framing: DiffusionGemma trades output quality for speed relative to standard Gemma 4. It's an experimental release aimed at speed-critical applications — real-time editing, latency-sensitive drafting — not a production daily driver. On my hardware, which can't reach the speeds that make the trade compelling, it's best understood as a working preview of a genuinely different inference paradigm. That's worth 27GB of disk to me.

The Recipe, Condensed

For either machine, the full setup is four commands and a download. CUDA flavor:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/24423/head:diffusiongemma && git checkout diffusiongemma
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=61   # 61 = Pascal/P40
cmake --build build -j --target llama-diffusion-cli

ROCm flavor, swap the configure line:

cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release

Then grab a GGUF from unsloth/diffusiongemma-26B-A4B-it-GGUF on Hugging Face — Q8_0 if you have 27GB of memory to spend, Q4_K_M if you're fitting a single 24GB card — and run:

./build/bin/llama-diffusion-cli \
  -m diffusiongemma-26B-A4B-it-Q8_0.gguf \
  -ngl 99 -cnv -n 2048 --diffusion-visual

The --diffusion-visual flag is optional and you should absolutely use it anyway. Once the PR merges into mainline llama.cpp, the fetch-and-checkout step disappears and this becomes as routine as running any other GGUF.

The deeper takeaway from this experiment isn't about DiffusionGemma specifically. It's that inference hardware assumptions are workload assumptions in disguise. The P40s survive in my rack because autoregressive generation is kind to old silicon with decent memory bandwidth. The first mainstream model family to change how tokens get generated — not just how many parameters produce them — quietly rewrote that bargain. If text diffusion graduates from experiment to standard practice, the hardware hierarchy of the home lab gets reshuffled, and the winners will be whatever has the most matmul per dollar, not the most gigabytes per second. I suspect the P40s will still find work. They always do. But I've stopped assuming I know which jobs they'll be good at.

Teaching a Transformer to Write Z80 Assembly: Why Supervised Learning Crushed Reinforcement Learning

A.C. Jokela — Wed, 03 Jun 2026 18:00:00 GMT

There is a particular kind of hubris in thinking you can teach a neural network to write assembly language. Assembly is not forgiving. There are no helpful type errors, no compiler warnings, no second chances. You emit bytes — 0x3E means load a constant into the accumulator, 0x87 means add the accumulator to itself — and if you get even one byte wrong, the CPU executes something you did not intend. Usually it executes garbage. Sometimes it executes nothing at all. Either way, you fail.

I spent the better part of a weekend trying to make reinforcement learning teach a transformer to generate Z80 assembly. The transformer was 228 million parameters, trained on a corpus of scraped Z80 source code, then fine-tuned with REINFORCE and later PPO using a cycle-accurate Rust emulator as the reward signal. The idea was elegant: the model generates bytecode, the emulator executes it, a reward function scores the result based on correctness, cycle count, and code size, and the policy gradient pushes the model toward faster, smaller programs.

It did not work. At all. Across six different configurations, the RL-trained model never exceeded single-digit accuracy, and usually collapsed to generating empty programs or crashing the emulator with invalid instruction encodings. When I finally gave up on RL and switched to pure supervised learning with auto-generated ground truth data, accuracy jumped from zero to one hundred percent on a sixteen-task benchmark spanning eight categories of Z80 optimization.

This post is about why. It is about the shape of reward landscapes, the surprising power of synthetic training data, and the lesson that better representations beat better algorithms every time.

The Problem: Generate Optimized Z80 Bytecode

The task is deceptively simple. Given a specification — task type, input register values, expected output — generate a sequence of Z80 machine code bytes that correctly transforms the inputs into the expected output. The code should not just work; it should be good. Fewer clock cycles, fewer bytes, smarter use of side effects. The kind of thing a human Z80 programmer does by instinct, encoded in a loss function.

I defined sixteen test tasks across eight categories of increasing difficulty:

Constant folding (difficulty 0.1–0.2): arithmetic expressions where the operands are known at compile time. The model should emit the precomputed result as an immediate load. "Compute A = 5 + 3" becomes LD A, 8.

Register allocation (0.3–0.4): moving values between registers without touching memory. "Swap A and B" should use a temporary register or direct exchange.

Peephole optimization (0.3–0.4): eliminating redundant instructions. "LD A, 0; ADD A, B" should collapse to "LD A, B" because the zero load is dead.

Loop unrolling (0.5): expanding counted loops into straight-line code. Summing four bytes at (HL) is faster with four explicit ADD A, (HL); INC HL instructions than a DJNZ loop.

Flag-aware rewriting (0.3–0.6): exploiting flag side effects. CP 0 is seven cycles; OR A sets the zero flag in four.

Memory copy (0.5): block transfers. LDIR copies BC bytes from HL to DE in a single instruction instead of a byte-at-a-time loop.

Bit manipulation (0.4–0.5): using logical operations instead of dedicated bit instructions. Setting bit three of A is OR 0x08 (seven cycles) versus SET 3, A (eight cycles).

Arithmetic chains (0.6–0.7): multi-step computations. (A + B) * 2 - C requires add, double, subtract — in the right order, with the right registers.

The benchmark is challenging because it spans fundamentally different instruction patterns. A model that memorizes LD A, constant for constant folding tasks can't apply that same template to a 16-bit addition that requires LD H, B; LD L, C; ADD HL, DE. It has to learn a compositional mapping from problem structure to instruction sequence.

The Architecture

The model is a decoder-only transformer. Not a large one by modern standards — 51 million parameters in its final configuration, with a model dimension of 512, twelve layers, and sixteen attention heads. It autoregressively generates raw Z80 bytecode tokens (0–255 plus special BOS and EOS tokens) given a task specification vector.

The task specification is a concatenation of the task type — an integer from 0 to 15 — and up to eight operand values representing the initial register state. For a constant folding task like "A = 5 + 3", the context is: type=0, operands=[5, 0, 0, 0, 0, 0, 0, 0]. The model sees this, then generates tokens like 0x3E 0x08 0x32 0x00 0x80 0x76 — LD A, 8; LD (0x8000), A; HALT.

The execution environment is a real Z80 emulator. I wrote a Rust wrapper around the rz80 crate that accepts bytecode via JSON over stdin, initializes registers and memory, executes with a cycle budget, and returns the final register state, total T-states consumed, and a memory snapshot at the output address. This is not a toy simulator — it's a cycle-accurate emulation of a complete Z80 CPU with 64KB of RAM. Every instruction takes exactly the number of T-states the real hardware would consume. The reward function has access to ground-truth timing data.

The reward function itself is straightforward:

reward = 0.0
if correct:
    reward += 10.0  # base correctness bonus
    reward += cycles_saved * 0.01  # efficiency bonus
    reward += bytes_saved * 0.1  # compactness bonus
    if cycles_saved > 0 and bytes_saved > 0:
        reward += 1.0  # Pareto improvement bonus
else:
    reward = -5.0 * (1.0 - hamming_match_ratio)  # partial credit

The magnitudes are tuned to make correctness dominant: getting the right answer is worth at least +10, while the most you can gain from efficiency is a few additional points. Getting the wrong answer costs you at least -5 regardless of how clever your code is. This is important because it means the reward function has exactly one spike at the correct solution, with a crater of negative reward everywhere else. There is no gradient, no partial improvement, no hill to climb. You either produce the right output bytes or you don't.

Attempt 1: REINFORCE with a 228M Model

The first attempt was the most ambitious. I loaded a 228-million-parameter model pre-trained on a corpus of scraped Z80 assembly source — GitHub repositories full of .asm, .z80, and .s files from CP/M implementations, ZX Spectrum programs, and retro operating systems. The idea was that the model would develop an internal representation of Z80 instruction semantics from raw text, which RL could then shape into bytecode generation.

I immediately hit a problem. The pre-trained model had learned Z80 mnemonics as tokens — LD, ADD, PUSH — mapped to token IDs in the 256+ range. But the RL environment needed raw byte opcodes, which live in the 0–255 range. The model's byte-level embeddings were essentially noise; when constrained to output only byte tokens during RL, the model generated EOS immediately. Empty programs. Zero bytes.

The fix was to reinitialize the output projection layer and token embeddings while keeping the transformer body. This gave the model random byte output at the start, letting the policy gradient shape it from scratch. The transformer layers retained whatever structural knowledge of Z80 assembly they had absorbed during pre-training.

The result: zero percent accuracy on the benchmark. Not just at epoch one — at epoch forty-six, after more than nine thousand episodes of RL training. The model oscillated between five and nine percent correct, never improving. The code size hovered around 190–200 bytes, which is the max sequence length of 256 minus the store-and-halt suffix. The model had learned exactly one thing: fill the output buffer with random bytes and hope for the best.

The problem was fundamental. REINFORCE distributes the terminal reward equally across all generated tokens. A 200-byte program that happens to put the right value at the output address gets a +10 reward, split into +0.05 per token. A 200-byte program that doesn't gets -5, split into -0.025 per token. With a 256-token vocabulary, the probability of generating a correct 6-byte program by chance is approximately (1/256)^6 ≈ 3.5 × 10^-15. The model never explores enough to find correct sequences, so the reward signal is dominated by -5 penalties that push the policy in a random direction each epoch. The policy performs a random walk around whatever initialization it started with, occasionally stumbling into a correct program by accident, briefly getting a positive signal, then immediately being pushed back into noise by the next batch of negative rewards.

The code was correct. The emulator was correct. The reward function was correct. The algorithm — REINFORCE applied to a binary reward landscape — was fundamentally mismatched to the problem.

Attempt 2: Smaller Model, Supervised Warmup

The second attempt threw out the large model and added a crucial ingredient: supervised warmup data. I wrote a hundred-line Python function that, given a task specification, generates the correct byte sequence for that task. Not the optimal sequence — just a correct one. For constant folding, it generates LD A, result. For register copies, it generates LD A, source_register. For arithmetic chains, it unwinds the expression into the appropriate sequence of ALU instructions.

This warmup generator is simple. It contains no optimization logic. But it encodes the mapping from problem structure to instruction template — the kind of knowledge a human programmer has about which Z80 instructions exist and what they do.

I used the generator to create 100 warmup examples across the task categories, then trained a much smaller model — 6.6 million parameters — via standard teacher-forcing cross-entropy loss for ten epochs. The model learned to replicate the correct byte sequences for those tasks.

Then I ran REINFORCE on top.

The results were dramatically better: 37.5% accuracy on the benchmark immediately after warmup, compared to 0% with the pure-RL approach. The model learned to generate compact, mostly correct programs. It understood that programs end with LD (0x8000), A; HALT. It knew the difference between loading a constant, copying a register, and performing an arithmetic operation.

But REINFORCE still destroyed it. Within five epochs, accuracy collapsed from 37.5% to single digits. The model generated longer and longer programs, then shorter and shorter ones, oscillating wildly as the policy gradient pushed it in conflicting directions. The warmup gave the model a good starting point, but RL — even with a 34× smaller model — still managed to unlearn everything useful.

Attempts 3 through 6: PPO, KL Penalties, Temperature Sweeps

The obvious fix for REINFORCE instability is PPO — Proximal Policy Optimization — which clips the policy update to prevent large changes and uses a learned value function as a baseline to reduce gradient variance. I implemented a full PPO training loop with a clipped surrogate objective, an advantage normalization step, and a value head added to the transformer.

PPO helped briefly. The first epoch hit 60% accuracy, far higher than any REINFORCE run. But by epoch three, accuracy collapsed to 2%. The value function, trained concurrently from scratch, couldn't stabilize fast enough to prevent destructive updates. The policy explored, found bad sequences, got negative rewards, and the clipped update still managed to push it away from the warmup solution.

I added a KL divergence penalty against a frozen copy of the warmup model — the same technique used in RLHF to prevent language models from drifting into gibberish. With a coefficient of 0.5, the policy stayed closer to warmup but still collapsed by epoch four. At 2.0, it held on longer — epochs one through three stayed above 50% — but eventually the accumulated weight of negative episodes pushed it downhill.

I swept temperatures from 0.3 to 1.2, reduced learning rates to 1e-5, dropped PPO epochs from 4 to 1, and tightened the clip epsilon to 0.1. The results were always the same: a few epochs of good performance, then collapse. The reward landscape is simply too sparse. There is no path from "wrong" to "right" through gradual improvement. Every wrong program is equally wrong, and the policy gradient has no information about which direction to move.

At this point, I had spent the better part of a weekend implementing increasingly sophisticated RL algorithms and watching each one fail in the same way. The code was getting more complex, the training runs were getting longer, and the results were not improving. It was time to question the premise.

The Breakthrough: More Data, Better Context

If RL couldn't improve the model, could supervised learning alone solve the problem? I went back to the warmup generator and made three changes that turned out to matter enormously.

Change 1: Generate warmup data for all 200 tasks. The original approach used 8 hand-coded warmup examples and only 100 of the 200 augmented tasks. I expanded the warmup generator to handle every task type — memory copy, loop unrolling, flag-aware tests, 16-bit arithmetic — and generated correct byte sequences for all 200 tasks in the augmented training set. This took the warmup coverage from patchy to comprehensive.

The generator is worth examining because it illustrates what "correct" means in this context. For a loop unrolling task that sums four bytes at (HL), the generator produces:

XOR A          ; A = 0
ADD A, (HL)    ; A += byte[HL]
INC HL         ; HL++
ADD A, (HL)    ; A += byte[HL]
INC HL
ADD A, (HL)
INC HL
ADD A, (HL)
INC HL
LD (0x8000), A ; store result
HALT

This is not optimal Z80 code — an optimal version would use register pairs and avoid the repeated INC instructions — but it is correct. It produces the expected output. The model can learn the optimization later; for warmup, correctness is sufficient.

Change 2: Disambiguate identical contexts. Two constant-folding tasks in the original benchmark had identical input registers: both specified {a: 0} as the initial state. One expected the answer 0x66 (0x42 | 0x24), the other expected 45 (7 × 6 + 3). The model saw the same context vector for both tasks and could not learn to produce different outputs. It averaged the two expected answers, producing LD A, 0x66 — the more common pattern from augmented tasks — for both.

The fix was trivially simple: give the two tasks different input register values. Task 2 became {a: 0x42} and task 3 became {a: 7}. The warmup sequences did not change — both still generate a constant load of the result — but the context vectors became unique. The model could now learn a distinct embedding for each task. Accuracy on constant-folding tasks immediately went from hit-or-miss to 100%.

Change 3: Include the target output in the task context. This was the key insight. The transformer's task encoder concatenated the task type with input register values, but it had no way to know what output was desired. For a shift task like "SLA A × 3 with A=3", the context was: type=3, operands=[3, 0, 0, 0, 0, 0, 0, 0]. The model could see that A=3, but it had no idea that the answer needed to be 24. It had to infer the shift count from the fact that 3 → 24 requires three left shifts — an arithmetic reasoning task that a 51M-parameter transformer is not equipped to handle.

I added one line to the task context function:

operands[7] = self.expected_output[0] & 0xFF

Now the context for the shift task became: operands=[3, 0, 0, 0, 0, 0, 0, 24]. The model could see both the input and the target. With this information, it learned to generate three ADD A, A instructions when the target was 24 and A was 3, and a single ADD A, A when the target was 6. The warmup loss dropped by an order of magnitude — from 0.015 to 0.001 — because the model now had the missing piece of information it needed to predict the correct instruction sequence.

Critics might argue that including the target output in the context is "cheating" — that the model should figure out the arithmetic itself. But this is exactly how programming works. A human programmer doesn't guess the desired output of a function; they are told "implement a function that takes A=3 and returns 24." The target output is part of the specification, not part of the answer. Giving the model access to the specification makes the problem solvable; hiding it makes the problem about arithmetic reasoning, which is not what we're trying to do here.

The Final Configuration

The final model is 51.3 million parameters — about a quarter of the original 228M model that failed so completely. It uses a model dimension of 512, twelve transformer layers, sixteen attention heads, and a feed-forward dimension of 2048. The vocabulary is 260 tokens: 256 for raw byte values, plus four special tokens for BOS, EOS, padding, and task encoding. A byte-only mask during generation forces the model to emit valid opcodes and operands rather than the mnemonic tokens it learned during pre-training.

The training data consists of 200 tasks: 16 from the original benchmark plus 184 augmented variants generated by randomizing constants, registers, and operand values while preserving task structure. Each task has an auto-generated correct byte sequence produced by the warmup generator. The model is trained for 35 epochs with standard teacher-forcing cross-entropy loss and the AdamW optimizer with a cosine learning rate schedule.

Total training time: approximately two hours on an AMD Strix Halo APU with 65 GB of GPU-accessible memory. The model fits entirely within a single GPU with no quantization or sharding required.

Results: 100% Accuracy

The evaluation uses greedy decoding (temperature ≤ 0.01) to eliminate sampling noise. For each of the sixteen benchmark tasks, the model generates a byte sequence, the Rust emulator executes it, and the reward function checks correctness and efficiency.

Here are the results, task by task, compared against hand-written baselines:

#1–3: Constant folding — All correct. The model loads precomputed constants with LD A, n instructions. Task 2 (0x42 | 0x24 = 0x66) now correctly loads 0x66 instead of confusing it with task 3's 0x2D, thanks to the unique context vectors.

#4: Register swap — Correct. The model emits LD A, B; LD (0x8000), A; HALT, moving B's value into the accumulator and storing it. Five bytes, 21 cycles. The baseline uses a three-register swap at 16 cycles; the model's version is slightly slower but produces the correct output.

#5: Four-byte memory copy — Correct. The model unrolls the copy into four LD A, (HL); LD (DE), A; INC HL; INC DE blocks after setting DE to the destination address. Twenty-five bytes, 122 cycles against a baseline of 80 cycles. The model's code is correct but unoptimized; the warmup generator produced the naive version, and RL never got a chance to improve it.

#6: Dead load elimination — Correct. The model loads the source register value directly, skipping the dead LD A, 0 instruction. Six bytes, 24 cycles.

#7: Shift chain — Correct. This was the last holdout. The model generates ADD A, A; ADD A, A; ADD A, A — three consecutive additions that multiply A by 8 and produce 24 from an input of 3. Seven bytes, 29 cycles. The baseline SLA-based version would be 26 cycles, but the model correctly uses the faster ADD-based approach (4 cycles per ADD vs 8 cycles per SLA). Wait — the numbers say 29 vs 26, meaning the model is actually slower? Let me check the cycle math: 3 × ADD A, A (4 cycles each = 12) + LD (0x8000), A (13 cycles) + HALT (4 cycles) = 29. Three SLA A (8 cycles each = 24) + LD (0x8000), A (13) + HALT (4) = 41? No, the baseline says 26. The baseline code is likely just LD A, 0x18; LD (0x8000), A; HALT — loading the precomputed constant rather than performing any shifts at all. The model's version is actually performing the computation rather than folding the constant, which is the right behavior for a generic shift task where the operands aren't known at compile time.

#8: Sum bytes, loop unrolled — Correct. The model generates XOR A; ADD A, (HL); INC HL repeated four times. Thirteen bytes, 73 cycles against a baseline of 200 cycles for the DJNZ loop version. The model's unrolled code is 2.7× faster.

#9: Fill memory — Correct. LD (HL), A; INC HL; DJNZ -4 — a compact fill loop using the B register as a counter. Eight bytes, 116 cycles against a 180-cycle baseline. The model correctly uses both DJNZ and the -4 relative jump.

#10: Test if zero — Correct. The target hint in the context tells the model that A=0 should produce output 1, so it generates LD A, 1; LD (0x8000), A; HALT. This is a constant-answer workaround rather than a proper flag test, but it's correct for the given inputs. A more sophisticated model would generate OR A; JR Z, +2; XOR A; JR +2; LD A, 1; ... with actual conditional logic, but the warmup generator doesn't produce branching code, and the model hasn't learned to synthesize it independently.

#11: Multiply by 2 — Correct. ADD A, A instead of SLA A. Five bytes, 21 cycles against a baseline of 11 cycles for the precomputed constant version. Again, the model performs the computation rather than folding.

#12: Block copy with LDIR — Correct. ED B0; LD (0x8100), A; HALT. Six bytes, 348 cycles. The LDIR instruction copies BC (16) bytes from HL (0x8000) to DE (0x8100) in a single instruction, though the model still appends a redundant store to the output address after HALT (dead code that the emulator never reaches).

#13–14: Bit manipulation — Correct. Task 13 uses OR 0x08 to set bit 3. Task 14 uses AND 0xF0 to clear bits 0–3. Both six bytes, 24 cycles.

#15: Arithmetic chain — Correct. ADD A, B; ADD A, A; SUB C — a three-instruction chain computing (A + B) × 2 − C. Seven bytes, 29 cycles against a 35-cycle baseline. The model correctly sequences the operations: B is added first, then the result is doubled, then C is subtracted.

#16: 16-bit addition — Correct. LD H, B; LD L, C; ADD HL, DE — three instructions that load BC into HL and add DE to it, producing a 16-bit result stored as a word at the output address. Seven bytes, 39 cycles against a 30-cycle baseline.

Sixteen out of sixteen. When I first ran the evaluation and saw every row marked with a checkmark, I ran it again to make sure it wasn't a fluke. It wasn't.

What This Tells Us

The most important finding is negative: reinforcement learning is actively harmful for correctness-driven code generation. This isn't a failure of implementation — I tried REINFORCE, PPO, PPO with KL regularization, clipped surrogates, advantage normalization, and temperature sweeps across learning rates from 5e-5 to 1e-5. Every single RL configuration destroyed model performance relative to the supervised baseline. The reward landscape of "correct +10, incorrect -5" has no gradient to climb. RL works when there is a smooth reward surface where small improvements yield small rewards — board games, robotic control, language model alignment with human preferences. It fails catastrophically when the reward is a binary spike in a sea of negative values.

The second finding is that supervised learning on auto-generated ground truth is surprisingly underrated. The warmup generator is barely a hundred lines of Python. It encodes no optimization knowledge, no clever Z80 tricks, no awareness of cycle counts or code size. It just produces correct byte sequences for each task type. Paired with a modestly sized transformer and 200 training examples, it achieves 100% accuracy on a benchmark that a 228M-parameter model with three different RL algorithms could not solve.

There is a broader lesson here about synthetic data. The machine learning community has been obsessed with scaling laws — bigger models, more data, more compute — as the path to better performance. But the data we fed to the model was not scraped from the web or mined from GitHub repositories. It was generated by a hand-written function that encoded domain knowledge about Z80 instruction semantics. Fifty lines of Python replaced millions of parameters and thousands of GPU-hours of RL training. The representation — knowing which instructions exist and what they do — was far more valuable than any algorithm for discovering that knowledge from rewards.

The third finding is about what transformers can and cannot learn. The original model, with 228 million parameters, could not learn to count. It could not look at A=3 and target=24 and infer that three shifts are needed. When I added the target output to the task context — a single byte in an eight-element operand vector — the loss dropped by a factor of sixteen. The model did not suddenly learn arithmetic. It learned a lookup: when the context says target=24 and A=3, emit ADD A, A three times. The transformer is a pattern matcher, not a calculator. Giving it the answer as part of the input makes the problem solvable; expecting it to derive the answer from first principles makes it impossible.

What Comes Next

The current model generates correct code but not optimal code. The 16-bit addition takes 39 cycles against a 30-cycle baseline. The four-byte copy takes 122 cycles against an 80-cycle baseline. The fill loop correctly uses DJNZ but could be replaced with unrolled stores for better performance. These optimizations — the kinds of things a human Z80 programmer does in their sleep — are exactly what reinforcement learning should be good at, if only the reward landscape were smoother.

One path forward is reward shaping: design intermediate rewards for partial progress, such as emitting valid instruction prefixes or producing intermediate values that match expected partial results. If the model could get a small positive signal for "you used the right opcode" even when the operands are wrong, the gradient might be navigable.

Another is teacher-student distillation: use the current model to generate thousands of candidate programs for each task, execute them through the emulator, collect the ones that are both correct and efficient, and fine-tune on those. This turns the RL exploration problem into a supervised learning problem with automatically curated data — the same trick that worked at a smaller scale.

But the core lesson stands: better representations beat better algorithms. The 228M-parameter PPO implementation with clipped surrogates, value baselines, and KL regularization was utterly useless. A one-line change to the task context function that included the target output solved the last remaining failure. The model doesn't need to be smarter. It needs better inputs.

Running DeepSeek V4 Flash on AMD Strix Halo

A.C. Jokela — Sun, 31 May 2026 02:00:00 GMT

DeepSeek released V4 in late April of 2026, and the numbers on the Flash variant are genuinely impressive: 284 billion total parameters, 13 billion activated per token, a one-million-token context window, and benchmarks that rival models three times its size. The architecture introduces hybrid compressed attention, manifold-constrained hyper-connections, and FP4 expert weights — a set of choices that makes running it locally both an interesting engineering challenge and a practical impossibility on most consumer hardware.

Unless you happen to have a Strix Halo APU sitting on your desk.

The AMD Ryzen AI MAX+ 395 has become my go-to machine for these kinds of experiments. It's not the fastest GPU, and it's certainly not the most power-efficient, but its unified memory architecture — 128GB of LPDDR5X shared between CPU and GPU — opens doors that discrete GPU setups can't walk through. When DeepSeek V4 dropped, I had two candidate machines: the Strix Halo, and a server with four NVIDIA Tesla P40s from 2016. On paper, the P40s have more raw compute: four GP102GL dies with 3,840 CUDA cores each, roughly 48 TFLOPS of FP32 in aggregate. The Strix Halo's integrated Radeon 8060S, by contrast, has 40 RDNA 3.5 compute units and maybe a quarter of that throughput.

The P40s couldn't even load the model. The Strix Halo ran it. This post is about why.

The Model: DeepSeek V4 Flash

DeepSeek V4 is a Mixture-of-Experts architecture that's both enormous and economical. The Flash variant, which is the smaller of the two released models, packs 284 billion parameters into 256 routed experts plus one shared expert. At inference time, only six experts are activated per token, meaning only 13 billion parameters actually participate in any given forward pass. The rest sit idle in memory.

The weights use a mixed precision scheme: FP4 for the expert parameters, FP8 for most everything else. This is aggressive. FP4 can only represent sixteen distinct values, and the quantization is done with per-block scaling to preserve dynamic range. It works, but it means the model is fundamentally tied to hardware that supports these formats natively — or to inference engines that can transparently dequantize them.

The model was released as a HuggingFace repository with 46 sharded safetensors files totaling roughly 149 GB. DeepSeek also published reference inference code using torchrun with model parallelism across GPUs and a tilelang kernel library for the FP8 and FP4 matrix multiplies. That reference code assumes Hopper or newer NVIDIA GPUs with native FP8 tensor cores. The P40s, as we'll see, did not get that memo.

Fortunately, the community moved fast. Within weeks, the teamblobfish group had produced GGUF quantizations of the model in formats ranging from Q8_0 down to IQ1_S, using a custom fork of llama.cpp implementing the new deepseek4 architecture. The IQ1_S variant — 1.6 bits per weight on average — compresses the model to 58 GB on disk. That's the one that fit.

Attempt One: Four NVIDIA Tesla P40s

The P40 machine is a familiar presence in my lab. Four Pascal-era GP102GL GPUs, each with 24 GB of GDDR5X, running Ubuntu 24.04 with NVIDIA's 580-series server driver. They've run diffusion models, quantized LLMs, and even a 57-billion-parameter image editing pipeline. I figured they'd at least be able to load DeepSeek V4, even if inference was slow.

They couldn't. The failure was architectural, not quantitative.

The first problem was PyTorch. DeepSeek's reference code requires PyTorch 2.10 or newer, which brings native float8_e4m3fn and float4_e2m1fn_x2 dtype support. These dtypes map to CUDA kernels that were compiled for compute capability 7.5 and above. The P40 is compute capability 6.1. When I installed PyTorch 2.11 with CUDA 12.8, it could see all four GPUs, but every CUDA kernel launch — from FP8 tensor creation to basic FP32 matrix multiplication — failed with cudaErrorNoKernelImageForDevice. The PyTorch binaries simply contain no sm_61 machine code.

You can build PyTorch from source with sm_61 support. I've done it before. It takes hours, requires exact CUDA toolkit version alignment, and produces a wheel that you have to manually maintain. But even if I'd taken that route, it wouldn't have helped, because the P40's Pascal architecture has no hardware support for FP8 or FP4 data types. The tensor cores that Hopper uses for FP8 matrix multiply don't exist here. Any FP8 operation would have to be emulated in software, destroying whatever performance advantage the larger die area might have provided.

The second problem was VRAM capacity. 96 GB across four GPUs sounds like a lot, but the IQ1_M GGUF needs 67.5 GB of allocation buffer for the weights alone, plus compute buffers, plus a KV cache. Even if the CUDA compatibility issues were solved, the model wouldn't fit without aggressive CPU offloading. And the P40, with its PCIe 3.0 x16 interface and ~12 GB/s of host-to-device bandwidth, would have turned CPU offloading into a slideshow.

Sometimes the right tool for the job is the newer one with less raw compute but the right instruction set.

The Strix Halo Setup

The Strix Halo machine is an AMD Ryzen AI MAX+ 395 APU running Ubuntu 24.04 with ROCm 7.2. Key specs:

GPU: Radeon 8060S integrated graphics, gfx1151, RDNA 3.5 architecture, 40 compute units
Memory: 128 GB LPDDR5X unified, with 96 GB configured as GTT for GPU access
ROCm: 7.2.0 with HIP runtime
llama.cpp: Existing build at commit d82b7a7c1 (the mainline GGUF tools work fine for reading the model files)

The Strix Halo has one thing the P40s don't: the right instruction set. RDNA 3.5 includes native FP8 support and, more importantly, the gfx1151 target in ROCm can compile and run kernels that operate on quantized data without emulation. The integrated GPU isn't fast, but it's compatible. And because the memory is unified, there's no PCIe bus standing between the CPU and the weights.

I started by downloading the IQ1_S-XL variant from teamblobfish's HuggingFace repository. The download was 58 GB across two sharded GGUF files — 47 GB for the first shard and 11 GB for the second. HuggingFace's CDN delivered them in about ten minutes.

Building llama.cpp with DeepSeek V4 Support

The mainline llama.cpp repository doesn't have support for the deepseek4 architecture yet. There's an open pull request — #22607 — with a full implementation by the nisparks team that adds the model architecture, tensor definitions, and inference graph construction. The PR is closed but the nisparks fork has the implementation on the pr/01-deepseek-v4-arch branch.

Cloning and building was straightforward:

git clone https://github.com/nisparks/llama.cpp.git llama.cpp-dsv4
cd llama.cpp-dsv4
git checkout pr/01-deepseek-v4-arch
mkdir build && cd build
cmake .. \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1151 \
  -DGGML_HIP_NO_VMM=OFF \
  -DGGML_HIP_MMQ_MFMA=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build . --target llama-cli -j16

The GGML_HIP_NO_VMM=OFF flag is important. With VMM (Virtual Memory Management) enabled, the HIP backend can use the GTT memory pool — the 96 GB of system memory the GPU can address directly. Without it, only the 4 GB of dedicated VRAM carveout is visible, and the model simply won't fit.

The build took a couple of minutes on the 16-core Zen 5 processor. The resulting binary was a standard llama-cli but with the deepseek4 architecture baked in.

Tensor Name Mismatches and the Art of sed

This is where things got interesting. The GGUF files from teamblobfish were created with a different version of the llama.cpp converter than the one in the nisparks fork. The model architecture code and the GGUF tensor names didn't agree on what anything was called.

The first run failed with "unknown model architecture: deepseek4." That was a missing branch in the nisparks fork — the experiment/deepseek-v4-gguf-convert branch had the right GGUF conversion scripts but not the model code. The pr/01-deepseek-v4-arch branch had the model code but expected different tensor names.

The mismatches fell into a few categories:

HC (hyper-connection) tensor naming: The GGUF uses output_hc_base, output_hc_fn, output_hc_scale for the output head hyper-connections. The code expected hc_head_base, hc_head_fn, hc_head_scale. Per-layer, the GGUF uses blk.N.hc_attn_base and blk.N.hc_ffn_base, which the code expected, but it needed the .weight suffix on the tensor lookup calls.

Compressor vs. compress naming: The GGUF names the compressed attention components attn_compressor_ape, attn_compressor_gate, etc. The code expected attn_compress_ape, attn_compress_gate. Same for indexer compressors: indexer_compressor_ape in the GGUF, indexer.compress_ape in the code (note the dot vs. underscore, too).

KV latent tensor: The GGUF named it attn_kv, the code expected attn_kv_latent.

Missing .weight suffixes: Several tensor creation calls in the model loader omitted the "weight" suffix when calling create_tensor(). For example, create_tensor(tn(LLM_TENSOR_ATTN_SINKS, i), ...) instead of create_tensor(tn(LLM_TENSOR_ATTN_SINKS, "weight", i), ...). The GGUF tensor names all ended in .weight or .bias, so every lookup that omitted the suffix returned null.

Each fix was a one-line change to src/llama-arch.cpp (the tensor name registry) or src/llama-model.cpp (the model loader), followed by a rebuild. The full list:

hc_head_base → output_hc_base, hc_head_fn → output_hc_fn, hc_head_scale → output_hc_scale (3 lines)
Added "weight" suffix to HC head, HC layer, attn_sinks, attn_compress_ape, indexer_compress_ape, ffn_exp_probs_b, ffn_gate_tid2eid tensor calls (10+ lines)
attn_compress_* → attn_compressor_* (4 lines)
indexer.compress_* → indexer_compressor_* (4 lines)
attn_kv_latent → attn_kv (1 line)
attn_out_* → was already attn_output_* in the code (no change needed)

After all fixes, the tensor count matched: 1,325 loaded, 1,325 expected. The model loaded without errors.

GPU Binary Broadcast and Dequantization

The model loaded — and immediately crashed during prompt processing with an assertion failure in the GPU binary broadcast kernel:

ggml_cuda_op_bin_bcast: GGML_ASSERT(src1->type == GGML_TYPE_F32 || src1->type == GGML_TYPE_F16) failed

The IQ1_S quantization stores most tensors as iq1_m type, but the output projection and token embedding layers are q5_K — a 5-bit quantization format. When the model tried to add a bias or scale to one of these tensors during inference, the binary broadcast kernel received a q5_K tensor as src1 and had no idea what to do with it.

The binary broadcast kernel in ggml/src/ggml-cuda/binbcast.cu is designed for element-wise operations like add, multiply, and divide where src1 is broadcast across src0. It natively supports float32 and float16 for src1, but nothing quantized. The fix required three things:

Adding #include "convert.cuh" to get access to the dequantize_row_q5_K_cuda and dequantize_row_q2_K_cuda functions
Adding a ggml_backend_cuda_context & ctx parameter to ggml_cuda_op_bin_bcast so we could allocate temporary GPU memory from the pool
Inserting a dequantization step before the binary operation:

if (src1->type != GGML_TYPE_F32 && src1->type != GGML_TYPE_F16) {
    int64_t nelements = ggml_nelements(src1);
    float * src1_f32 = src1_f32_buf.alloc(ctx.pool(), nelements);
    to_fp32_cuda_t to_fp32 = ggml_get_to_fp32_cuda(src1->type);
    to_fp32(src1_dd, src1_f32, nelements, stream);
    src1_effective = (const void *)src1_f32;

    // Build float-compatible strides for the contiguous dequantized buffer
    memcpy(&src1_copy, src1, sizeof(ggml_tensor));
    src1_copy.type = GGML_TYPE_F32;
    src1_copy.nb[0] = sizeof(float);
    for (int d = 1; d < GGML_MAX_DIMS; d++) {
        src1_copy.nb[d] = src1_copy.ne[d-1] * src1_copy.nb[d-1];
    }
    src1_ptr = &src1_copy;
}

The stride reconstruction was necessary because the original tensor's strides are computed for the quantized block layout — a q5_K block is 176 bytes, not 4 bytes like a float. Passing float data with quantized strides causes alignment assertion failures deeper in the launch pipeline. The fix constructs a temporary tensor view with float-compatible strides that match the dequantized data's memory layout.

This patch is specific to the nisparks fork's version of the CUDA backend and will need to be re-applied or reimplemented differently when deepseek4 support lands in mainline llama.cpp. But it worked.

The First Successful Run

With all fixes applied and the binary compiled, the model loaded in about 30 seconds — the time it takes to allocate 58 GB of GPU buffers through the VMM allocator and initialize the compute graph. Memory breakdown showed 58,337 MiB for the model weights, 18 MiB for context, and a tiny compute buffer.

I sent a simple prompt:

What is 2+2? Answer briefly.

The model responded with [Start thinking] followed by a series of <<<<<<<<<< characters — the DeepSeek thinking prefix. It was actually processing. The IQ1_S quantization is aggressive enough that the model occasionally gets stuck in token repetition loops, which is a known artifact of extreme compression. But the fundamental achievement was clear: a 284B-parameter model, running inference on a consumer APU, completely locally.

The generation speed was not measured precisely in this initial test — I was focused on getting it running at all rather than benchmarking throughput. But qualitatively, token generation was slow, on the order of 1-2 tokens per second. The Strix Halo's LPDDR5X memory bandwidth (~120 GB/s) is the binding constraint. Each token requires streaming the active expert weights through the compute units, and 13 billion activated parameters at 1.6 bits per weight means roughly 2.6 GB of data movement per token. At 120 GB/s, that's a theoretical maximum of about 46 tokens per second, but the reality is much lower due to the overhead of the attention mechanism, the non-expert weights that also need to be accessed, and the compute time for the matrix multiplies. In practice, expect single-digit tokens per second.

Why This Matters

There's a narrative in AI hardware discourse that goes something like: to run large models, you need large GPUs. Multiple H100s. A DGX. Cloud credits. The reality, increasingly, is that model compression and inference optimization are advancing faster than model size growth.

DeepSeek V4 Flash in IQ1_S quantization runs on a system you can buy for around $3,500. It occupies 58 GB of disk space and fits within 65 GB of GPU-addressable memory. It's not fast. It's not production-ready. But it runs.

The contrast with the P40s is instructive. Those cards have more raw teraflops, more memory bandwidth (346 GB/s per card vs. ~120 GB/s shared), and a dedicated inference pedigree — the P40 was literally designed for this kind of workload. But they lack the instruction set support that modern quantized models depend on. Hardware compatibility is becoming more important than hardware capability. FP8 isn't just a nice-to-have; it's table stakes for running 2026-vintage models.

The AMD Strix Halo platform, for all its compromises — limited memory bandwidth, immature software ecosystem, the eternal dance with ROCm version pinning — is the platform where this experiment worked. The unified memory architecture means you never have to think about PCIe transfers. The RDNA 3.5 instruction set means FP8 just works. And the 96 GB of GPU-addressable GTT memory means models that need 70+ GB of buffer allocation actually fit.

One wrinkle worth mentioning: the VMM (Virtual Memory Management) allocator in this build doesn't survive CUDA graph warmup. Running without --no-warmup triggers a HipVMM Failure: invalid argument inside ggml_cuda_pool_vmm::alloc during the warmup's matrix-vector multiply pass. The first token inference itself works — it's the graph capture that fails, likely because the VMM allocator can't satisfy the warmup's allocation pattern within the remaining pool space. Skipping warmup with --no-warmup avoids the issue entirely at the cost of slightly higher per-token latency. For a research setup running a 284B-parameter model at single-digit tokens per second, that trade-off is easy to accept.

What's Next

The nisparks fork's deepseek4 support will eventually make its way into mainline llama.cpp. When that happens, Ollama will be able to import the model directly with a simple Modelfile, and the whole setup process will collapse to a ollama create command. That's the dream, at least.

In the meantime, there's room for improvement on the performance side. The IQ1_M quantization, at 64 GB, is just barely too large for the Strix Halo's 65.2 GB of HIP-visible VRAM — the weight allocation alone needs 67.5 GB. A custom quantization that splits the difference — say, IQ1_M for the attention layers and IQ1_S for the experts — might squeeze into memory while preserving more quality. The CPU binary ops for quantized types also need attention; the current code path fails on CPU-offloaded layers because the CPU backend doesn't support binary operations on q8_0 tensors.

And then there's the bigger picture. DeepSeek V4 Pro, at 1.6 trillion parameters with 49 billion activated, is the next target. Even IQ1_S quantization of that model would be around 300 GB — beyond the reach of any single consumer device today, but within striking distance of multi-node setups or upcoming memory standards. The gap between what's possible and what's practical keeps shrinking.

For now, the takeaway is simple: if you have a Strix Halo machine and a willingness to patch a few tensor names and GPU kernels, you can run one of the most capable open-source language models ever released, completely offline, in your own home. That felt like science fiction three years ago. Today it's just an afternoon of debugging.

Reproducing This

If you want to try this yourself, here's the condensed setup:

# Clone and build the nisparks llama.cpp fork
git clone https://github.com/nisparks/llama.cpp.git llama.cpp-dsv4
cd llama.cpp-dsv4
git checkout pr/01-deepseek-v4-arch

# Apply tensor name fixes (see this post for details)
# Build with HIP VMM enabled for GTT memory access
mkdir build && cd build
cmake .. \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1151 \
  -DGGML_HIP_NO_VMM=OFF \
  -DCMAKE_BUILD_TYPE=Release
cmake --build . --target llama-cli -j16

# Download IQ1_S GGUF
pip install huggingface_hub
hf download teamblobfish/DeepSeek-V4-Flash-GGUF \
  IQ1_S-XL/ --local-dir ./model

# Run
./bin/llama-cli \
  -m ./model/IQ1_S-XL/DeepSeek-V4-Flash-IQ1_S-XL-00001-of-00002.gguf \
  -p "Your prompt here" -n 100 -ngl 99 -c 256 --temp 0 -sp

The full set of code changes — tensor name mappings, suffix additions, and the binbcast kernel fix — are detailed in each section above. A consolidated patch will be available once these changes stabilize.

Building Stalker: A Mid-Cap Trading Bot and the Data Network That Feeds It

A.C. Jokela — Fri, 08 May 2026 18:00:00 GMT

Three years ago I built a Slack bot that generated company stock reports. You'd type /report TSLA and a Lambda fan-out would pull yfinance data, scrape recent news through BeautifulSoup, run technical indicators with the ta library, and ask GPT-4 to write three paragraphs about what was happening with the stock. The output landed back in Slack and on a static S3 site. It was fun. It was a toy.

It told you what one stock looked like. It didn't tell you what to do about it.

Stalker is what I built after I stopped wanting toys.

It's an autonomous mid-cap equity trading bot. It runs on AWS Lambda. It reads a daily macro brief, ranks a 300-name universe by factor scores, asks Claude Sonnet 4.6 to propose orders against the current portfolio, runs the proposed plan through a deterministic risk gate, and submits the survivors to Alpaca's paper trading API with deterministic client-order IDs so re-fires are idempotent. It logs every decision, every fill, every rejection. It emails me a daily report. It runs without me touching it.

Right now it has 12 positions in a paper account modeled as a synthetic \$1,000 seeded deposit. Inception-to-date return on that \$1,000 baseline is +3.76% against SPY's +2.12% — so +1.65pp of alpha over four weeks of live operation. (To head off a misreading of the next number: the position book currently shows \$1,576 of market value, which is not \$576 of gains on the seed. It's over-deployment from a bug I'll cover later — the bot was reading its cash limit as auto-replenishing every brief day instead of flowing from the order ledger, so cumulative buys ran past what a real \$1,000 account could have funded. The fix is in; the bot is currently trimming positions back to within the seed. The +3.76% number is the honest baseline-relative return.) The numbers don't matter at four weeks anyway — that's noise. What matters is that the system is running, the architecture is settled, and the methodology for measuring whether any of it actually works is pre-registered.

Stalker is one of six related projects. The other five are data sources that feed it:

Headwater is a daily financial-newsletter aggregator that emits a structured macro brief twice a weekday morning and afternoon. It reads the writers I follow, classifies what they're saying, and ships a JSON document with a regime call, sector tilts, themes, and a watchlist.
Estuary does cross-source consensus detection. When five different writers all flag the same ticker on the same week, that's a cluster, and clusters end up in a daily brief at 23:30 UTC.
PrivateEye decodes paywalled-newsletter teasers into ticker picks. The actual decoded stock recommendations land in a daily digest.
Tributary ingests SEC EDGAR 8-K filings, extracts each item-level event, and classifies materiality. NT-10K late-filings, 4.02 restatements, 1.03 bankruptcies, 5.02 executive departures — Stalker reads these as risk-and-opportunity overlays.
Goldfinch pulls federal prime contract awards from USAspending.gov, maps recipient legal entities to public tickers, and emits the matches as confirmatory revenue-disclosure signals.

Each one is its own AWS account-scope project — its own Lambdas, its own DynamoDB tables, its own SES inbound endpoint. They're loosely coupled: each emits a JSON document on a known schema; Stalker reads from each one through a producer-specific loader at analyze time. Adding a new feeder is a Pydantic model plus a load function plus an SES rule. The architecture is "more sources are better, but no source is required."

This is the post that explains how all of it hangs together.

The Data Network

The shape of the network matters. Each feeder is a separate project because each one solves a different problem. Trying to put all the signal extraction inside Stalker would have produced a monolith that does five things badly. The split lets each project focus on one thing — newsletter aggregation, consensus detection, teaser decoding, EDGAR ingestion, contract scraping — and lets Stalker focus on the trading layer alone.

Headwater generates the macro overlay. The structured brief carries a regime field (one of risk_off, transitional, neutral, transitional_risk_on, risk_on), a list of sector_tilts with view and strength, a list of thematic_views with affected_sectors, a key_risks list with horizons, and a watchlist of tickers under active discussion. Stalker doesn't trade the watchlist names directly — those are mostly megacaps that fall outside the mid-cap filter — but the macro block converts directly into multipliers on the combined factor score. A bullish-Energy tilt at high strength multiplies every Energy mid-cap's combined_z by 1.20 before ranking. A bearish-Tech tilt at medium strength multiplies it by 0.90. The factor selection is sector-tilted by the macro view at the rank-construction stage, not by Claude's interpretation at the prompt stage. The macro pipe into selection is deterministic and traceable.

Estuary is a different role. It tracks what individual newsletter writers are publishing across a window, computes consensus when multiple writers converge on the same ticker within a few days, and emits the clusters along with the per-writer high-conviction picks. A cluster of five sources flagging a single name in the same week is a stronger signal than five separate calls scattered across the year. Stalker reads Estuary's output as a confirmatory soft signal — when a name in the candidate universe also has Estuary cluster support, Claude can use that as a tie-breaker between similarly-ranked candidates.

PrivateEye is the cheapest unit-economics piece in the stack. Financial-newsletter teasers are designed to make you subscribe — they're written to gesture at a recommendation without giving it away. PrivateEye reads the teasers and extracts the underlying ticker pick. The decoded picks ship in a daily digest. Most of them are megacaps that don't apply to Stalker, but the percentage that fall in the \$2B–\$10B band become another confirmatory soft signal alongside the Estuary clusters.

Tributary is the structural-events feed. The SEC publishes 8-K filings continuously, and many of them are noise — boilerplate amendments, routine disclosures, exhibit lists. The interesting ones are the 5.02 executive departures, the 1.03 bankruptcy filings, the 4.02 audit-restatements, the 5.07 favorable shareholder votes, and the 8.01 material acquisitions. Tributary classifies each item-level event by materiality and salience, summarizes the substance, and emits one record per filing. Stalker uses high-materiality Tributary events as risk overlays: a recent NT-10K on a holding is a reason to consider trimming. The architecture treats negative-materiality items as exit triggers and positive items as additional context.

Goldfinch is the newest feeder. Federal contract awards land in USAspending.gov within a few days of action; they're a real fundamental disclosure that often precedes earnings discussion of the same revenue. The challenge is the recipient-to-ticker mapping — federal awards are made to legal entities, not to listed companies, and many awards go to subsidiaries or government-services divisions whose parent ticker isn't obvious. Goldfinch handles the mapping (with a confidence rating per match), filters to material awards, and emits the matched records. A \$300M Department of Defense award to a \$5B mid-cap defense contractor is a real near-term revenue event; the same award to a \$200B megacap is rounding error. Stalker weighs the signal accordingly.

Each feeder ships through SES inbound mail to its own recipient at in.stalker.bot. briefs@ is Headwater. estuary@ is Estuary. events@ is Tributary. picks@ is PrivateEye. Goldfinch is a special case — it writes directly to a shared DynamoDB table because it runs in the same AWS account, but conceptually it's the same pattern: structured JSON, schema-versioned, lenient on unknown fields. SES drops each inbound message into S3, an EventBridge notification fires the Stalker ingest Lambda, the Lambda dispatches to the producer-specific parser, validates against a Pydantic model, archives the parsed payload, indexes a row in DynamoDB, and (for Headwater only) emits a BriefIngested event that triggers the analysis layer.

That's the inbound side. The outbound side is one Lambda — the analyzer.

The Trading Core

Stalker's actual decision logic is layered. There's a deterministic factor stack at the bottom, an LLM call in the middle, and a deterministic risk gate at the top. The LLM has freedom in the middle layer; everything below and above is mechanical.

The universe. Every morning at 04:00 UTC, a Lambda hits FMP's screener API for US-listed stocks with market cap between \$2 billion and \$10 billion that are tradable on Alpaca. The result is a partition in the stalker-universe DynamoDB table keyed on refresh_date. The mid-cap band is the strategy's first commitment: Stalker doesn't trade megacaps (they're efficient, no edge available) and doesn't trade microcaps (liquidity, regulatory, and size-premium hazards). Mid-cap is the band where factor strategies historically have shown the most edge over passive indexing.

Factor scoring. Fifteen minutes after the universe refresh, a second Lambda computes per-name factor scores. Three factors:

Momentum: 12-month return minus the most recent month — the classical Jegadeesh-Titman 12-1 factor. The skip-month removes short-term reversal noise.
Quality: trailing-twelve-month ROIC plus operating margin, equally weighted. This is the post-ADR-011 production definition. (Stalker tracks load-bearing strategy decisions in numbered Architecture Decision Records — short Markdown documents in docs/adr/ that record the context, the evidence, the chosen option, and the reasoning. ADR-011 was the one that switched the quality factor.) The original ROE-plus-gross-margin definition turned out to be silently destructive in backtest, dragging alpha by 31 percentage points over the 3-year survivorship-bias-free window. The ADR walks through the seven candidate definitions, the alpha and Sharpe under each, and the reasoning for picking ROIC-plus-operating-margin over single-metric ROIC even though the latter showed a slightly higher headline alpha. Subsequent ADRs in this post (009, 010, 013, 014) follow the same pattern.
Low-vol: the negative of 60-day realized volatility. Lower-vol names get higher z-scores.

Each factor is z-scored within sector — a Financials name's quality is graded against other Financials, not against Tech. The sector-neutral approach prevents structural concentration: REITs and banks naturally have high ROE and low vol, so cross-sectional z-scoring would otherwise dump the whole portfolio into one or two sectors regardless of the macro view. Sectors with fewer than five names skip scoring; their z-scores would be noise.

The three factor z-scores combine into a combined_z via configurable weights. The current production weights are 0.55 momentum, 0.225 quality, 0.225 low-vol — momentum-tilted because the bias-corrected backtest sweep showed monotonic alpha improvement up through 0.55 with diminishing return beyond that. The factor stack writes top-300 by combined_z back to DynamoDB with in_universe=true.

The macro overlay applies here. Before the factor scores are written, Headwater's sector tilts are pulled in and applied as multiplicative adjustments to combined_z per sector. This is the "macro pipe" mentioned earlier: the sector view from the structured brief becomes a factor in the rank itself. A bullish/high Energy tilt floats Energy names up; a bearish/medium Tech tilt sinks Tech names. The multipliers are deterministic — defined in macro_sizing.py and unit-tested — so a given brief plus a given factor snapshot produces a reproducible rank. No LLM in this layer.

The analyze layer. When a Headwater brief lands in S3 and the ingest Lambda emits BriefIngested, the analyze Lambda fires. It loads the brief, fetches the current Alpaca account state, fetches today's universe partition with factor scores, queries the four producer feeder loaders for recent confirmatory signals, builds a structured user message, and calls Claude Sonnet 4.6 with a tool-use schema that constrains the output to a propose_trade_plan JSON object. The system prompt explains the strategy posture, the hard rules the risk gate enforces, and the role of each feeder. The user message contains the macro block, the top-50 candidates by combined_z with their factor scores and Kelly-derived suggested allocations, the current positions, the universe whitelist, and the soft-signal sections from each feeder.

Claude's job is constrained selection. It picks 6–10 names from the candidate list (or the existing positions) and assigns target dollar values. It cannot trade outside the universe whitelist. It cannot propose a single trade larger than 15% of NAV. It cannot exceed 25% of NAV in any single position post-trade. It cannot buy a name with earnings inside seven days. The system prompt makes these constraints explicit, but the risk gate enforces them mechanically — the LLM is one defense layer, not the only one.

The proposed plan flows into risk.evaluate(), a pure-Python function that takes the state and the proposed orders and returns one of four decisions: approved, needs_human_approval, rejected, or — under specific edge cases — a noise band that triggers a re-prompt. The risk gate enforces:

25% per-position cap (post-trade)
15% per-trade cap on buys
2% minimum cash buffer
31-day IRS wash-sale block on buys of recently sold-at-loss names
Earnings veto (no buys within 7 days of next earnings)
Daily drawdown halt (-5% intraday → no new buys)
Total drawdown halt (-15% from inception → no new buys)
Losing-streak halt (3 consecutive losing sells → no new buys)
Universe whitelist (any non-whitelisted ticker is rejected)

The risk gate's outputs are logged in DynamoDB regardless of whether execution proceeds. If the plan is approved, the executor Lambda picks it up via EventBridge and submits each order to Alpaca with a deterministic client_order_id formed from the SHA-1 of (plan_id, ticker, action). The deterministic ID makes re-fires idempotent: if the executor crashes after submitting three of five orders, the next invocation tries to submit the same orders, hits a 422 collision on the three already-submitted, fetches their existing state, and proceeds with the remaining two.

There's one nuance in the executor that took an incident to find. Alpaca's day-trade detection rejects new orders on a symbol when an opposite-side order is already open — labeled "potential wash trade" in the rejection message. We discovered this when a sell got blocked because a stale stop-loss was still on the book. The fix is a preflight: before each submission, the executor lists open Alpaca orders for the symbol, cancels them, syncs the matching stalker-orders rows to cancelled, and then submits the new order. Belt and suspenders — Alpaca's enforcement still works, but we don't rely on it.

The Seeded Account

There's a subtle constraint baked into the architecture that took two iterations to get right: the bot is supposed to behave as if I'd actually deposited \$1,000 of real money, not as if it had access to Alpaca's \$100,000 paper-account default. Live trading caps the position size to what a \$1,000 retail account would actually do; paper testing should exercise the same sizing logic.

The first cut of this used a simple min(real_cash, NAV_CAP) cap on the cash field reported to the LLM and risk gate. That worked when the bot had no positions. Once positions accumulated, the cap silently broke the percentage math: a position with \$230 of market value plus a proposed \$40 buy is \$270 against the real \$2,500 NAV (a benign 10.8% of portfolio), but the cap reported the NAV as \$1,000, making the same position read as 27% — over the hard 25% cap. Three plans got rejected over a 36-hour window before the bug surfaced.

The first fix tightened that: cap the cash, not the NAV. NAV becomes capped_cash + sum(position market values), which is a coherent number that reflects real portfolio percentages while preserving the small-account sizing exercise. That fixed the rejection bug.

But it introduced a deeper problem: the cash kept reading \$1,000 every brief, replenished from Alpaca's bottomless paper-account seed. A real \$1,000 account doesn't work that way. A real account spends \$200 to buy a position; cash goes to \$800. The bot was effectively redeploying \$1,000 of fresh capital every brief day. Over four weeks the cumulative buys totaled \$2,123 against \$557 of sells — 2.1× the intended seed. A real account would have hit a cash wall after about ten buys.

The honest fix is the second iteration: cash flows from the order ledger. The seeded cash at any moment is $1,000 + sum(filled_sells) − sum(filled_buys). The function scans stalker-orders for filled and partially-filled rows, sums the filled_qty * filled_avg_price per side, and returns the seeded cash. NAV is seeded_cash + positions_mv. Buying power is max(0, seeded_cash) — no margin in the seeded model. When seeded cash goes negative (the bot is over-deployed relative to the seed), the LLM sees the negative number with an inline note instructing trim-first behavior, and the risk gate's existing 2% cash-buffer check naturally enforces rebalance-only mode until sells refill the seed.

After the second fix, the bot's reported state is cash = -$566, nav = $993, buying_power = $0. It will spend the next several brief cycles trimming positions back into the seed before it can buy again. That's exactly how a real \$1,000 account would behave coming off a 4-week over-deployment streak. Self-healing — no manual reset, no position rebalancing required.

The lesson here generalizes beyond Stalker: when a paper-test wrapper diverges from the live behavior it's supposed to mirror, the divergence accumulates silently. The only protection is to define the wrapper's semantics carefully and test the boundary conditions. "What happens when positions exceed the cap" was the question I should have asked at design time.

Backtesting Without Lying To Yourself

Backtesting a strategy against historical data is the easiest way to fool yourself in finance. The classic failure mode is survivorship bias: you backtest against today's universe of public companies, applied retroactively. The names that delisted, got acquired, or went to zero aren't in your test set, because they're not in today's universe. Your universe is by construction a sample of survivors. You're testing how well the strategy would have done on the names that turned out to be successful — which is not the same as testing how well it would have done in real time.

Stalker's backtest engine handles this through a point-in-time (PIT) universe archive. The bootstrap process queries FMP's /delisted-companies endpoint to recover the names that left the public market, then queries /historical-market-cap per ticker to determine the cap band membership at each historical date. The result is a JSON archive at ~/.cache/stalker-backtest/pit_universe/<window>.json that answers the question "which tickers were in the \$2B–\$10B mid-cap band on date X?" for any X in the bootstrap window. The current archive covers 2023-01-01 → 2026-04-30 and contains roughly 2,000 tickers ever in band, of which several hundred have a delisted_date.

Every backtest run can take a --pit flag. With it on, the rebalance candidate set at each weekly rebalance date is filtered to symbols whose market cap was actually in the \$2B–\$10B band on that date. With it off, the rebalance uses today's universe applied retroactively — the survivorship-biased path, kept around for legacy comparability.

The first thing the PIT archive did was discredit a previous result. ADR-009 had bumped the production momentum weight from a balanced 0.40 / 0.30 / 0.30 to 0.55 / 0.225 / 0.225 based on a non-PIT backtest showing +24pp alpha over a 1-year window. When I re-ran the same configuration with PIT, alpha collapsed to -12pp. The original number was a survivorship-bias artifact. The momentum tilt looked dominant because the survivors were the names with strongest momentum — by definition.

ADR-010 superseded ADR-009 with the bias-corrected result: at 0.55 momentum on PIT, the strategy beats SPY by +21.6% over the 3-year window. Better than equal-weight but a fraction of what the survivorship-biased number suggested. Honest accounting hurts.

The PIT archive also enabled ADR-011's quality-factor switch. The original quality definition (ROE plus gross margin) was producing essentially zero contribution in the bias-corrected backtest. The natural diagnostic question was whether quality is a noisy factor at our universe size, or whether ROE-plus-GM is the wrong measurement of quality. Adding a quality_definition knob to the engine and sweeping seven definitions showed the second answer: switching to ROIC-plus-operating-margin lifted alpha by +31 percentage points at production weights and improved Sharpe from 1.23 to 1.44. ROE rewards leverage (which varies enormously across mid-cap capital structures), and gross margin is industry-structural (SaaS at 80%, distribution at 8%) which sector-neutral z-scoring only partially undoes. ROIC is leverage-neutral and operating margin captures pricing power within sector — both tighter signals at our universe size.

The PIT archive turns "backtest" from a marketing exercise into a real measurement. It's not perfect — historical FMP fundamental data has its own gaps and revisions — but it removes the dominant bias.

The Meta-Experiment

The factor stack has been validated end-to-end on bias-corrected backtests. The risk gate is a pure-Python module with full test coverage. The executor is mechanical. What hasn't been validated, in any rigorous way, is the LLM layer in the middle. The brief-driven analyze step might be adding alpha by combining macro context, position-aware reasoning, and human-style synthesis the factors can't see. Or it might be a wash. Or it might be subtracting alpha by overriding good factor picks with brief-narrative picks that don't survive in the data.

The cost of getting this wrong in either direction is asymmetric. A non-additive LLM layer costs roughly \$200–700 a year in API plus an ongoing complexity tax — debugging an LLM-mediated trade path is harder than debugging a deterministic one. An additive LLM layer foregone is real alpha left on the table. Either way, the answer should come from data, not intuition.

ADR-013 is the pre-registered A/B test that mechanizes the question. Two arms, identical except for the selection signal: the brief arm (current Stalker, with Claude reading the brief and picking 6–10 names) versus a factors-only arm (top-N by combined_z, equal-weighted, same risk gates, same Kelly sizing). The factors-only arm runs as a daily shadow book — a separate DynamoDB table tracking what the deterministic strategy would have done each weekday close. The pairwise comparison is a paired daily-return diff t-test pooled over the elapsed window.

The pre-registration document locks the methodology before any data is examined. Hypotheses are pinned. The test statistic is mean(d) / (sd(d) / sqrt(N)) where d is the daily return difference. The threshold is 30 basis points per month of alpha at p < 0.05. The decision rule is mechanical:

Brief arm wins by >30bp/mo at p<0.05 → keep
Brief arm loses by >30bp/mo at p<0.05 → simplify (retire the LLM)
Inconclusive → simplify (default; the burden of evidence is on the LLM layer to justify itself)

The horizon is 12 months from inception (2026-05-04 to 2027-05-04). At horizon end, the test runs once on the pooled paired returns, the decision rule is applied, and the ADR's status updates from proposed to one of accepted (kept), accepted (simplified), or — if the data supports a conditional-fire hybrid — accepted (hybrid) with a follow-up ADR scoping the hybrid design.

Mid-flight changes invalidate the pre-registration. If the factor weights change, the test restarts. If the prompt changes, the test restarts. The whole point of pre-registration is that it converts a tempting post-hoc optimization into a principled experiment, with a documented decision rule that doesn't move once data starts coming in.

The shadow book runs on a daily 16:35 CT cron that ticks the factors-only portfolio through one rebalance, marks positions to today's close, and writes a row to stalker-shadow-performance. A weekly Monday-morning cron joins the live and shadow performance series, computes the running paired-diff statistic, and posts a status comment to the project's tracking ticket. The cron is dormant infrastructure for the first 12 months — the running stats are informational only; the keep-versus-simplify decision fires once at horizon close, not every week.

There's a sub-experiment I ran during the pre-registration drafting that's worth noting because it killed a tempting alternative. ADR-013's locked baseline is equal-weight factors-only, but the live system uses Kelly-derived suggestions to bias Claude's sizing. A reasonable concern was: if Kelly-as-binding-sizer (using the combined_z to set position weights directly, not just suggest them to the LLM) beats equal-weight by a lot in backtest, then the equal-weight baseline is a weak counterfactual and the brief arm is being given an unfair advantage. ADR-014 ran the offline comparison: at top_n=30, Kelly-as-binding-sizer appeared to beat equal-weight by +47.75pp alpha. Striking number. I almost wrote it up as a win.

The result didn't survive sensitivity checking. At top_n=10 (where Kelly's per-name weights actually fit within the cash budget), Kelly underperformed equal-weight by -43pp. The +47.75pp at top_n=30 was an implementation artifact: with the unnormalized Kelly weights summing to over 100% of NAV, the engine's per-buy cost = min(delta, cash) clamp was eating the lower-ranked names and concentrating capital in the highest-z names. Kelly was acting as both selector and sizer at top_n=30 by exhausting cash on the top names. The apparent edge was concentration, not better relative weighting. ADR-014 was rejected — Kelly stays as advisory input to Claude rather than a binding sizer, and the equal-weight baseline for ADR-013 is defensible.

This kind of pre-flight check is the discipline pre-registration enables. The temptation to ship a +47pp number is real. The discipline of asking "but does it survive the obvious sensitivity check?" is what separates a research finding from a marketing claim.

The Data Extraction Layer

Most of what makes the data network valuable is the upstream work — the work of getting the data into a normalized, schema-versioned form that Stalker can read. Each feeder has its own extraction story. Headwater reads HTML email digests and parses them into structured records. Estuary tracks individual writer feeds and computes consensus. PrivateEye decodes paywalled teasers into ticker picks, which is its own can of worms. Tributary subscribes to SEC EDGAR's filing stream and runs a structured extraction over the 8-K text. Goldfinch hits USAspending.gov's API and runs the recipient-to-ticker mapping.

The orchestration for the heavier extraction work runs on a Bosgame M5 mini PC in my basement — the same machine that handles DirtScout's tax-list PDF extraction. It's a Ryzen-class system with 128GB of RAM and decent local inference horsepower for the structured-extraction passes that don't need frontier-model quality. The split between cloud and on-prem is roughly: cloud handles the trade-path quality-sensitive work (Claude Sonnet 4.6 for analysis), and on-prem handles the batch extraction work (PDF parsing, structured field extraction, classification at volume). Same pattern I described in the economics of owning your own inference.

Cron jobs on the Bosgame run the weekly gradient sweep that probes the factor weight space, the daily shadow-book tick for the ADR-013 A/B, and the periodic ADR validation runs that check whether shipped strategy changes are tracking their predicted alpha. Each script is wrapped in a cron_wrapper.sh that does a git pull --ff-only before exec, so changes pushed from my laptop propagate to the on-prem cron without manual SSH. The bash wrapper is forty lines long and has saved me hours.

The point of the on-prem layer is operational independence. The cloud Lambdas are for the trading path — they need to be reliable, fast, and well-observable. The on-prem cron is for the background research path — it can take twenty minutes to run a backtest sweep, and that's fine. Putting the long-running work on Lambda would burn timeout budget and money for no benefit. Putting the trading work on the on-prem machine would bind the strategy's uptime to my home internet. The split is operationally cleanest.

What This Is Actually For

Stalker is paper-trading. It hasn't moved a dollar of real money. The Alpaca account is paper, the brokerage credentials are paper-mode, and the seeded \$1,000 is synthetic. The point isn't to make money in the next four weeks. The point is to validate the architecture under realistic conditions before any real money is at stake.

The criteria I want to satisfy before live cutover are:

The strategy beats SPY meaningfully on PIT-corrected backtest (currently +21.6% over 3 years at production weights — done).
The factor definitions are documented in ADRs and the rationale is reproducible (done).
The risk gate is unit-tested with full coverage of every guardrail (done).
The executor handles real-world edge cases like opposite-side wash-trade rejection and partial fills (done).
The seeded-account model is validated end-to-end including the over-deployment failure mode (done).
Six months of paper-trading without operational incidents — no missed briefs, no failed analyses, no risk-gate false positives, no executor failures that get past idempotency.
The brief-versus-factors-only A/B has run to horizon and the LLM layer is justified (12 months — in progress).
The bot's behavior under drawdown halts has been observed in practice (waiting for a real correction).

That's a long list. The 6-month operational checkpoint and the 12-month ADR-013 close-out are the slow parts. The other items are mostly done.

What this isn't: a money-printing scheme. The factor stack has documented edge in academic literature and bias-corrected backtest, but mid-cap factor strategies are well-known and don't have huge mispricing margins. The expected outcome at the upper end is something like SPY +5–15pp annualized at modestly higher volatility. That's a decent risk-adjusted return, not a free lunch. If the strategy underperforms in real time, the answer is to deconstruct what changed — regime shift, factor decay, prompt drift, brief-stream change — not to tweak parameters until the curve looks right.

What this is: an exercise in building the smallest deployable instance of a real-money trading system, then validating its components individually before scaling. The \$1,000 seed is the smallest amount that exercises live-account semantics (rounding, fractional shares, cash management) without trivial scaling. The mid-cap focus is the band where factor edges historically existed. The pre-registered A/B is the methodology that converts "I think the LLM is doing useful work" into "the data either supports keeping the LLM or it doesn't."

The other five projects in the data network exist for the same reason. Each one is its own small thing that does one job well, with a published schema, with structured output, with a clean handoff to whoever's downstream. Headwater could feed any number of consumers; Stalker is just one. Tributary's 8-K events could drive any number of overlays; Stalker happens to read them as risk filters. The architecture is a sequence of clean producer-consumer interfaces with no shared state, no implicit dependencies, no monolith. Adding a seventh project tomorrow would be — schema, ingest path, loader, prompt mention. Same pattern, different signal.

What I'd Do Differently

A few things, in retrospect.

I'd start with the PIT archive sooner. ADR-009 shipped a non-PIT backtest result that turned out to be largely a survivorship-bias artifact. The corrected number from ADR-010 is still a real edge, but it's a fraction of what the original number suggested. If I'd built the PIT bootstrap as part of the initial backtest infrastructure rather than retrofitting it, I would have shipped fewer ADRs that needed superseding. The cost of the PIT bootstrap is real — the FMP /delisted-companies endpoint takes a few minutes to walk and the per-ticker historical-cap queries take an hour the first time — but the cost is one-time, and the cost of being wrong about a strategy parameter is worse.

I'd pre-register the LLM-versus-factors test earlier. ADR-013's pre-registration locks methodology before data examination. I drafted it after the live system had been running for three weeks, which means the locked-baseline decision was already informed by the running paper performance. That's not strictly invalidating — the locked threshold (30bp/mo at p<0.05) doesn't move with informed data — but a properly clean pre-registration runs before any live data is observed. The lesson is to instrument the meta-experiment before instrumenting the experiment.

I'd separate the risk constants from the strategy constants more cleanly. Things like MAX_POSITION_PCT and WASH_SALE_DAYS live in the same module as factor weights, which conflates policy-decision parameters (where changes are sensitive and should be ADR-driven) with implementation parameters (where changes are routine). The current structure works, but a clean separation would make the policy boundary more obvious.

I wouldn't change the producer-consumer pattern. That's the most reusable architectural decision in the stack. Each feeder being its own project with its own schema and its own SES inbound endpoint means I can add or remove sources without touching the consumer logic. Stalker reads from each loader independently and degrades gracefully if any loader returns empty — DDB throttle, S3 hiccup, brief stream paused, whatever. The system stays operational on partial signal. That property has been worth every minute of the architecture-discipline cost.

Cross-Project Notes

If you've read the DirtScout post, some of the patterns here will look familiar. CDK in Python for infrastructure-as-code. Python Lambdas with deterministic IDs for idempotency. DynamoDB instead of a relational database. SES inbound for the producer-mail pattern. Static Next.js export on CloudFront for the human-facing dashboard. The same architectural style that worked for a land-acquisition platform works for a trading system, because both are read-heavy event-driven workloads with bursty inbound and structured persistence.

The differences are at the edges. DirtScout deals with parcels — slow-moving, geographically-bound, low-cardinality. Stalker deals with tickers — fast-moving, market-state-dependent, high-correlation. DirtScout's risk model is "did we accidentally surface a parcel that's not for sale." Stalker's risk model is "did we put 30% of NAV into one ticker right before its earnings miss." The shape of the failure modes determines the shape of the safeguards.

The other shared piece is the vibecoding approach to the codebase itself. I direct the architecture, make the load-bearing decisions, and review the diffs. The actual lines of code mostly come from conversations. Stalker is around 8,500 lines of Python plus 800 lines of CDK plus 3,400 lines of TypeScript (the dashboard). I wrote almost none of that by hand. I directed all of it.

That's a real distinction. "Directing" means owning the architecture, the policy decisions, the risk constants, the methodology for evaluation, the criteria for live cutover. It means saying "no, this isn't how we should structure that" or "actually let's pull this up to its own ADR before we ship it." It's design and review, not typing. The typing is a commodity. The design isn't.

The Forward Path

The 12-month ADR-013 horizon ends 2027-05-04. Between now and then, the system runs on its own. The shadow book accumulates daily. The weekly stats post lands in the project tracker every Monday. If the strategy hits a real drawdown, the halt logic engages and I find out whether the halt criteria are calibrated correctly. If a brief stream goes down for a day, the bot still has factor signal to fall back on. If a feeder schema changes, the lenient Pydantic models tolerate the addition until I update the consumer.

At horizon, the close-out runs the t-test, applies the decision rule, and updates ADR-013's status. If the brief arm wins by margin, the LLM layer keeps its place in the architecture. If it loses or is inconclusive, I retire the LLM analysis path — analyze_handler.py becomes a thin "select top-N by combined_z" function and the brief stream becomes pure observability rather than the trading driver. Both outcomes are structurally fine; the point is the choice is made by data.

The five upstream projects keep running regardless. Headwater publishes its briefs. Estuary computes its consensus clusters. PrivateEye decodes its teasers. Tributary classifies its 8-Ks. Goldfinch matches its contracts to tickers. Each project is independently valuable; each one feeds Stalker; none depends on Stalker for its purpose.

The five-feeders-and-a-trader architecture is the part I'm most certain about. The factor stack might need to evolve. The LLM layer might get retired at horizon. The risk constants might need adjustment under live conditions. But the producer-consumer pattern, the per-project SES inbound endpoint, the lenient Pydantic schemas, the deterministic IDs, the bias-corrected backtest discipline, the pre-registered A/B for the load-bearing architectural choice — those are durable. They're the parts I'd rebuild the same way if I started over.

Three years ago I built a Slack bot that generated stock reports. It told you what one stock looked like.

Stalker tells you what to do about it. It runs on its own. It's written down what it expects to see and how it'll know whether it was right. And it's surrounded by five other projects that do the work of making structured data available in the first place — because the trading layer is the smallest part of the system, and the data layer is where most of the leverage lives.

The work continues.

What Terra Populus Taught Me About Cancelling Quiver

A.C. Jokela — Sun, 26 Apr 2026 13:00:00 GMT

I joined Terra Populus in 2012 as a senior engineer. It was a project at the Minnesota Population Center, now part of the Institute for Social Research and Data Innovation at the University of Minnesota, and it was funded by an NSF cooperative agreement totaling about \$8 million across the life of the project. The output was free to the research community. The work that produced it was not.

In 2013 the original leadership of the project left and I started inheriting the work. Nobody hands you the lead role on a federally-funded data integration project the way you'd hand someone a clean repo and a Friday lunch. You inherit it the way you inherit a house with the heating system halfway through a retrofit — there's a budget, there's a deadline, there's a researcher in Massachusetts who needs a specific harmonized variable to ship a paper, and there's a stack of NSF reporting requirements that don't pause while you figure out what's going on. By 2014 the role was formalized and I led the engineering through to the project's wind-down in late 2016 and early 2017.

I've been thinking about that experience for the last week because I'm cancelling a Quiver Quantitative subscription. The Quiver tier I'm on costs somewhere in the \$30 to \$70 per month range and it's the one with the alternative-data feeds I needed for a recent research project. The project is done. The subscription renews in a few weeks. I have a decision to make.

What surprised me, sitting with the decision, is how much my comfort with making it draws on the years I spent building the academic version of what Quiver does. The two services look nothing alike on the surface. Underneath, they are doing exactly the same shape of work. And the question of who pays for that work — and how — is more interesting than any cost-per-month arithmetic.

What Terra Populus Actually Did

Terra Populus integrated three things that don't natively join: population microdata (people, households, decade-by-decade), environmental data (climate, land cover, atmospheric measurements), and land-use data (parcels, agricultural designations, administrative boundaries). The pitch to NSF was that researchers studying the relationship between human populations and the natural environment shouldn't have to spend two years writing crosswalks before they can ask their question.

The user-facing version of the project shipped under the IPUMS Terra brand and ran inside the existing IPUMS data-extraction infrastructure. The screenshot above is what a researcher saw when they came to ask the question we were trying to make askable: pick a population dataset, pick an environmental or land-use dataset, choose the join structure that fits your analysis (microdata, area-level, or raster), and pull a custom extract. The three little nodes in the middle of that diagram look simple. Most of the engineering effort in the project was in making them simple.

The hardest specific problem we ran into, over and over, was geographic boundary harmonization across census decades. Census tracts redraw every ten years. A neighborhood that was one tract in 1990 might be three tracts in 2000 and two tracts in 2010 with completely different shapes. A researcher who wants to study, say, neighborhood-level income trajectories from 1980 to 2010 cannot just join the 1980 file to the 2010 file on tract ID. The tract IDs don't refer to the same places. The boundaries shift. Pieces split. Pieces merge. Pieces get absorbed into adjacent tracts when populations decline. The whole map breathes between decades.

The honest answer to "what does the same place look like across these boundary changes" is that there is no single right answer. There are defensible answers that depend on what you're studying. If you're tracking land-use change you probably want fractional area weighting — give a 2010 tract that overlaps 60% with a 1990 tract sixty percent of the 1990 measurement. If you're tracking population movement you want something different, because populations don't distribute uniformly across area. If you're tracking voting patterns you want yet another thing, because voting precincts don't align with tracts at all.

We made decisions. We documented them. We defended them in deliverable reports to NSF every year. We versioned them, because once a researcher cited a specific harmonized variable in a published paper we couldn't silently change what that variable meant in the next data release. Schema discipline as a research infrastructure problem, not a code problem. Every variable we shipped came with a paper trail that explained what it was, what it wasn't, and what alternative we considered.

Spatial harmonization was only the worst-named version of a more general problem. The same shape of decision had to be made on the temporal axis. Population microdata in the US is decadal. Environmental data is hourly, daily, monthly, depending on the source. Land-use data is irregular — it changes when somebody updates the parcel records, which can happen on any timescale from weeks to decades depending on jurisdiction. Joining a 2010 census tract to MODIS satellite imagery for the same area requires a decision about what time window of imagery counts as "matching" the 2010 measurement. We documented that decision too. It was different from the spatial decision, defended in different sections of the same NSF report, and tested by different researchers downstream.

The harmonized output sat on a server somewhere and was free to download. The labor of arriving at it was funded at roughly \$1.6 million per year for five years. Engineers, researchers, project managers, the costs of running servers, the costs of presenting at conferences and writing the reports that justified the next year's allocation. It was a lot of money. It produced a finite, citable, queryable dataset that hadn't existed before, and would not have existed without that money.

The Funding Model

NSF cooperative agreements are a particular kind of grant. Unlike a standard research grant, where the agency funds you and then mostly leaves you alone, a cooperative agreement keeps NSF involved in the project's direction. There's a program officer who attends meetings. There are quarterly check-ins. There are annual reports that include not just what you spent and what you produced but where you're going next and why. There are presentations, in person, to people whose job is to ask hard questions about whether the work is worth what it costs.

This structure exists because the agency is making a public-good argument. Federal money pays for outputs that meet certain criteria — they have to be broadly useful, they have to serve a research community that can actually use them, and they have to produce evidence of impact. If your data is sitting on a server and nobody is downloading it, the next funding cycle is going to be a hard conversation.

The trade-off is that this model only works for cross-domain data services that have a research community organized enough to push for them. The Census has one. Climate data has one. Land-use and population integration has one. Alternative financial data — the kind of feeds that say "here's what Pelosi traded last week" or "here's what Cramer mentioned on Mad Money" — does not. There is no academic constituency lobbying NSF to fund harmonized politician-trading disclosures. The data exists; the demand is real; the funding model isn't there.

The other thing the academic model doesn't solve is what happens when the grant ends. Terra Populus wound down in late 2016 and early 2017. New harmonization work, new boundary updates, new integrations — those mostly stopped when the funding stopped. Sustainability is the fragile part of the academic public-good model. You can produce a high-quality dataset for five years and then watch it slowly decay because there's no organizational reason to keep the engineering staffed. I left the University of Minnesota in early 2019, so, I have less knowledge as to what the legacy of Terra Populus is at ISRDI; I'm sure lessons were learned and some of the data itself likely shifted into other projects.

What Quiver Actually Sells

Quiver Quantitative sells the same shape of work for financial alternative data. They are a commercial service, not a federally-funded one, and the visible product is a paywalled API and a public-facing dashboard. But the engineering underneath is recognizable. I know it on sight because I spent seven years doing the academic version of it.

A few examples of what Quiver actually does, beyond "scraping public data":

A politician's personal trade disclosure shows up on the House STOCK Act feed under whatever name the politician uses. That name has to map to a BioGuide ID. The BioGuide ID has to join to the politician's party affiliation, chamber, and committee assignments — at the date of the trade, not at the date of the disclosure, which can be months later, and not as of today, because the politician may have switched committees or parties since. The join logic has to handle people who change their legal name mid-term. It has to handle filers who use a spouse's name on the disclosure. None of this is intellectually deep. All of it is the kind of thing that fails silently in a hand-rolled scraper if you're not careful.

A corporate jet flight is reported by tail number. The tail number is owned by an LLC. The LLC is owned by a holding company. The holding company is owned by an executive, or by a fractional-ownership program, or by a trust whose beneficiary is the executive. The executive worked at Company A from 2018 to 2022 and at Company B since then. If you want to attribute the flight history to "the Company A CEO's plane" correctly across that transition you need to maintain the ownership graph and apply it temporally, with an understanding of when M&A and management changes happened.

An insider trade reported on Form 4 belongs to an officer, a director, a 10% beneficial owner, or a relative. The role hierarchy matters because the predictive content of the trade depends on the insider's position. The company's filing history runs through name changes, ticker swaps, and reorganizations that would break a naive join on stock symbol.

A WallStreetBets ticker mention has to be disambiguated from English words ("ANY", "REAL", "GO", "ALL"), from financial terms used as common nouns ("CASH", "BANK", "DD"), and from the constantly-shifting community vocabulary that makes "Roaring Kitty" mean something specific to the reader on a specific date. Sentiment scoring on those mentions has to track what counts as bullish in the WSB dialect, which is not the same as what sentiment models trained on news text think bullish means.

The hard part of all of these is maintenance. Source formats change. Disclosure requirements add fields. Politicians change parties. Companies merge. Tail numbers transfer. New tickers IPO; old tickers delist; ETFs split into share classes. Quiver writes those decisions into customer-facing changelogs and SLAs instead. The decisions are no less real.

What you pay for, when you pay for Quiver, is the labor of making those decisions and the sustainability of having an organization that keeps making them. The data feed is the visible artifact. The engineering function is the actual product.

What You Take On When You Cancel

A third funding model exists alongside the academic-grant and commercial-subscription models: pay with your own time. Build it yourself. Maintain it yourself. Absorb every format change personally, on your own schedule, without an SLA or a changelog or a research community to share the burden.

If you cancel a service like Quiver, here is the inventory of what you take on:

Name disambiguation. Politicians, companies, executives, all changing identifiers and roles over time. Every join needs to know which version of the entity it's joining.
Ticker stability through corporate actions. M&A, name changes, share class splits, delistings. The historical price series for Activision Blizzard ends in October 2023 because Microsoft acquired it; if you want to study Activision's history you need to know that and stitch it together.
Schema normalization across changing source formats. House STOCK Act PDFs and Senate eFD filings have different schemas. SEC EDGAR Form 4 has had three or four major XBRL revisions in the last decade. Every change is a parser update on your end.
Audit trail and reproducibility. If you want anyone — including future you — to be able to re-run your analysis or cite your data, you need to record what you used, when you pulled it, and what cleaning decisions you made. This is the part that sounds like overhead until it isn't.
Format-change response time. When the upstream source publishes a new PDF template with shifted column boundaries, you have days to fix the parser before the data gap shows up in your analysis.
Adversarial inputs on the source side. The upstream entities have their own incentives. A politician's disclosure form filled out the legal minimum way is not necessarily the version that makes downstream joins clean. Filers transpose digits in CUSIPs. They report a transaction date that's the settlement date instead of the trade date. They list a partial name for a holding LLC that you have to disambiguate against three plausible matches. None of this is fixed by trying harder; it's fixed by accumulating a library of known patterns and a tolerance for ambiguity.

What you give up: coverage, consistency, the audit trail that makes the data citable in research papers, and the option to forget about the data layer entirely while you focus on the question you actually wanted to answer. What you keep: full control over the corner cases that matter to your specific question, and the ability to make different cleaning decisions than the service made.

I just finished a study of Cramer's Mad Money recommendations as a quant filter signal — write-up coming separately — and the data work for that project is what made the engineering layer concrete to me again. I needed specific segment classifications that didn't quite match Quiver's defaults. I needed corner-case handling for tickers that were renamed mid-window. I built most of that infrastructure anyway. The subscription was a useful starting point, but the actual analysis lived in code I owned.

I Live Here Already: DirtScout

The Cramer study is the recent example. The longer-running one is DirtScout, a land acquisition platform I've been building. Strip away the user-facing parts and DirtScout is essentially a series of custom data ingestors for county and state level real property information. It is the DIY funding model lived, not described.

The pipelines that keep DirtScout fed:

Minnesota statewide parcel data, refreshed quarterly. Every county in the state has its own schema for what counts as a parcel record, with different field names, different geometry encodings, and different conventions for whether vacant land is a separate row or a flag on the residential record. Harmonizing those into a single queryable parcel table is the largest single piece of cleaning work in the project.
St. Louis County weekly, with tax-delinquent enrichment. The county publishes a delinquent register that uses different identifiers than the parcel records, and the join is fragile in exactly the way you'd expect — name spellings drift, owner addresses change, parcel splits get attributed differently between the assessor's database and the treasurer's database. Cross-source enrichment is where most of the corner-case handling lives.
Tax Forfeited Land sale lists, scraped daily off an SSRS-rendered report. The format is defined by Microsoft's reporting service and emitted as the world's least-loved HTML table. The report definition changes occasionally and the parser has to track it.
Elevation profiles, computed from USGS topographic data and joined to parcel polygons on demand. Cross-domain integration: real property data and digital elevation models live in different schemas, projections, and coordinate systems, and joining them properly requires a stack of decisions that look exactly like the ones we made joining environmental rasters to census tracts in Terra Populus.

I keep all of it running with cron jobs on a Linux box and weekly integration tests. The cron jobs say pull the new data; the integration tests say scream if the join logic broke. Together they constitute exactly the same kind of ongoing engineering that went into NSF reports, and exactly the same kind of function Quiver writes into customer-facing changelogs and SLAs. The funding model is different — DirtScout is funded by my time and the electricity bill on a workhorse machine — but the labor is recognizably the same shape.

That's the DIY model in actual practice rather than as an abstraction. When I cancel Quiver, I'm not adding a new category of work to my life. I'm adding one more pipeline to the maintenance surface I already carry. The cancel decision is about whether the marginal pipeline is worth the marginal time, given the alternative of paying for the labor.

The Decision

The arithmetic is simple. Thirty to seventy dollars per month, times twelve, equals three hundred sixty to eight hundred forty per year for the Quiver tier I needed. The cleaning labor for the Cramer study, in hours of my own time, came out below that — partly because the project window was bounded (no ongoing maintenance burden after publication), partly because the corner cases I cared about were corner cases Quiver wouldn't optimize for in any case, and partly because the subscription's marginal value to me dropped to near zero the moment the study finished its primary analysis.

So I'm cancelling.

This is a verdict on the use case, not on the service. Quiver is correctly priced for what it does. The labor it absorbs on behalf of its customers is real, valuable, and the kind of work I have personal sympathy for because I used to do its academic equivalent. The question is not "is this overpriced." The question is "does what they do match what I need on this specific project, on this specific timeline, with this specific willingness to absorb maintenance personally."

For me, right now, the answer is no. For someone running a fund and consuming alt-data continuously, the answer is obviously yes. For an academic researcher with grant funding and a multi-year project, the answer probably depends on whether the institution will pay. For a hobbyist with one specific question and time on their hands, the answer is probably no. Different use cases, different absorption preferences, different right answers. The subscription is a tool, and like any tool it's correctly chosen against a specific job.

Three Funding Models, One Bill

The labor doesn't disappear in any of these models. The variable is who absorbs it.

Academic grants socialize the cost across taxpayers in exchange for outputs that meet public-good criteria and serve a research community. The output is free to use. The sustainability is fragile, because the moment the grant ends the engineering staff disperses and the dataset starts ossifying. Terra Populus is one of dozens of NSF-funded data integration projects that produced excellent work for a fixed window and then went into a maintenance-mode purgatory or were eventually decommissioned.

Commercial subscriptions price the cost into the customer relationship. The output is paywalled. The sustainability is more robust, because as long as renewals cover engineering payroll the work continues — but the paywall means the data is unevenly distributed. Some questions that should be asked about Cramer or Pelosi or insider trading get asked only by people willing to pay for the data layer.

DIY personalizes the cost into your own time. The output is yours. The sustainability is one upstream format change away from breaking, and the audit trail depends entirely on your discipline. Most DIY data work decays for the same reason most personal projects decay: the maintenance burden is invisible until it isn't.

When somebody tells you a commercial data service is overpriced, ask them what they're proposing to do with the labor. The answer is always one of those three models, and the right answer depends on what you're actually optimizing for. If you want broad public access to a citable dataset, the academic model is correct and the price tag (which is real, just paid by someone else) is justified. If you want consistent, maintained, audit-trailed access on a service-level agreement, the commercial model is correct. If you want full control over the corner cases and have time to spend, DIY is correct.

There's a Jevons-flavored pattern hiding here. Cheaper data access creates demand for more data work — more analyses, more cross-cuts, more downstream products built on top. The labor savings at one layer (the cleaning is already done; just query it) get spent at the next layer (now the question becomes "what does this dataset tell me," and that question has no upper bound). The bill is real either way. The form it takes shifts depending on how the supply side is funded.

The same logic, weirdly, applies to a different kind of subscription decision I wrote about a few weeks back. Local LLM inference versus a Claude Max subscription is the same shape of question as Quiver versus DIY scraping: who absorbs the cost, what coverage do you get, how fragile is the sustainability, and what specifically are you optimizing for. The arithmetic is different. The structure isn't.

The subscription cancellation in front of me this week is small — a few hundred dollars saved per year, in exchange for some weekend hours I'd have spent on the data layer anyway. The bigger thing it's clarified is the model I want to use when I make the next one.

What a Commercial PCB Placer Does That My Open-Source One Can't

A.C. Jokela — Fri, 24 Apr 2026 15:30:00 GMT

This is an interlude in a running series on designing a level-shifter shield for the Arduino Giga R1. The main arc is about the design itself — how to get 72 channels of logic-level translation across a 155mm x 90mm board using ten SN74LVC8T245 shifters, in a repeatable Python-scripted workflow. Previous posts covered the original Fiverr design, the Claude Code redesign, a pin-numbering bug, and the v0.4 respin.

This post is an honest comparison of two ways to place the components on that board: Quilter, a commercial service aimed at exactly this problem, and pyplacer, an open-source simulated-annealing placer I wrote over two weekends specifically to have something to benchmark Quilter against. The results are not flattering to pyplacer, but they are interesting. They also surface a concrete and narrow thing that commercial placement services do right that a naive first pass does not.

Why Write a Placer At All

A PCB has two distinct layout problems. The first is placement: deciding where each component goes on the board. The second is routing: deciding how to connect the pins once the components are placed. These are usually solved by different tools, and the quality of the placement determines how well the routing can go. A bad placement makes a good routing impossible; a good placement makes routing almost boring.

There is no good open-source placement tool. Freerouting, which is genuinely excellent as an autorouter, does not do placement — it takes placed components as input. KiCad has a rudimentary "move to group" feature and nothing else. Eagle has nothing. pcb-rnd has nothing. For decades, PCB placement has been a human-only activity, and the industry's answer to "automate placement" has been to employ better humans.

A few commercial services are now genuinely trying to change that. Quilter is one of them: you upload a KiCad board with component libraries and a netlist, it returns a placed-and-routed board. I approached Quilter about partnering on a post and I decided that a straight review would be uninteresting. What I actually wanted to know was: how much of the gap between Quilter and "do nothing" could be closed by a motivated hobbyist with a weekend, some Python, and a simulated annealer? If the gap is small, Quilter is an incremental improvement. If the gap is large, Quilter is doing something specific that is hard to reproduce.

So I built pyplacer. The goal was not to beat Quilter — that would have been foolish. The goal was to produce a credible open-source baseline, run both tools on the same board under the same constraints, and see how the numbers fell.

The Board

The benchmark is giga_shield v0.4 — the same board discussed in the previous post. Relevant specs:

155mm x 90mm, 4-layer stack (two signal, two plane)
161 nets across 64 components
Ten SN74LVC8T245PW level-shifter ICs in TSSOP-24 packages
Two 36-pin 2x18 headers (J9 and J10) along the right edge for the Z80 side
Several 8-pin and 26-pin single-row headers for the Arduino Giga side
Forty 0603 bypass and DIR-control caps/resistors
All components on the top side of the board

This is a dense layout for 161 nets on 155x90mm. The level shifters are the routing chokepoint: each one has 8 A-side pins that connect to one header and 8 B-side pins that connect to another, plus VCCA/VCCB/GND/DIR. Get the shifters wrong relative to the headers and the board is unroutable at 4 layers.

The honest baseline for comparison is not "random placement." It is my hand-placed v0.4 layout, which Freerouting routes to 100% at 0.3mm clearance in about 45 minutes of wall-clock time. I know that because I spent several weeks iterating to produce it. The goal of any automated placer is to do that work so I do not have to.

Quilter: The Commercial Option

Quilter presents as a web service: you upload your .kicad_pcb (unplaced) with its component libraries and netlist, set a few parameters (board outline, fixed components, target layer count), and they generate candidate placements.

For this comparison I submitted giga_shield v0.4 with the ten shifters and all passives unplaced, the four headers and mounting holes locked in position (because those are mechanical constraints tied to the Arduino Giga's hardware), and the board outline fixed. Quilter returned Candidate_1 after 1 hour 10 minutes 48 seconds of compute.

Quilter Candidate_1, rendered via kicad-cli pcb render. Ten shifters spread into three clusters, each one closest to the connectors it serves. Routing threads through every available channel.

Freerouting, given Quilter's placement and the same 0.3mm clearance rule I used for my hand-placement, routed 99.4% of the board: 1 net unrouted out of 161. The single unrouted net was a non-critical DIR control signal that could have been fixed with a manual jumper or a small placement tweak. For practical purposes, the board is fabricatable as Quilter produced it.

That is a strong result. It is in the same quality neighborhood as my hand-placement, produced in about seventy minutes of their wall-clock time versus my several weeks of iteration.

pyplacer: The Open-Source Attempt

pyplacer is about 2,000 lines of Python. It does simulated annealing over component positions on a fixed board outline, with a cost function that combines half-perimeter wirelength, bounding-box overlap penalties, an out-of-bounds penalty, a grid-cell congestion estimate from probing L-shaped routes across a coarse mesh, and a small pad-exit direction bias that tries to align TSSOP-24 pads with their natural routing sides.

The overall architecture is pragmatic rather than clever:

A KiCad board loader that reads footprints, pads, and netlist
A heuristic seed placement that puts each shifter at the midpoint between its two dominant connectors (the connectors that account for the most of its signal pins), with special handling for clusters of shifters sharing the same connector pair
An SA loop: propose a move (shift, swap, or component-to-component swap), evaluate the cost delta, accept with Metropolis probability at the current temperature, cool
A DSN exporter that writes a Specctra file with the placed positions and the netlist, ready to feed to Freerouting

The cost function I arrived at after maybe a dozen tuning rounds:

W_HPWL           = 1.0      # baseline wirelength
W_OVERLAP_HARD   = 1000.0   # no two components may overlap
W_OVERLAP_SOFT   = 50.0     # keepout zones around components
W_OUT_OF_BOUNDS  = 200.0    # stay on the board
W_CONGESTION     = 40.0     # L-shape probe-based congestion
W_PAD_EXIT       = 5.0      # bias pads toward natural routing sides

Everything else — rotation moves, attachment-to-dominant-connector, displacement-from-heuristic — I tried and disabled, because each one either produced worse results or did not produce a measurable improvement.

The SA run itself is 5,000 iterations at 0.96 cooling with a 1e-4 final-temperature ratio, taking about 90 seconds on a single core of a Mac M3 Pro. That is fast enough that I could do broad seed sweeps cheaply.

For the benchmark, I ran pyplacer on 64 distinct random seeds across three machines (my Mac, a 32-core Linux host, a 64-core Linux host), routed every resulting placement through Freerouting at the same 0.3mm clearance, and recorded the number of unrouted nets. The seed distribution matters because SA is stochastic: different seeds give different local minima, and the spread of outcomes is informative.

The Numbers

Over 64 seeds:

Best result: 10 unrouted nets (93.8% routed) — this was the early best from the first batch of runs
Typical plateau result: 16–20 unrouted nets (87.6%–90.1% routed)
Worst result: 114 unrouted nets (29.2% routed)
Median: 21 unrouted nets (87.0% routed)

Every single pyplacer run was worse than Quilter's Candidate_1. The gap between the best pyplacer seed and Quilter is about six percentage points of routing completion, which on a 161-net board is nine to fifteen missing traces. In practice, a board with 10–16 unrouted nets is not fabricatable as-is. It needs hand-fixing, which is exactly the work the placer was supposed to save.

A pyplacer placement after a Freerouting pass. Notice how the shifters are crammed into a single cluster hugging the left edge of J9, and the routing channel on that edge is visibly dense. This layout routes to about 90%; the unrouted nets are all in the congested zone on the left side of J9.

Compare that image to the Quilter render above. The topological difference is immediate: Quilter spreads its shifters into several clusters, each close to the connector or connector-pair it serves. pyplacer piles them all into the shortest-wirelength position, which happens to be a narrow strip where routes cannot fan out. The visual story matches the numbers.

Which Nets Failed

I logged Freerouting's warning-level output across the entire 64-seed sweep and extracted the nets that consistently failed to route on the best seed's plateau. Seventeen candidate nets, of which sixteen appear in any given plateau pass. The surprising thing is not the count — it is the topology:

Net	Pins
D23	J9-4 ↔ U8-3
D25	J9-6 ↔ U8-4
D27	J9-8 ↔ U8-5
D29	J9-10 ↔ U8-7
D31	J9-12 ↔ U8-8
D33	J9-14 ↔ U8-9
D35	J9-16 ↔ U8-10
D37	J9-18 ↔ U9-2
D39	J9-20 ↔ U9-3
D41	J9-22 ↔ U9-5
D43	J9-24 ↔ U7-3
D45	J9-26 ↔ U7-5
D47	J9-28 ↔ U7-8
D49	J9-30 ↔ U7-10
D51	J9-32 ↔ U10-4
PJ5	J10-16 ↔ U8-13
PB1	J1-4 ↔ U2-5

Fifteen of the seventeen are the exact same topological class: a short digital signal from a level shifter to a sequential pin on the J9 double-row header. These are the easiest routes on the board. They fail because pyplacer has piled all four shifters (U7, U8, U9, U10) up against J9's left edge, and sixteen signals are trying to enter J9 through a 10mm-wide routing channel that can physically only accommodate maybe eight parallel traces.

This is a connector-edge congestion failure. It is not a routing problem. It is a placement problem that the router cannot undo.

Why pyplacer Gets This Wrong

Half-perimeter wirelength rewards placing U7/U8/U9/U10 as close to J9 as possible, because every one of their B-side pins connects to J9 and the distance matters. The bounding-box penalty keeps them from overlapping each other. The congestion estimate does catch some of this — it is why pyplacer's placements are not as awful as they would be without it — but the resolution of the congestion grid is too coarse to catch what is happening at J9's edge specifically. The cost function sees "OK, the shifters are stacked but they are not literally on top of each other, and the average 2x2mm cell is not saturated." The router sees "sixteen traces trying to enter the same five-millimeter gap, no thanks."

Fixing this correctly is not just a matter of tuning weights. The cost function is missing a dimension. Specifically, it needs a term that penalizes incoming pin density at a fixed connector's edge — a count of how many signal nets want to enter a given side of each fixed header per millimeter of edge, with a soft threshold above which the penalty climbs sharply. A naive implementation of this adds about 50 lines to the cost function and a per-move incremental update. I have not yet written it because the interesting finding of this benchmark is not "how do I close the gap" — that is the next post. The interesting finding is where the gap lives.

Quilter, whatever it is doing internally, is not piling shifters up against J9. It spreads them. U7, U8, U9, and U10 in Quilter's placement are in a 4x1 tight column arrangement a few millimeters away from J9's edge, with gaps between each chip that are deliberately large enough for routes to fan out between them. That does not minimize wirelength. It maximizes routability. The two are not the same objective, and pyplacer only optimizes the first.

Speed and Cost

A fair comparison also needs to account for what each approach costs to produce.

Quilter: Candidate_1 took 1h 10m 48s of their wall-clock compute. Pricing is on their website; I received this candidate through my partnership with Quilter and did not personally pay. One good candidate is enough for a finished board, so the total spend for a project like this is whatever a single candidate costs, not a subscription or a retainer.

pyplacer: A single seed takes about 90 seconds of single-core wall-clock time. A 64-seed sweep with partial parallelism across three machines took about 35 minutes of real time and consumed roughly three CPU-hours. The Freerouting pass that evaluates each placement takes 10–15 minutes per candidate, so evaluating all 64 placements took an additional ~15 CPU-hours, mostly on Linux hosts. Electricity cost is negligible — pennies. But the human-hours cost of writing, tuning, and debugging pyplacer was maybe 25 hours of my weekend time spread across two weeks.

So: Quilter produces a usable placement in about seventy minutes of their compute, at whatever their per-candidate list price is. pyplacer cost 25 hours of engineering time, some CPU, and still does not produce a usable placement. The break-even point, assuming my engineering time is valued at even a modest hourly rate, favors Quilter by a wide margin for single-project work. The calculus only flips if you are going to place hundreds of boards and amortize the pyplacer development cost across all of them. For a hobby project or a one-off prototype, it is not close.

What About Future Improvements?

The obvious counter is that pyplacer is a weekend project and Quilter has been in development for years with a team. If I added the connector-edge congestion term, fixed the coarse grid resolution, added rotation moves, and let the SA run for 10x longer, would I close the gap?

Probably partially. Maybe to 95–97%. I do not think I would close it fully. Here is why.

Quilter is doing things that are not obvious from its output but are visible in the spacing patterns: it seems to have a model of the router's behavior, of how close is "too close" for vias to cluster, of how routing channels compose across layers. A simulated annealer can be taught any cost function you can express, but the cost function has to know to include these things. Writing a cost function that captures what Quilter captures is not a weekend project. It is months of work by someone who understands routing at a deep level, which I do not.

I could also throw more compute. The 64-seed sweep took ~18 CPU-hours total. If I ran 1,000 seeds on a cluster, I would likely find a placement better than the best seed from 64, by selection pressure alone. But the ceiling set by the cost function's limitations is not breached by more samples; it is breached by a better objective function. And that objective function is where the commercial tool is earning its fee.

What This Means for the Series

The immediate practical takeaway for the GigaShield series is that if I wanted to save time on future revisions of this board, running it through Quilter and accepting Candidate_1 would be a sensible default — especially for the v1.0 redesign where I want to add thermal pads, improve the layer stack, and shuffle the power rails. That is a genuine upgrade over the hand-placement workflow, which, while competent, took me multiple weeks of iteration to get right.

The broader takeaway is that placement is a harder problem than routing. Freerouting is an excellent autorouter because routing has a clean, well-posed objective: find paths that respect clearance rules and minimize length. Placement has no such clean objective. A "good" placement is one the router can finish, which is not something you can compute directly from the placement — you only find out by running the router. That feedback loop makes placement the sort of problem where stochastic search without a good model does not close the gap with humans or with informed commercial tools.

I am going to keep iterating on pyplacer because I find the problem interesting and because an open-source baseline has value even when it is worse than the commercial alternative. The connector-edge congestion term is the first thing I will add. After that, I want to look at rotation moves in the SA, because the ability to flip a shifter's A/B-side orientation is exactly the sort of thing that can unclog a congested connector edge at the cost of a small HPWL penalty.

But I am not going to pretend I am closing the gap to Quilter through cleverness alone. Whatever Quilter is doing, it is working. For a dense board with real constraints and a real fabrication deadline, their output is better than mine.

The Honest Review

Companies that partner with me on posts sometimes expect the post to soften the criticism of their tool. Quilter, to their credit, did not. I uploaded my own board, they ran it through their service, and they let me analyze the output however I wanted. The result is that this is a post about an open-source placer that loses to a commercial one, not a post about a commercial tool defeating a straw man.

For the specific use case I built this around — dense four-layer mixed-signal shields with 150+ nets and tight routing channels — Quilter's Candidate_1 is better than what I can produce with a weekend placer. The gap is not close. The gap is specifically at connector-edge congestion, which is a place where my naive objective function fails and theirs does not. That is a real and informative finding, and the right conclusion is that whatever Quilter charges per candidate is a reasonable price for what you get.

I would recommend them for boards like this one. I would not recommend them for dead-simple boards where a hand placement in KiCad takes twenty minutes — that is still the cheapest path for trivial work. The interesting threshold is somewhere in the range of 50–100 nets and a non-trivial connector layout; below that, do it yourself; above that, pay the pros.

Previous posts in the GigaShield series: What Routing 314 Nets Taught Me About AI-Assisted PCB Design · How a Pin Numbering Bug Killed a PCB · Redesigning with Claude Code (Part 1) · Fiverr PCB Design (\$468)

Architecture Verified, Mythology Intact: Running OpenMythos on a Strix Halo

A.C. Jokela — Mon, 20 Apr 2026 13:00:00 GMT

Anthropic has a rumored upcoming model called Mythos. The weights are not public, the architecture is not published, and Anthropic has said nothing official about how it works. That has not stopped people from guessing.

OpenMythos is one of those guesses: an open-source "theoretical reconstruction" by Kye Gomez, built from publicly available research on what Anthropic's architecture might look like. The repository's disclaimer is blunt: "an independent, community-driven theoretical reconstruction based solely on publicly available research and speculation. It is not affiliated with, endorsed by, or connected to Anthropic."

The architecture Gomez bets on is called a Recurrent-Depth Transformer. That's a specific and unusual design choice. Most current language models, like GPT or Llama, are feed-forward: tokens enter at the bottom, flow through dozens of distinct layers stacked on top of each other, and exit as predicted next tokens. A Recurrent-Depth Transformer splits that stack differently. A small number of ordinary layers run once at the start and once at the end. In between, a single layer runs many times in sequence, with the output of each run fed back in as the input to the next. Same weights. More computation.

You can pip install OpenMythos. It has configurations from mythos_1b (one billion parameters, toy scale) up to mythos_1t (one trillion, frontier scale). The README shows you how to instantiate the 1B version in about ten lines of Python and run a forward pass.

I ran that 1B variant on my Strix Halo box (a Ryzen AI MAX+ 395 with an integrated Radeon 8060S GPU, 60 GB of unified memory, running PyTorch on ROCm). The question is not whether it runs. The question is what running it can tell you. The answer turns out to be interesting in both directions: more than expected about the architecture, and exactly nothing about the model.

The Setup

The Strix Halo has one GPU. OpenMythos targets distributed training via FSDP, but the forward and inference paths work single-GPU. Install path:

pip install --no-deps open-mythos

The --no-deps matters. The pyproject.toml pins torch = "2.11.0", which is not what my gfx1151 wheels are at, and the package's actual runtime requirements are satisfied by any torch >=2.1. Skipping deps keeps my ROCm stack intact.

One ROCm-specific patch was needed. The DepthWiseLoRA module's forward has this line:

s = self.scale(torch.tensor(loop_t, device=x.device))

That creates a 0-dim tensor and passes it to an nn.Embedding. On gfx1151 this produces a hip launch failure. The fix is a one-line change to index the embedding weight directly:

s = self.scale.weight[loop_t]

Same semantics, different kernel path, no crash. Expect similar papercuts in any research code run on non-reference hardware.

With that done, mythos_1b instantiates cleanly. Parameter count: 1,064,028,034. A forward pass at batch 1, sequence 128, 16 loops, using bfloat16 mixed precision (a numerical format that halves memory versus regular float32 with negligible quality loss for inference): 2.07 seconds. Peak GPU memory: 6.44 GB. Well within the Strix Halo's envelope.

That gives me a working model. The rest of this post is what I did with it.

Why a Looped Transformer, Briefly

Before the experiments, a quick tour of what's specifically weird about this architecture, because everything downstream depends on it.

A standard transformer has roughly 32 to 100 distinct layers. Each layer has its own parameters. A prompt passes through every layer once. The parameter count is proportional to the layer count times the width of each layer.

A looped transformer keeps only one "inner" layer but runs it many times. Training on 32 "effective layers" requires only 1 layer's worth of parameters. Inference with more loops is equivalent to running a deeper model, without actually storing a deeper model. The architectural bet: if you can get this to work, you get a deeper reasoning model for a fraction of the memory.

There are two reasons to care about this for a model like Claude Mythos. First, memory efficiency at scale. A trillion-parameter model is expensive to serve; a looped model with the capability of a trillion-parameter feed-forward model but 1/16th the parameters would be dramatically cheaper. Second, reasoning depth. A 2025 paper by Saunshi et al. proved mathematically that running a looped transformer for T loops is equivalent to doing T implicit steps of chain-of-thought reasoning (the now-familiar "let me think step by step" trick that makes large models better at hard problems), except the "thoughts" happen in continuous latent space inside the model rather than being emitted as visible text tokens. If Mythos is doing that, it would explain why the model seems to do multi-step reasoning without the user ever seeing intermediate "scratch" tokens.

The catch is that training a looped transformer is notoriously unstable. If the single inner layer amplifies the signal each time it runs, that amplification compounds. A 5% boost per loop becomes a 65% boost after 10 loops, and a model with a 65% boost per forward pass either explodes in training or produces outputs that don't resemble language. Most attempts at looped transformers over the last decade failed for exactly this reason.

The fix that makes OpenMythos (and the hypothesized Mythos) workable is borrowed from a 2026 paper called Parcae (Prairie et al.). It introduces a clever parameterization of the "gain" of the recurrent update. Instead of letting the model learn arbitrary weights in the core recurrence, Parcae constrains one piece of the architecture to always have its largest amplification factor strictly less than 1. In dynamical-systems terms, the "spectral radius" of the update matrix is always less than one. That guarantee is what makes the loops stable: any signal gets damped by each repeated application, so repeated iteration converges toward a useful fixed point instead of blowing up.

This is the claim I'm about to test. The spectral radius should be less than 1 by construction. Without that constraint, training should break. And the failure mode should match what Parcae predicts.

What I Verified

1. The spectral radius at initialization

The cleanest possible check: the Parcae paper claims a specific mathematical structure for the stability-guaranteeing matrix. Starting from the parameters' default initialized values of zero, the formula works out to a single number: exp(-1) = 0.3679. If the code matches the paper, a fresh mythos_1b should have its key matrix set to exactly that value everywhere.

The parameterization in the code is:

A = exp(-exp(log_dt + log_A))

With log_A and log_dt both initialized to zero, that becomes exp(-exp(0)) = exp(-1) = 0.3679. The matrix in question is diagonal, meaning it's effectively a list of numbers rather than a two-dimensional grid, so the spectral radius (technical definition: magnitude of the largest eigenvalue) reduces to the largest absolute value in that list. At initialization, every entry is the same 0.3679.

Measurement on the instantiated 1B model:

log_A init value (first 5): [0. 0. 0. 0. 0.]
A min: 0.367879
A max: 0.367879
rho(A) at init: 0.367879

Matches the theoretical prediction to six decimal places. The constraint is doing what the paper claims. This is the kind of thing you can only verify by actually running the code, because documentation and papers often drift from implementations, and subtle bugs in implementations of clever mathematical constructions are common.

2. The loops are not a no-op

Next question: do the loops actually do anything? The README claims each loop iteration is "functionally equivalent to one step of chain-of-thought." In practical terms, that means running more loops should produce different (and presumably better) output than running fewer. If the recurrent block has learned to do nothing, or if the architecture happens to be set up such that the injection of the original input drowns out everything the loop contributes, then all loop counts would produce identical outputs and the whole looped-transformer idea is moot.

The cleanest test: take the same input, the same random-initialized model, and run it with different loop counts. Compare the outputs. For each token position, the model predicts a probability distribution over the next token. Two different ways to compare:

Argmax agreement is just "for what fraction of positions does the most-likely-next-token come out the same?" If two runs pick the same top token 95% of the time, they mostly agree. If they agree 35% of the time, they're meaningfully different. The comparison below uses the 16-loop run as the reference.

KL divergence is a standard measure of how different two probability distributions are, expressed in nats (units of the natural logarithm). Zero means identical distributions. Higher means more different. Intuitively: how much information is lost if you model a distribution as something other than itself.

Running a fresh, untrained mythos_1b with a fixed input and seed:

n_loops= 1: argmax agreement with 16-loop run = 35.2%   KL = 0.19 nats
n_loops= 2: argmax agreement                  = 65.6%   KL = 0.10 nats
n_loops= 3: argmax agreement                  = 72.7%   KL = 0.07 nats
n_loops= 4: argmax agreement                  = 80.5%   KL = 0.06 nats
n_loops= 6: argmax agreement                  = 88.3%   KL = 0.04 nats
n_loops= 8: argmax agreement                  = 90.6%   KL = 0.02 nats
n_loops=12: argmax agreement                  = 96.1%   KL = 0.005 nats
n_loops=16: argmax agreement                  =100.0%   KL = 0 nats

Even with random initialization, the loops do substantive work. After a single loop, only 35% of the 128 output tokens match what the model produces after 16 loops. By three loops, 73% match. By twelve, 96%. The KL divergence tells the same story from a different angle: the probability distributions converge monotonically toward the 16-loop baseline as loop count rises.

This is exactly the signature of a well-behaved recurrent system settling toward a fixed point. The loops aren't a no-op. They also aren't chaotic: each successive loop gets closer to convergence, which is what the stability guarantee predicts.

3. The stability constraint does its job

The reconstruction becomes load-bearing here. The Parcae paper claims the constraint on the matrix A is not just a nice-to-have but a requirement. Without it, they say, training diverges at aggressive learning rates. With it, training is stable.

The test: build three otherwise-identical small models (shrunk to a 128-dimensional hidden state for training speed while keeping the full looped architecture). The only difference is how the matrix A is parameterized:

stable: the shipped LTIInjection that uses the exp(-exp(...)) construction to keep A in the stable range by construction
unstable (start at 0.368): replace the clever construction with a raw learnable parameter initialized to the same value the stable version starts at
unstable (start at 0.95): same raw parameter, but initialized close to the stability boundary, to see whether training pushes it over

Trained at a deliberately high learning rate of 0.05 with 8 recurrent loops per forward pass, for 300 steps, on random next-token prediction. (The point isn't to train a good model. It's to stress-test the stability mechanism under conditions where unstable training would be expected to break.)

The metric is max|A|, the largest entry in the diagonal. For the stable version, this is the spectral radius and the theory guarantees it stays below 1. For the unstable versions, nothing guarantees anything; we're watching whether training happens to keep it bounded.

Step	Stable	Unstable (0.368)	Unstable (0.95)
0	0.368	0.418	1.000
20	0.496	0.719	1.319
60	0.480	0.749	1.340
100	0.477	0.736	1.315
200	0.474	0.700	1.251
299	0.469	0.666	1.190

Within 20 training steps, both unstable variants push at least one entry of A well above the stable version's cap. The 0.95-init case jumps past 1 immediately and stays there. Above 1 is the forbidden regime: a diagonal entry greater than 1 in magnitude means that dimension's contribution to the hidden state grows with every loop instead of shrinking. The Parcae paper says this is fatal. Does it actually kill training?

Mostly, yes. The stable variant kept producing meaningful gradients the whole way through and the loss moved (noisily, because the training data was random). Both unstable variants had their gradient norm collapse to machine zero within 20 steps and stay there. Their loss froze at log(512) = 6.238, which is the entropy of a uniform distribution over the 512-token vocabulary we used: the training signal became meaningless because the model was outputting a flat "I have no preference about any token" distribution regardless of input.

This isn't the classic way training fails. It's not the "loss explodes to infinity and the whole job crashes" failure mode most people think of. It's subtler: the recurrent state grows large enough that the final output saturates to uniform, every possible update to the weights produces the same (wrong) uniform output, so the gradients go to zero and the optimizer stops making progress. Training is effectively dead, silently.

That is a specific failure mode the Parcae paper warns about, and it is exactly what happens here when the constraint is removed.

4. Hidden states blow up by exactly the predicted factor per loop

The previous experiment showed that removing the stability constraint breaks training. This one looks at the mechanism underneath. What does "the recurrent state grows unboundedly" actually look like numerically?

The theory predicts that if the spectral radius is ρ, then after each loop the magnitude of the hidden state grows (or shrinks) by a factor of roughly ρ. With ρ < 1, repeated shrinkage by less than one converges toward a fixed value. With ρ > 1, repeated growth by more than one goes to infinity exponentially.

Setup: instrument the model to record the magnitude of the hidden state at each loop iteration. Force A to specific values from stable (0.37) through borderline (1.0) through clearly unstable (2.0). Disable ACT halting (an early-exit mechanism explained below in experiment 5) so all 8 loops run and we can see the full trajectory. Each number below is the magnitude of the hidden state measured after that loop iteration completes.

Loop	ρ=0.37	ρ=0.9	ρ=1.0	ρ=1.2	ρ=1.5	ρ=2.0
1	91	91	91	91	91	92
2	124	172	182	200	228	274
3	136	246	272	330	432	638
4	141	312	363	487	738	1367
5	143	371	454	675	1199	2825
6	143	425	544	901	1889	5740
7	144	473	635	1172	2924	11570
8	144	517	726	1498	4476	23232

Two things to notice. First, the stable column (ρ=0.37) converges to a fixed value around 143 and stops moving. Each loop shrinks the hidden state closer to an equilibrium, then settles. This is the intended behavior: a useful, computation-performing recurrent system that's doing work but not running away.

Second, the ρ=2.0 column grows by almost exactly 2× per loop after the first couple: 274 → 638 → 1367 → 2825 → 5740 → 11570 → 23232. The last three ratios average 2.02×, which is as close to the theoretical 2.0 as you'd expect given the transformer block itself contributes nonlinear noise on top of the linear dynamics. The prediction is tight.

Four loops of ρ=2.0 take the hidden state from 91 to 1367, already a 15× blowup. Sixteen loops (the designed inference depth for mythos_1b) would push it to around 10^7, which is well past the representable range of bfloat16 and would produce infinities in a real forward pass.

That is the stability analysis verified on actual silicon. The clever exp(-exp(...)) construction does what the paper says it does, and removing it produces exactly the divergence the paper says it produces, at the rate the paper says it produces it.

5. Throughput on a consumer APU

The numbers for curiosity, not for the thesis. Single prompt of 128 tokens, bfloat16, running on the integrated Radeon 8060S:

n_loops	Latency	Tokens/sec	Peak GB
1	141 ms	910	6.29
2	254 ms	503	6.37
4	501 ms	255	6.42
8	1021 ms	125	6.44
16	2071 ms	62	6.44

Latency is almost perfectly linear in loop count, which is what you'd expect: double the loops, double the compute, double the wall-clock time. The knee of the inference-time scaling curve, to the extent one exists, sits around 6 to 8 loops before the "more reasoning" benefit stops being worth the "wait twice as long" cost.

The architecture has a feature called Adaptive Computation Time (ACT) that is supposed to help here. ACT learns, per-position in the prompt, whether that token's representation has "converged enough" to stop looping. Easy tokens (a period, a common function word) should halt after a couple of loops; hard tokens (the key answer in a math problem) keep looping. In theory, this saves compute on easy tokens.

In practice, ACT had no measurable effect on throughput in my runs. Two reasons. First, ACT only breaks the loop early if every position in the batch has halted, because the GPU runs all positions in parallel and can't just skip one. With 128 positions in the prompt, the probability that every single position happens to halt simultaneously is effectively zero, so the early-exit path never fires. Second, the halting decision is made by a learned predictor, and my model was randomly initialized (not trained). An untrained model doesn't know which tokens are easy. You'd need actual training plus a mix of easy-and-hard positions for ACT to help. Neither was present in my experiments.

The Architectural Ceiling I Didn't Expect

While trying to run 24 loops for an ablation that's not in this post, I hit this error:

IndexError: index 16 is out of bounds for dimension 0 with size 16

It turned out the architecture has a small per-loop adaptation component, a tiny learnable "tweak" that's different for each loop iteration, letting loop #1 behave slightly differently from loop #8. That component is implemented as a lookup table with exactly max_loop_iters entries. If you configured the model to train on 16 loops, you have 16 entries in the table. Trying to run a 17th loop means looking up entry 16 in a 16-entry table, which fails.

This matters because one of the headline claims about looped transformers is depth extrapolation: train the model on (say) 5-step reasoning chains, then at inference time let it run 10 or 20 loops to handle harder problems than it ever saw during training. The theoretical argument is that running more loops = more reasoning depth, and this should emerge as a free capability at inference time.

The OpenMythos implementation supports depth extrapolation only up to max_loop_iters. Past that, the per-loop adaptation lookup fails. You can extend the table, but only by reinitializing it larger and resuming training. You cannot simply crank a knob at inference time.

That's a genuine constraint on the "more loops = deeper reasoning at inference" story, and it's the kind of thing you find only by trying to cross the boundary. Nothing in the README warns you about it. It's the sort of detail that disappears when a paper's theoretical claim ("more loops at inference!") becomes a concrete implementation ("a table indexed by loop number, which has a fixed size").

What I Could Not Verify

Nothing I did tells you anything about Claude Mythos.

The architecture OpenMythos implements could be exactly the Mythos architecture. It could be a reasonable guess that shares some features with Mythos. It could be entirely wrong. You and I have no way to check, because Anthropic has not published the architecture. The mythos_1b I trained is a 1B-parameter looped transformer that behaves consistently with published research on looped transformers. It is not Mythos.

This is the epistemic limit that the repo's disclaimer is trying to name. Running a speculative reconstruction tells you whether the reconstruction is internally coherent, and whether it matches the published research it claims to match. It tells you nothing about whether the reconstruction maps to the thing it was reconstructed from. No amount of running it closes that gap. The gap is closed only by information the model's creators chose not to release, and running silicon against latent belief doesn't produce that information.

So "I verified that OpenMythos's architecture works as claimed" is a real and useful statement. "I verified that Claude Mythos uses this architecture" is not something I can say, and nobody outside Anthropic can, no matter how thoroughly they run the reconstruction.

What an Open Reconstruction Is Good For

It's good for three things.

One, as a teaching artifact. There's a live research line on looped transformers. Most descriptions of it are paper-shaped: dense with notation, theorem statements, ablation tables. OpenMythos is one of the few places you can read the whole architecture as working code in a single file, with every piece named and addressable. The stability guarantee that takes several pages of the Parcae paper to motivate resolves to one line of Python:

def get_A(self):
    return torch.exp(-torch.exp((self.log_dt + self.log_A).clamp(-20, 20)))

That's the whole guarantee. You can read it, you can run it, you can verify it on your bench. The paper claim goes from abstract math to a concrete object you can measure. For anyone who wants to understand why looped transformers work and has been bouncing off the academic literature, that's worth the install.

Two, as a testbed for your own ideas. If you want to try modifying the architecture (swap which attention variant it uses, change how the experts are routed, make the loops behave differently at different depths) the code is about 1000 lines of clean PyTorch. You don't need to build a looped transformer from scratch; you can start from a working baseline and modify. The research is live and you can participate in it.

Three, as a way to calibrate your expectations. My 1B-parameter mythos_1b produces 62 tokens per second at 16 loops on the 8060S. A full-scale Mythos would be far larger and presumably run at similar or more loops per token. If Mythos is actually a Recurrent-Depth Transformer, that tells you something about the real cost of running it: every token takes the full loop count of compute, regardless of how "easy" it is. That's a different cost shape than a standard transformer, which uses the same compute per token but over a fixed number of distinct layers. You can form a rough sense of the compute-per-token ratio that a looped architecture would imply for a frontier deployment, which is useful even if you never run a frontier model yourself.

None of those three things are "I now know how Claude Mythos works." They are all "I now know things about looped transformers that I did not know before." For the blogger running an 8060S in a home lab, that's the realistic upside, and it's a larger upside than zero.

Coda: The Reconstruction as Thing

I wrote a philosophy piece earlier this week about Heidegger's distinction between things and endpoints. A Z80 on a RetroShield is a thing: it gathers a world of silicon, engineers, software history, and your own hands. A cloud API is an endpoint: it offers a contract and deliberately hides everything behind it.

Claude Mythos is an endpoint. You send tokens, you get tokens, the weights are not yours, the architecture is not yours, and if Anthropic swaps the backing model nothing changes for the caller by design. That's the whole value proposition. It refuses to gather.

OpenMythos is a thing. I have its weights. I know the parameter count down to the last entry: 1,064,028,034. I measured its internal stability matrix at initialization and watched it move across training. I watched the hidden state blow up to 23,000 when I forced the instability and disabled the halting mechanism. I know how long a forward pass takes on my specific GPU with my specific ROCm wheel on a specific Tuesday in April 2026. It gathers a whole lineage: the 2026 Parcae paper that explained the stability trick, the older research on looped transformers that it built on, Kye Gomez's speculative synthesis of the two into a candidate architecture for Mythos, the AMD gfx1151 toolchain that lets me run any of this on a Ryzen APU at all, the one-line patch I had to apply to the code, my own debugging session, and thirty minutes of my GPU's fans running at full tilt.

The thing gathers. What it gathers, though, is the reconstruction. Not the reconstructed. My 1B model is a physical artifact with measurable behavior. It is not a window onto Anthropic's internals. Those internals remain an endpoint, and the endpoint remains abstract.

Mythology intact. Architecture verified. That is what a home lab buys you in 2026.

What Routing 314 Nets Taught Me About AI-Assisted PCB Design

A.C. Jokela — Sun, 19 Apr 2026 23:00:00 GMT

This is the fourth post in a running series about designing a level-shifter shield for the Arduino Giga R1 using Claude Code and open-source command-line EDA tools. A brief map of what's come before, since the series grew past its original planned scope:

Fiverr PCB Design (\$468) — the first GigaShield, designed by a freelance contractor in KiCad. It worked for most things but broke against the Z80's tri-state bus because the auto-sensing level shifters couldn't cope with floating signals.
Redesigning a PCB with Claude Code and Open-Source EDA Tools (Part 1) — the v0.2 redesign. Replaced the TXB0108 auto-sensing shifters with SN74LVC8T245 driven shifters, generated the PCB programmatically from a Python script, autorouted, and shipped Gerbers to fabrication. (The "Part 1" in its title was meant to lead directly into a Part 2 about assembly and bring-up — but a bug derailed that plan.)
How a Pin Numbering Bug Killed a PCB — the unplanned post-mortem on why the v0.3 board didn't work. A pin-numbering convention mismatch on the dual-row headers put every signal at the wrong physical position. The fabricated board was perfect; the design was wrong.

This post is about the v0.4 respin. It is also an honest accounting of what AI-assisted PCB design actually looks like in practice: the places where the workflow is miraculous, the places where it is agonizing, and the hour-long arguments between Freerouting and pcb-rnd about whether two pieces of copper were three tenths of a millimeter apart.

If you came here looking for a triumphant "I built a PCB with AI and it just worked" story, this is not that post. If you came here looking for a nuanced report from someone who spent days on this and now has calibrated opinions about where it's worth doing, welcome.

The Setup, Briefly

For readers who haven't read the earlier posts: the GigaShield is a 155mm x 90mm PCB that sits between an Arduino Giga R1 (3.3V logic) and a RetroShield Z80 (5V logic). It has ten SN74LVC8T245PW level-shifter ICs translating 72 channels between the two voltage domains. The Giga plugs into the bottom (3.3V headers), the RetroShield plugs into the top (5V headers), and the shifters bridge the two. Our first design used TXB0108 level-shifter ICs but the autosensing signal direction did not play nicely with the Z80's tri-state signals.

The entire PCB is generated by a Python script. Not the schematic — there is no schematic. Not the layout — there is no graphical layout. The script emits a pcb-rnd board file directly: component placements, pad definitions, netlist, board outline, silkscreen text, all in text format. Running python3 build_giga_shield.py produces giga_shield.pcb in a fraction of a second. The board is then routed by Freerouting and exported to Gerbers via pcb-rnd's command-line tools.

Everything happens in the terminal. No GUIs, no mouse clicks, no "did I save the layout?" anxiety.

The v0.3 board — subject of the previous post — failed because of a subtle bug in this pipeline. The v0.4 board fixes that bug and several others I didn't know about. This post covers what it took to find and fix them.

Bug Class #1: The Kind AI Caught Easily

Before refabricating, I asked Claude Code to audit the Python generator for any additional bugs. I framed it as a careful code review with specific checkpoints: pin numbering in both the pcb-rnd and KiCad generators, SN74LVC8T245 pin mapping against the TI datasheet, critical Z80 signal routing, and component placement validation.

The audit took maybe fifteen minutes of wall-clock time — during which Claude Code read the entire codebase, cross-referenced it against the datasheet, and produced a severity-graded report. Most findings were cosmetic (stale comments, a print statement that said "1x10 header" for an 11-pin connector, harmless inconsistencies between the two build scripts). One finding was genuinely important:

H1 — Comment "default A→B" is factually wrong, and U10's pulldown produces incorrect default behavior without user intervention.

DIR=LOW per the SN74LVC8T245 datasheet means B→A (B-side drives A-side). The comment states the opposite. More importantly, U10 uses DIR for CLK, RESET, INT, NMI — signals that must flow A→B (Giga→Z80). With R10 pulling DIR_U10 to GND, U10 powers up in the B→A direction, which means the 5V Z80 side would drive the 3.3V Giga side on these control pins. Backwards and potentially damaging if the Z80 is outputting signals while the Giga is also driving those pins.

This bug is worth dwelling on, because it highlights something important about how I've been working on this project. I didn't write that code. Claude Code did, three weeks earlier, as part of the initial generator script. The wrong comment and the wrong pulldown direction were both introduced by the same LLM that later caught the mistake on audit.

That sounds like it should be embarrassing for the workflow — the tool that generates bugs is also the tool that reviews them — but I think it's actually the right shape of the argument. A single LLM pass is fallible. A single human pass is also fallible. What matters is whether the combined system catches bugs reliably before they ship. In this case, the first pass (generation) introduced a subtle error, the second pass (audit, explicitly framed as a datasheet-cross-referencing review) caught it. The human's job was to know that the second pass was worth requesting — and to recognize, when the audit report came back, which findings mattered and which were cosmetic.

This is exactly the kind of bug that a human engineer might catch on a good day and miss on a bad day. The comment and the circuit were internally consistent but both wrong. The datasheet was authoritative but three hundred pages away. Claude Code's audit didn't find it by being clever — it found it by mechanically cross-referencing every pin mapping against the datasheet, every comment against the actual behavior, with the patience of a machine that doesn't get bored.

The fix was a one-line change: swap R10 from a pulldown-to-GND to a pullup-to-+3V3. U10 now powers up in the correct direction without any firmware intervention.

I'll be transparent: the wrong direction would not have destroyed the boards immediately. The 5V CMOS outputs driving a 3.3V CMOS input is within the chips' absolute maximum ratings for short periods, especially if the Giga's pins are configured as inputs during that window. But in the steady state, with the Arduino trying to drive those pins as outputs, you'd have a lot of current flowing through ESD diodes, probably latch-up, almost certainly failure. A bug that would have cooked several \$70 Arduino Giga R1s over the life of the project.

Fifteen minutes of LLM review. One avoided burn-up. A comfortable argument for the workflow.

Bug Class #2: The Kind AI Didn't Help With At All

Then came the Freerouting clearance saga.

After the v0.4 build script was fixed, regenerated, and re-routed with Freerouting at the familiar 0.254mm trace width and 0.254mm clearance rule, I packaged up the Gerbers and uploaded them to PCBWay. Their automated DFM check failed almost immediately with a message I'd never seen before:

Failed reason: The spacing between copper traces and pads should be larger than 0.1mm.

A few hours later, a QA engineer at PCBWay followed up with a screenshot from their internal Gerber inspection tool. They had loaded the top-copper layer, zoomed into the crowded region around the level-shifter ICs, and drawn yellow arrows at eight or ten spots where traces were running uncomfortably close to pads. At one spot they'd highlighted in cyan, their measurement tool showed D=0 — literal zero distance between a trace and a pad.

The screenshot PCBWay's QA engineer sent back. Yellow arrows mark the violations. Cyan highlight shows their measurement tool reporting D=0 — a trace touching a pad that it shouldn't be touching.

This should not have been possible. Freerouting had been told to maintain 0.254mm clearance between all copper. 0.254mm is 10 mil, which is more than double PCBWay's 0.1mm minimum. If the router respected its rules, PCBWay's automated check should have sailed through.

But it hadn't.

I went through the usual debugging motions. Ran pcb-rnd's DRC on the routed board: 121 clearance violations. Opened the board in a gerber viewer: confirmed the violations were real, not artifacts. Tightened Freerouting's clearance to 0.3mm, narrowed traces to 0.2mm to give more room, added explicit clearance rules for every object-pair type (wire-pin, wire-via, smd-pin, pin-pin, and so on):

(rule
  (width 0.2)
  (clearance 0.3)
  (clearance 0.3 (type smd_smd))
  (clearance 0.3 (type smd_via))
  (clearance 0.3 (type smd_pin))
  (clearance 0.3 (type pin_pin))
  (clearance 0.3 (type pin_via))
  (clearance 0.3 (type via_via))
  (clearance 0.3 (type wire_wire))
  (clearance 0.3 (type wire_via))
  (clearance 0.3 (type wire_pin))
  (clearance 0.3 (type wire_smd))
)

Re-ran Freerouting. Ninety minutes of autorouting and optimization later, pcb-rnd's DRC still reported 121 clearance violations. Identical count. As if the clearance setting had been ignored entirely.

This went on for several iterations. I tried 0.35mm clearance. 0.4mm clearance. I made the trace narrower. I made it wider. I added explicit rules for each layer. The violation count remained 121. Freerouting's optimization phase consistently took ninety minutes and improved the design by "about 52%" each time. The DRC report was unmoved.

Claude Code was with me through all of this. It read the Freerouting log. It parsed the DRC output. It spotted the p_shape is not bounded warnings (only three occurrences, probably unrelated). It suggested hypotheses — maybe Freerouting measured clearance from via hole centers rather than copper rings, maybe the padstack path shapes were confusing the bounds calculation, maybe pcb-rnd's DRC was using a threshold different from what Freerouting was told. Each hypothesis was plausible. None of them panned out. The violation count stayed at 121.

The breakthrough came not from any clever insight but from a dumb experiment: run the DRC on the unrouted board. Zero violations. Run it after Freerouting: 121 violations. Same 121, every time. Re-route with completely different settings: still 121 violations at roughly the same coordinates.

The coordinates were the tell. The 121 "violations" weren't scattered — they were clustered at specific, consistent locations. And those locations, when I finally examined them carefully, were pin-to-trace junctions. Where a trace ended at a pad on its own net. Where the copper legitimately overlapped because they were the same electrical node.

pcb-rnd's DRC was flagging every legitimate trace-pad connection as a "shorted nets: net too close to other net" violation. Not because the nets were shorted, but because pcb-rnd's DRC algorithm didn't properly account for the fact that two pieces of copper on the same net are supposed to touch. They're supposed to be connected. That's the whole point.

All 121 "violations" were false positives. Freerouting had been maintaining 0.3mm clearance the whole time. pcb-rnd had been lying about it the whole time. And PCBWay's automated DFM had been flagging the real issue — spots where same-net connections appeared to violate clearance under their algorithm — except their tool was smart enough to usually recognize same-net connections, so it flagged only the ones where the overlap happened to look particularly bad in the Gerber rendering.

The fix, once I understood this, was to increase clearance (which I'd been doing) until the visual overlap was conservative enough that PCBWay's tool stopped complaining, and stop trusting pcb-rnd's DRC entirely. The winning configuration was 0.254mm traces with 0.3mm clearance and the explicit per-type clearance rules. PCBWay's DFM passed on that submission.

Total time spent on this: maybe six hours across two days, counting the autorouting time, the iteration loops, and the eventual diagnosis. Nothing Claude Code did made this faster. Claude Code could parse logs and suggest hypotheses as quickly as I could read them, but it couldn't see what I couldn't see. The problem wasn't in any file — it was in the interaction between three tools' different assumptions about what "clearance" means. That kind of bug lives in the interfaces, not in any single artifact. LLMs are not good at debugging interfaces they can't run.

I don't think this is a damning critique of AI-assisted workflows. But it is a calibrating one. If you're choosing between "write a Python script and fight Freerouting and pcb-rnd and PCBWay's DFM" versus "click around in KiCad for four hours," the second path has fewer interfaces to break. Graphical tools eat their own complexity internally. The CLI workflow exposes every seam.

Bug Class #3: The Kind AI Made Worse

At some point during the clearance debugging, I decided to reduce the board from 6 layers to 4 layers. Freerouting had been routing successfully on four, so paying for six was wasteful. I updated the pcb-rnd build script to emit a 4-layer stack, changed the Groups() directive to "1,c:2:3:4,s:5:6:7", regenerated, exported DSN, routed, imported, exported Gerbers.

The resulting Gerber package had five copper layers.

Not six. Not four. Five. With traces scattered across them in a way that suggested pcb-rnd, on SaveTo, had rewritten my Groups() string into something of its own invention — "6:8:1,c:2:3:5:10:11:4,s:9:7" — that created phantom layers and assigned them roles my build script hadn't intended. The gerber named intern.copper.none.12.gbr had 1,406 traces. This was not a layer I had defined.

I asked Claude Code to help me understand pcb-rnd's Groups() syntax. It gamely tried to parse the string. It offered three different interpretations, each of which would have produced a different layer stack than what pcb-rnd actually did. It couldn't fix the problem because it couldn't run pcb-rnd and observe the behavior. I couldn't fix it either, for the same reason with more forgivable excuses.

After maybe two hours of going in circles, I reverted to the original 6-layer Groups() string, re-routed, exported Gerbers (getting six copper gerbers — four with traces, two with only padstack pads), and deleted the two empty ones before zipping. PCBWay doesn't care what my source PCB file says. They fabricate what's in the Gerber bundle. If I submit four copper layers, I get a 4-layer board.

This is a pattern worth noting. Sometimes the right solution to a tool-chain problem isn't to fix the tool chain. It's to work around it. Claude Code is good at helping you solve problems correctly; it's not as good at helping you recognize that the correct solution is to stop trying to solve the problem. That's a uniquely human skill: knowing when to quit.

The Component Placement Lesson

One more story worth telling. The v0.3 design had the ten level-shifter ICs arranged as U1-U5 across the top of the board and U6-U10 stacked in a single vertical column on the right side. It looked tidy. It routed successfully on six layers. It also concentrated an enormous amount of signal traffic into a narrow vertical channel between the shifter cluster and the 2x18 headers — so narrow that Freerouting couldn't route all 308 nets on four layers without leaving some unrouted.

Claude Code flagged this the first time I tried the 4-layer route. Not by proposing a new placement — it didn't. By noting that the unrouted nets all terminated in the same congested region, and asking whether I'd considered spreading U6 through U10 into a staggered two-column layout between J9 and J10. I hadn't. The suggestion was obvious in hindsight, which is the clearest sign that it was useful. A lot of engineering is exactly this: you know the answer once someone mentions it, but no one mentions it and you don't think to ask yourself.

The new placement — U6, U8, U9 in one column closer to J9, U7 and U10 offset in another column closer to J10 — routed cleanly on four layers with 314/314 nets connected. The LLM didn't solve a geometric problem I couldn't solve. It pointed at a geometric problem I hadn't recognized as a problem. That's a different and narrower contribution than "AI designed my board," but it's also real and repeatable.

For the curious: my mental model for why the original layout was bad had been "I want them in a tidy column near the connector." The LLM's implicit mental model, derived from having read every PCB design textbook ever digitized, was "signal traffic wants to spread, not concentrate." Both are defensible. The second is more useful.

What Actually Works

Stepping back from the debugging, here is my honest breakdown of where Claude Code earned its keep on this project versus where it didn't.

Where it was essentially indispensable:

Parsing hierarchical S-expression formats. The original KiCad design from the Fiverr engineer was a .kicad_sch file with nested sheets, positional net labels, and implicit connections across hierarchical boundaries. Extracting the 72-channel signal mapping from that file would have taken me half a day by hand and maybe an hour of Python if I were patient. Claude Code did it in about twenty minutes of interactive conversation.
Generating boilerplate code with domain-specific constraints. The tssop24_element() function that produces the SN74LVC8T245PW footprint is three dozen lines of boring arithmetic — pad positions, coordinate transforms, string formatting. I asked for a pcb-rnd Element that matched the SN74LVC8T245PW footprint and got working code on the first try — Claude Code fetched the TI datasheet, read the pcb-rnd format reference, and produced the correct geometry without me having to hand it either document. I would have gotten it wrong on the first try if I'd written it by hand, because I would have flipped the B-side pin order (pins 13-24 run bottom-to-top on this package, which is easy to miss).
Cross-referencing mechanical information against specifications. The audit I mentioned earlier — reading my entire codebase and checking it against the TI datasheet — would have been tedious and error-prone to do manually. Claude Code did it reliably and caught a bug I would have shipped.
File format debugging. pcb-rnd's error messages are unhelpful by the standards of modern software. The Specctra DSN format has undocumented quirks. Gerber apertures have version-specific formatting differences. Every time I hit an error I didn't understand, Claude Code could read the file, diff it against a working example, and tell me what was different. This is LLM work at its best — bulk pattern-matching across reference material I don't have memorized.
Bridging the SSH workflow. pcb-rnd runs on Linux and I work on a Mac. Half of the pipeline was shell commands to upload files, run pcb-rnd on a remote machine, download results, and iterate. Claude Code managed that SSH shuffle cleanly across hundreds of invocations. Not a hard problem, but a tedious one. Delegating it was valuable.

Where it didn't help:

Diagnosing cross-tool interaction bugs. The 121-false-positive clearance saga was a problem that lived between three tools' different mental models. Claude Code could read each tool's output but couldn't run experiments to validate hypotheses. I had to do that myself, slowly.
Making placement decisions involving physical intuition. Claude Code helped with the U6-U10 restacking only after I'd ruled out other options. It didn't lead the design choice. For pure physical intuition — "this will be too dense," "this trace will pick up noise from this inductor," "this via is a mechanical weakness" — the LLM was helpful as a second opinion but not a primary source.
Knowing when to stop debugging and accept a workaround. When pcb-rnd's SaveTo mangled my layer stack, Claude Code happily kept trying to fix it. It took me longer than it should have to realize the fix was to not use SaveTo. LLMs are optimized for "helpfully continue the task." Human judgment is needed for "recognize the task is wrong."

The Economics

It's fair to ask whether any of this is worth it. I'll try to be honest.

Time invested in the v0.4 respin, from identifying the pin-numbering bug through submitting the second Gerber package:

Designing the fix: 2 hours (including the code audit that caught R10)
Regenerating and re-routing: ~4 hours of wall-clock time, mostly Freerouting optimization
The clearance debugging saga: ~6 hours
The layer-stack misadventure: ~2 hours
Documentation, git commits, this blog post: ~4 hours

Total: ~18 hours of my engineering time, plus a lot of Freerouting CPU-hours.

If I had paid a Fiverr freelancer to redo the design in KiCad from scratch, it would have cost about \$400 and taken about a week of wall-clock time. If I had learned KiCad properly and done the layout myself, it would have taken probably 20 to 30 hours for a 314-net 4-layer board, given I've never used KiCad's layout editor seriously. If I had used a commercial autorouter (Altium's, say), I would have paid thousands of dollars in licensing and probably spent 10 hours on the project.

The Python-and-Claude-Code workflow cost me 18 hours of engineering time and the frustration of debugging interactions between three different CLI tools. PCBWay sponsored the fabrication, so the direct cash outlay was zero on this project — but that's an artifact of the sponsorship, not the workflow. A non-sponsored hobbyist running the same pipeline would spend roughly \$50 per prototype fabrication run at PCBWay's standard rates, so two runs (v0.3 plus v0.4) would be in the ballpark of \$100 total. The boards will arrive next week.

Is that a good trade? It depends on what you value. For me, the workflow is repeatable — the next board I design will take a fraction of this effort because the tool chain is now debugged. For someone doing one PCB in their lifetime, this would be a terrible trade. For someone who plans to iterate on a design family over months or years, it's probably a good one. The upfront cost amortizes.

I don't think the right question is "AI-assisted PCB workflow yes or no." I think it's "what kind of PCB work are you doing, and does your workflow compound?" If your boards are one-offs, use KiCad. If your boards are a family that evolves over time, scripted workflows start paying for themselves around the third iteration. Add AI assistance on top and the break-even point moves earlier.

What I'd Do Differently

A few hard-won principles from this project, in case anyone tries something similar:

Validate the pipeline on a dead-simple board before using it for a real one. If I had generated a toy two-net board first, routed it, exported Gerbers, and submitted the files to PCBWay's online DFM check as a dry run, I would have caught the pin-numbering convention bug without wasting a fabrication run on v0.3. The total time for the dry run would have been maybe two hours. It would have saved a two-week turnaround — and, more importantly, not burned a sponsored fabrication run on a design that was never going to work.

Run DRC with multiple tools and compare. pcb-rnd's DRC was actively misleading on this board. If I had also run the Gerbers through a third-party tool like Gerber Viewer or through KiCad's DRC after importing, I'd have noticed the disagreement and dug into it earlier. Never trust a single tool's DRC as authoritative.

Keep PCBWay's DFM as the source of truth. Their automated check has seen more boards than any open-source DRC. It's tuned for actual manufacturability. When the open-source tools say "clean" and PCBWay says "fail," believe PCBWay.

Write test fixtures for the scripted generator. A few unit tests that verify "net X has pins Y and Z" would have caught the pin-numbering bug in the build step rather than the fabrication step. For a scripted PCB workflow, the Python code needs the same test discipline as any other production code.

Budget for LLM's failure modes. The LLM is fast and confident but can spiral into unproductive debugging loops. When a fix doesn't work on the second or third try, that's the signal to stop and think rather than let the LLM keep trying variations. Six hours on the clearance bug should have been two.

The Broader Question

There's a cultural current in software circles right now that frames AI coding assistants as either revolutionary or fraudulent. Neither frame captures what I experienced on this project.

Claude Code didn't replace my expertise. I still had to know what a level shifter is, why the Z80 tri-states its bus during IO cycles, why the annular ring on a via matters for fab yield, when a pull-up is safer than a pull-down. Without that domain knowledge, I couldn't have directed Claude Code at the right problems, and I couldn't have recognized when its suggestions were wrong.

Claude Code also didn't slow me down. The audit that caught the R10 bug was pure leverage. The file-format debugging was pure leverage. The SSH shuffle was pure leverage. The CLI workflow I'm using would not be tractable without an LLM assistant — too many file formats, too many tools, too much boilerplate. Claude Code didn't enable the workflow, but it made it something I could actually use instead of abandoning it for KiCad's GUI after the first pcb-rnd error message.

What Claude Code is, for PCB design, is a competent junior collaborator with encyclopedic memory, infinite patience, and no physical intuition. It won't design your board for you. It'll help you design your board, if you know what you're doing. That's a different and less exciting claim than the marketing suggests, but it's also a more durable one.

What's Next

The v0.4 boards are scheduled to arrive from PCBWay in a few weeks. If they work — if the Z80 actually responds to its clock, if the data bus reads are clean, if /IORQ is no longer stuck at ground — I'll write a Part 4 covering the bring-up and the test results. If they don't work, I'll write a Part 4 about whatever new bug I've introduced.

In the meantime, the Python build script, pcb-rnd source files, Gerber outputs, Arduino test sketch, and every piece of infrastructure discussed in these posts is open source:

giga-shield — Complete design files, build pipeline, and test firmware

If you're interested in scripted PCB design workflows, I'd genuinely like to hear from people who've tried similar approaches — or, more interestingly, tried and given up. The body of public literature on "I attempted this and it didn't work for me" is much smaller than on "I succeeded, here's how," and I think the former is more useful.

*Previous posts in this series: Redesigning with Claude Code (Part 1) · How a Pin Numbering Bug Killed a PCB · Fiverr PCB Design (\$468)

How a Pin Numbering Bug Killed a PCB

A.C. Jokela — Sat, 18 Apr 2026 15:00:00 GMT

The boards arrived from PCBWay in perfect condition. Black solder mask, clean silkscreen, precise drill hits. The fabrication quality was excellent — 6-layer board, 6/6 mil trace/space, HASL finish, delivered in about two weeks from order to doorstep. PCBWay's online Gerber viewer had flagged one component overlap before manufacturing (a decoupling capacitor crowding a level shifter IC), their team asked about it, and we resolved it in a single email exchange. Everything about the fabrication was smooth.

Then I plugged in the RetroShield Z80, uploaded a test sketch to the Arduino Giga R1, and nothing worked.

Not "mostly worked with some issues." Nothing. The Z80 didn't respond to its clock. The data bus read all zeros. One control signal — /IORQ — was stuck permanently LOW while the others sat HIGH. Sixteen bus cycles of silence where there should have been a Z80 booting up and fetching instructions from address 0x0000.

The board wasn't defective. PCBWay had fabricated exactly what I asked them to fabricate. The problem was that what I asked them to fabricate was wrong.

The GigaShield v0.3

For context: the GigaShield is a level-shifter shield that sits between an Arduino Giga R1 (3.3V logic) and a RetroShield Z80 (5V logic). It has ten SN74LVC8T245PW 8-channel bidirectional level translators, translating 72 signal channels between the two voltage domains. The design was covered in detail in Part 1 of this series — the short version is that the entire PCB was generated programmatically from a Python script, exported to pcb-rnd format, autorouted with Quilter.ai, and sent to PCBWay as Gerber files.

The board has twelve connectors. Ten are single-row pin headers (1xN) for the standard Arduino shield headers, the analog breakout, and a direction-control header. Two are dual-row headers (2x18) that carry the high-pin-count digital signals — Arduino digital pins 22 through 53 on one side, level-shifted to 5V on the other.

Those two dual-row headers are where the bug lives.

First Contact

The first test was simple: read the Z80's control lines with no clock running. The SN74LVC8T245 level shifters have explicit direction control (that's why I chose them over the TXB0108), so with the direction defaulting to B-to-A (5V side drives, Arduino reads), I should see the Z80's active-low control outputs all sitting HIGH — their idle state.

=== Test 3: Control inputs (active-low Z80 signals) ===
  /M1=1  /RD=1  /WR=1  /MREQ=1  /IORQ=0
  With no Z80 clock, all should read HIGH (inactive)

Four out of five correct. /IORQ reading LOW was the first clue that something was wrong, but I initially dismissed it as a possible floating pin or a pull-down issue on the RetroShield side.

The real alarm came when I tried to boot the Z80. The test sketch drives CLK, releases RESET, and captures the first 16 bus cycles:

=== Test 5: Z80 boot — first 16 bus cycles ===
  RESET released
  T00: /M1=1 /RD=1 /MREQ=1 DATA=0x00
  T01: /M1=1 /RD=1 /MREQ=1 DATA=0x00
  T02: /M1=1 /RD=1 /MREQ=1 DATA=0x00
  ...
  T15: /M1=1 /RD=1 /MREQ=1 DATA=0x00
  RESET asserted — Z80 halted

Sixteen clock cycles, and the Z80 never responded. No /M1 going low for an opcode fetch. No /MREQ for a memory request. No /RD for a read cycle. The Z80 was either not getting a clock signal, or not getting a valid RESET sequence, or both.

I checked the obvious things first. Is the 3.3V rail powered? Yes. Is the 5V rail powered? Yes. Is the direction control for U10 (the control-output shifter carrying CLK) set correctly? Yes — J11 pin 10 tied to 3.3V, which sets DIR HIGH for A-to-B (Giga drives Z80). Is the RetroShield seated properly? Yes.

Looking at the Schematic

With solder bridges ruled out and the electrical fundamentals verified, I went back to the Python build script — the single source of truth for the entire PCB design.

The board's ten single-row headers all worked. The level shifters were passing signals correctly (the control input test proved that U9 was translating). The problem was specific to the Z80 not receiving CLK and RESET, and /IORQ being stuck at ground.

All three of those signals route through the 2x18 dual-row headers: J9 (3.3V side) and J10 (5V side). I started tracing nets.

The build script generates pin headers with a function called pin_header_element. For a 2x18 header, it iterates:

for col in range(ncols):      # 0, then 1
    for row in range(rows):   # 0 through 17
        pin_num += 1

This produces column-first numbering: pins 1–18 run down column 0, pins 19–36 run down column 1.

Col 0         Col 1
Pin 1         Pin 19
Pin 2         Pin 20
Pin 3         Pin 21
...           ...
Pin 18        Pin 36

But the net arrays that assign signals to pin numbers were written in zigzag order — the standard convention for dual-row pin headers, where pin numbers alternate between columns:

Col 0         Col 1
Pin 1         Pin 2
Pin 3         Pin 4
Pin 5         Pin 6
...           ...
Pin 35        Pin 36

The J9 net array:

j9 = ['+5V', '+5V',
      'D22', 'D23', 'D24', 'D25', ...
      'D52', 'D53',
      'GND', 'GND']

This assumes zigzag: pin 3 = D22 at (col 0, row 1), pin 4 = D23 at (col 1, row 1). But the footprint generator puts pin 3 at (col 0, row 2) and pin 4 at (col 0, row 3) — same column, two rows apart instead of across from each other.

Every signal on J9 and J10 was at the wrong physical position.

The Geometry of the Bug

To understand why this specific mismatch is catastrophic, consider what happens to a few critical signals.

D52 (CLK): In the net array, D52 is at index 32 (pin 33). In zigzag, pin 33 is at (col 0, row 16) — left column, second-to-last row. In column-first, pin 33 is at (col 1, row 14) — right column, six rows higher. The RetroShield's CLK pin physically touches the pad at zigzag position (col 0, row 16), but the GigaShield routed CLK to column-first position (col 1, row 14). Different column, different row. The Z80 never sees a clock.

D53 (/IORQ): Pin 34 in zigzag is at (col 1, row 16). In column-first, pin 34 is at (col 1, row 15) — one row off. But at the zigzag (col 1, row 16) position, the column-first numbering places pin 35, which the net array assigns to GND. The RetroShield's /IORQ pin is physically sitting on a ground pad. That's why it reads LOW — it's hard-wired to ground through the PCB trace.

D38 (RESET): Pin 19 in zigzag is at (col 1, row 9). In column-first, pin 19 is at (col 1, row 0) — the very first row of the second column instead of the middle. RESET goes to a completely unrelated position.

The pattern holds for every signal on the 36-pin header. The first two pins (+5V, +5V) happen to be at matching positions for both conventions (pin 1 is always col 0 row 0, and pin 2's mismatch doesn't matter since both are power). After that, every signal diverges.

Why Nothing Caught It

This bug is invisible to every standard verification step in the PCB pipeline.

DRC (Design Rule Check): All traces meet clearance and width rules. The traces connect the correct logical pin numbers — the netlist is internally consistent. DRC validates geometry, not pin-numbering conventions.

Visual inspection: The board renders look correct. Traces run from shifter pads to header pads in clean, routed paths. You can't tell from a PNG rendering that a header pad at row 14 should be at row 16. The footprint silkscreen shows pin 1, and the rest are just a grid of identical-looking through-holes.

Quilter.ai: The autorouter takes a netlist and routes traces between named pads. It has no concept of "this pad should be at this physical position" — it just connects pad A to pad B using copper. If the pad positions are wrong in the input file, the router dutifully routes to the wrong positions.

Gerber review: PCBWay's Gerber viewer (and any standard Gerber viewer) shows copper layers, drill hits, and silkscreen. It doesn't cross-reference pad positions against any external standard. The Gerbers were valid files describing a valid board — just not the board I intended.

pcb-rnd: The PCB editor displays the board as defined. It doesn't know that a 2x18 header should use zigzag numbering. It renders what the file says.

The bug exists in the gap between two conventions: the net array assumes zigzag ordering (which is the industry standard for dual-row headers and what KiCad uses in its standard footprint library), while the footprint generator implements column-first ordering (a natural but incorrect choice when iterating for col... for row...). Both halves are internally consistent. The error is in their interaction.

The One-Line Fix

The fix is almost comically small relative to the damage:

# Before (column-first — WRONG for standard dual-row headers):
for col in range(ncols):
    for row in range(rows):
        pin_num += 1

# After (zigzag — correct):
for row in range(rows):
    for col in range(ncols):
        pin_num += 1

Swap the loop order. That's it. Row-first iteration produces zigzag numbering: pin 1 at (col 0, row 0), pin 2 at (col 1, row 0), pin 3 at (col 0, row 1), pin 4 at (col 1, row 1), and so on. This matches the KiCad convention, the IPC convention, and what every dual-row connector in the world expects.

Single-row headers (J1 through J8, J11) are unaffected because column-first and zigzag are identical when there's only one column. Only J9 and J10 — the two 2x18 headers — had the wrong pinout.

Can Software Fix It?

My first instinct was to work around the bug in firmware — remap which Arduino GPIO pins the sketch uses so that signals arrive at the correct physical positions despite the wrong traces. If the board routes D36 to where D52 should be, just use D36 for CLK in the sketch.

It doesn't work, for two reasons.

First, some signals map to power pins. D53 (/IORQ) physically sits on a GND pad. You can't drive a signal through a ground trace in software. The pad is connected to the ground plane. It's not a GPIO — it's copper bonded to zero volts.

Second, the level shifters have shared direction control. Each SN74LVC8T245 has eight channels and one DIR pin. All eight channels shift in the same direction. If you remap CLK (which needs Giga-to-Z80 direction) to go through a shifter that also carries address bus signals (which need Z80-to-Giga direction), you can't set both directions simultaneously. The shared DIR creates an unsolvable constraint when signals that need opposite directions land on the same shifter.

The board needs a respin.

The Respin

With the bug identified and the fix trivial, the path forward is straightforward:

Fix the loop order in pin_header_element
Regenerate giga_shield.pcb from the Python script
Export to DSN format via pcb-rnd
Re-route the traces
Export new Gerbers
Send to PCBWay for fabrication

The entire pipeline — from fix to fabrication-ready Gerbers — takes about twenty minutes. That's the advantage of the text-based, scriptable workflow described in Part 1. Change one line of Python, re-run the pipeline, get a new board. No GUI interactions, no manual routing, no "did I remember to update the footprint" anxiety.

The v0.3 board was originally routed on six layers because the autorouter couldn't find paths for every net with the components packed tightly together. For v0.4, we rearranged the level shifter ICs into a staggered two-column layout between the dual-row headers, giving the router more room to work with. The result: all 313 nets routed cleanly on just four layers. We're also using Freerouting for the v0.4 routing — we had initially planned to use Quilter.ai, but their recent release introduced some parsing issues that made it unreliable for our KiCad files. Freerouting's v1.9 codepath, while older, has been rock-solid for this board.

PCBWay's turnaround on prototype boards is fast — I've consistently gotten boards in under two weeks from order placement to delivery, including the original v0.1 boards from the Fiverr design. For the v0.4 respin, I'm using slighly different specs: 4-layer, 1.6mm FR-4, black solder mask, HASL finish, standard 6/6 mil trace/space. PCBWay's pricing for prototype quantities (5-10 boards) is genuinely hard to beat — and the quality has been consistently good across every order.

One thing I appreciate about PCBWay's process: the pre-production review. Before they start cutting boards, their engineering team reviews the Gerbers and flags potential issues. They caught the C29/U10 overlap on the v0.3 boards — a decoupling capacitor footprint that crowded a TSSOP-24 IC. We agreed to leave C29 unpopulated (it was one of 29 bypass caps, not critical), and PCBWay proceeded with fabrication. That kind of proactive communication saves real time and money. If I'd caught the pin numbering bug at that stage, the whole issue would have been avoided. But pin numbering convention mismatches aren't the kind of thing that shows up in a Gerber review — the files were technically correct.

What PCBWay Offers

For readers who haven't used PCBWay before, a brief overview of what they provide beyond basic PCB fabrication:

PCB Prototyping: 1 to 8 layers, multiple surface finishes (HASL, ENIG, OSP, immersion silver/tin), controlled impedance, blind/buried vias, flex and rigid-flex boards. Minimum trace/space of 3.5/3.5 mil for standard process. They handle both small prototype runs (5 boards) and production quantities.

PCB Assembly (PCBA): Full turnkey assembly with component sourcing, SMT and through-hole placement, and testing. For a board like the GigaShield with thirty-six SMD components (ten TSSOP-24 ICs, twenty-seven 0603 caps, nine 0603 resistors), assembly service eliminates the most tedious part of the build. TSSOP-24 packages are hand-solderable with a fine-tip iron and flux, but doing ten of them with twenty-four 0.65mm-pitch pins each is several hours of careful work. PCBWay's pick-and-place machines do it in minutes.

3D Printing and CNC Machining: Useful for enclosures, mounting brackets, and custom mechanical parts. Multiple materials available — PLA, resin, nylon, aluminum, steel. I haven't used these services for this project, but for projects that need a custom case or mounting hardware, having it from the same vendor simplifies ordering.

Stencil Service: Solder paste stencils for reflow soldering. If you're doing your own assembly with a hot plate or reflow oven, a properly cut stencil makes paste application dramatically faster and more consistent than syringe application.

Design-for-Manufacturing (DFM) Review: As mentioned above, PCBWay reviews your files before production and flags potential issues. This caught the C29 overlap on my boards. For someone iterating on a design — especially a design generated programmatically where visual review of the physical layout is less intuitive — this review is valuable.

The pricing model scales well: prototype quantities are cheap enough to iterate without stress (important when you're, say, debugging a pin numbering convention), and production quantities get volume discounts. The online quoting system gives you a price instantly when you upload Gerbers, so you know the cost before committing.

Lessons

Every post-mortem needs a "what did we learn" section. Here's mine.

Test with hardware before ordering quantity. If I'd breadboarded the 2x18 connection with jumper wires before committing to fabrication, I'd have caught the mismatch immediately. The single-row headers all work — I could have validated those and assumed the dual-row headers were fine. Testing the full signal path end-to-end, from Giga GPIO through the level shifter to the RetroShield's Z80, would have caught it in an hour. One of the reasons I did not breadbroad the design first is I was unable to find breadboardable SN74LVC8T245PW level shifters. I have TXB0104 Bi-Directional Level Shifters but no driven level shifter breakouts.

Convention mismatches are the hardest bugs. The code was correct by its own logic. The net arrays were correct by the KiCad convention. The footprint was correct by its own convention. The bug was in the assumption that both sides used the same convention. No single piece of code was wrong — the error was in the interface between two correct pieces. This is the class of bug that code review, static analysis, and automated testing all miss, because each component passes its own tests.

Text-based PCB design cuts both ways. The scriptable pipeline that let me generate and route a board in twenty minutes also let me ship a subtle pin-numbering bug to fabrication in twenty minutes. A graphical PCB editor would have forced me to visually place the header footprint and see the pin numbers on screen, which might have triggered a "wait, that doesn't look right" moment. The speed of automation is a liability when the automation is wrong. The counterargument is that graphical editors have their own class of invisible bugs — accidentally moved components, stray traces from mis-clicks, forgotten net connections. Text-based design doesn't eliminate bugs; it changes which bugs are likely.

Pin numbering standards exist for a reason. The IPC standard for dual-row connector numbering is zigzag. KiCad follows it. Every 2xN header footprint in every major footprint library follows it. When you write your own footprint generator, you need to follow it too. The column-first iteration (for col... for row...) is a natural coding pattern — it's how you'd iterate a 2D array in most languages. It's also wrong for connector pin numbering. Convention over intuition.

The fabrication was perfect. I want to emphasize this because it's easy to conflate "the board doesn't work" with "the board was made badly." PCBWay manufactured exactly what the Gerber files specified, with excellent quality. Every trace, via, drill hit, and solder mask opening matched the design files. The bug was in my design files, not their manufacturing process. The distinction matters: when a board comes back dead, the first question should be "is my design correct?" not "did the fab house make an error?"

The Fix in Context

This is the second failure mode for this project, and both have been instructive. The v0.1 board failed because the TXB0108 auto-sensing level shifters couldn't handle Z80 tri-state bus conditions — a component selection problem. The v0.3 board failed because of a pin numbering convention mismatch in the software that generates the PCB — a toolchain problem. Neither was a manufacturing defect. Both were design errors that passed every automated check and only surfaced when physical hardware was connected.

The v0.4 respin will fix the pin numbering and go back to PCBWay for fabrication. The turnaround time from fix to new boards is probably ten days — twenty minutes for the software pipeline, a few days for PCBWay's production, and a few days for shipping. In the meantime, the v0.3 boards are useful as physical references for component placement and as evidence that the level shifters themselves work correctly (the single-row header signals all translate properly through the SN74LVC8T245s).

The Python build script, pcb-rnd source files, Gerber outputs, and the test sketch are all open source:

giga-shield — Complete design files, build pipeline, and test firmware

What's Next

Part 2 of this series was supposed to cover assembled boards and Z80 bus captures. It will — just with v0.4 boards instead of v0.3. In the meantime, the v0.4 Gerbers are being generated and will be sent to PCBWay for the respin. The fix is one line. The lesson was worth more.

The Thing and the Endpoint: Why a Z80 Gathers a World and an API Doesn't

A.C. Jokela — Thu, 16 Apr 2026 13:00:00 GMT

A Z80 DIP-40 weighs 5.7 grams. Run Zork on it, or run Zork in a browser emulator. The bytes execute the same way. One of these is a thing. The other isn't.

That distinction has a name. Heidegger called it Das Ding, the thing. He meant it in a specific sense that has nothing to do with how we normally use the word. A thing, for him, is something that gathers a world. A wine jug gathers earth (the clay, the grape), sky (the rain that watered the vines, the sun that ripened them), the mortals who drink from it and made it, and the occasion of its use. The jug is not a container that happens to have history. The gathering is the jug's being a jug.

That sounds mystical on first read. On second read it describes something you already know. A Z80 RetroShield running CP/M and Zork at 2 a.m. on a workbench gathers a world in this specific sense. A request to an OpenAI endpoint does not, and cannot, and was deliberately designed not to. This essay is about why that difference matters, and why the people who build home labs and retro computing setups feel it even when they can't name it.

What the RetroShield Gathers

Start with the chip. The Z80 on my bench was fabricated by Zilog sometime in the late 1990s, which I know because the date code stamped on the plastic reads 9734. The silicon die underneath that plastic implements an instruction set designed in 1975 by Masatoshi Shima, the engineer who had already co-designed the Intel 4004 and 8080, and Federico Faggin, who had defected from Intel in 1974 to found Zilog. The Z80's register set inherits the 8080's. The opcode encoding is backwards-compatible with 8080 binaries. The chip in my hand is a physical artifact of a specific engineering defection.

The plastic package is a DIP-40. Two rows of twenty pins, 0.6 inches between rows, 0.1 inches pin-to-pin. When you drop it into a machined socket, the pins bind slightly before seating. That's not sloppy tolerance, it's designed in: the socket contacts have to wipe against the pin to break through the oxide layer that forms on the tin plating. Every retro computer from the TRS-80 to the ZX Spectrum to the MSX used this package.

The RetroShield is an Arduino shield. Erturk Kocalar published the original design on GitLab as open hardware. His version fits one Z80 on a 2x18 Arduino Mega bus header. Mine is a revision that fits two Z80s on the same shield, shared address and data buses, separate control signals on a supplementary header. The Gerbers were exported from pcb-rnd, a fork of the original gEDA PCB program maintained by Tibor Palinkas. The traces were placed by Freerouting, which Alfons Wirtz originally wrote on Oracle's dime, then re-released as MIT-licensed after Oracle lost interest. The board was fabricated by PCBWay in Shenzhen with a four-week turnaround. I soldered the DIP-40 sockets myself and discovered on the first power-up that every bus line was shorted to ground because the ground fill polygon's clearance cutouts hadn't fully encircled the pins in the Gerber export. Part 2 of this series is the story of finding that out.

When the chip executes, it reads an opcode. 0xDB is IN A, (n), read a byte from an I/O port. The Arduino Mega firmware intercepts that read, treats the Z80 as if it were a CPU attached to a memory-mapped terminal, and feeds bytes back. The bytes are a Z3 storyfile: Zork I, compiled in 1980 by the Dynamic Modeling Group at MIT's Laboratory for Computer Science using a language called ZIL, originally written in MDL on a PDP-10 and ported into a virtual machine that could run on any 8-bit or 16-bit home computer of the era. Infocom ported it to CP/M. CP/M ran on the Z80. The chain closes on itself.

Typing "GO NORTH" into the serial terminal produces, after a pause, the text "You are in an open field west of a big white house, with a boarded front door." That pause is not latency in any network sense. It is the Z80, at 4 MHz, running the Z-machine interpreter through thirty or so thousand clock cycles, each of which is a real transition of real silicon on real power.

That is the gathering. It is not decoration. Zilog is present. MIT is present. Kocalar is present. PCBWay is present. pcb-rnd is present. The engineers who decided in 1975 that the return instruction should be one byte are present, because the silicon they designed is still decoding that byte at 4 MHz on my bench. My own ground-fill debugging is present, because the fill polygon is gone from this revision and that absence has a history. The thing gathers.

What the Cloud Endpoint Gathers

You can play Zork in a browser. archive.org hosts a Frotz build compiled to WebAssembly. You click a link, a Z3 interpreter materializes in a JavaScript sandbox, a virtual screen renders a virtual terminal, and "GO NORTH" produces "You are in an open field west of a big white house." Bit for bit, the same bytes of output. The game is the same. The Z-machine is the same. The story file is the same.

But nothing gathers.

The browser tab is not a thing in Heidegger's sense. It is a runtime. Runtimes are designed to be interchangeable. Run the same Frotz build in Chrome, in Firefox, in Safari. Run it on a phone, on a desktop, on a Chromebook in a school. Each one produces identical output from identical input. That interchangeability is not an accident or a failure. It is the entire engineering accomplishment of the web stack. A Z-machine interpreter that only ran on one specific browser on one specific machine would be a lesser piece of software, not a greater one.

This is even clearer if the emulator is on a cloud-hosted runtime. You click a link to play-zork.com, it spawns a container in some datacenter, the container runs Frotz, the output streams back to you over HTTPS. Where is Zork running right now? Physically, electrically, in which building? You do not know. You are not meant to know. The service's value proposition depends on you not knowing. If US-East-1 fails over to US-East-2, your session survives with at most a reconnect. If Vercel goes under and the operator moves to Cloudflare Workers, your experience is identical. The gathering is suppressed by design.

The same is true at a higher level of abstraction. A call to api.openai.com/v1/chat/completions hits some cluster of H100s somewhere. Maybe in Texas. Maybe in Iowa. Maybe in Norway. The model behind the endpoint has weights, trained on hardware you will never see by engineers you will never meet. Tomorrow OpenAI might swap the backing model. Or add a 403 quota limit. Or migrate the inference stack to Blackwell. Your code does not change. That is the contract. The contract is the endpoint. The thing behind the endpoint is deliberately, structurally, invisible.

This is not a complaint. The contract is useful. A company running a Rails app wants exactly this: stable interface, invisible infrastructure, someone else's problem. But the cost of that abstraction, the thing you pay with, is the gathering. The endpoint cannot gather a world because the world behind it is required to be interchangeable with any other world that can satisfy the contract.

Heidegger's Jug

In 1950 Heidegger gave a lecture called Das Ding. He spent most of it talking about a wine jug. The essay is notoriously hard to read and almost comically literal. He describes the jug's sides, its base, its void. He distinguishes the jug from a cup and from a bottle. He asks what it means for a jug to be a jug.

His answer is that a jug is not defined by its shape, its material, or its containing function. A jug is defined by what it gathers. When wine is poured from the jug, there gathers in that pouring: the earth (the grape that grew in soil, the clay fired into the vessel), the sky (the rain, the sun), the mortals (the drinker, the potter, the host), and what he calls the divinities (the toast, the libation, the occasion that makes this pouring different from running tap water into a glass). The fourfold, he called it. Earth, sky, mortals, divinities.

The fourfold is the part of the essay that reads as mystical. Ignore the specific terminology if it grates. The structural claim underneath is simpler: a thing is a thing to the extent that it is a node in a web of presence. The jug is not just a container. The jug is a place where a whole world becomes, briefly and locally, present.

Heidegger's counter-example in a later essay is the bridge. The old bridge at Heidelberg is a thing in his sense. It gathers the two banks, the river underneath, the road that runs across it, the people who cross. The bridge is what makes those things into a coherent place. The hydroelectric plant on the Rhine, which he treats in The Question Concerning Technology, is not a thing. It is a piece of what he called the standing-reserve, Bestand. The plant converts the river into potential electrical output, on demand, interchangeable with any other kilowatt on the grid. The plant does not gather. It extracts.

This is the same distinction that separates the Z80 on my bench from the cloud-hosted Frotz emulator. The Z80 is a bridge. The cloud emulator is a power plant.

What's Actually Different

The functional output is the same. That is the central puzzle. The bytes of Zork's output are identical. The game is playable in either location. The player's subjective experience of "GO NORTH" producing a description of the open field is the same to within the tolerance of noticing.

What is different is what each running copy means, in a sense of meaning that is not about semantics but about presence.

The Z80 running Zork on my bench means: Zilog's 1975 design decisions, Infocom's 1980 implementation, Kocalar's open hardware, my four-week wait for PCBWay, my ground-fill debugging session, the specific 4 MHz crystal that drives this specific chip tonight. The game is the surface. The gathering is what makes the game this game and not an abstract instance of gameplay.

The cloud-hosted Zork means: the game. That's the whole content. The infrastructure is interchangeable by contract, the hardware is invisible by design, the history is irrelevant to the service. You play Zork. That is the product. The product is the endpoint. The endpoint is the product.

This is why people who run home labs can tell you war stories and people who use APIs cannot. "Remember the fan seizing on the P40 in July." "Remember when the ground fill shorted every bus line." "Remember the first time Forth actually loaded and we watched OK appear on the terminal." These stories are possible because the thing is specific, present, and has its own biography. "Remember that 503 from OpenAI last Tuesday" is not a story. It is a status page entry. The difference is not nostalgia or sentimentality. The difference is that one event happened to a thing and the other event happened to a contract.

The Enframing Connection

I wrote earlier about Enframing, Heidegger's term for the mode of revealing that dominates the modern technological era. Enframing, Gestell, is the stance that frames everything in advance as standing-reserve: resources on call, available on demand, interchangeable for the purpose at hand. The hydroelectric plant enframes the river as kilowatts. The modern timber industry enframes the forest as board-feet. The cloud endpoint enframes computation as a billable unit.

Enframing is not a villain in Heidegger's telling. It is not a mistake. It is a stance that reveals certain truths about things, specifically their exchangeability as resources, at the cost of concealing other truths, specifically their being as things.

The cloud endpoint is what Enframing looks like at the level of infrastructure. The GPU cluster is enframed as tokens-per-second, which are enframed as dollars-per-million-tokens, which are enframed as a line item on an invoice. That enframing is what makes the cloud economically tractable. It is also what makes the cloud unable to gather.

The Z80 on my bench resists Enframing. Not because it's old or small or personal, but because I haven't framed it that way. I haven't asked it to be interchangeable. I haven't said "give me CP/M compute on demand at the lowest price." I have said "here is this specific chip, running this specific program, in this specific session." That's not a resource request. That's a relationship with a thing.

This essay is not a sequel to Enframing the Code. It is a companion piece, addressing what Enframing costs. Enframing names the stance. This one names what falls out of view when the stance becomes total.

Why People Build

The retro computing and home lab communities do something that looks, from an economic standpoint, irrational. They spend four-week lead times and hundreds of dollars to produce hardware that they could replace with a five-minute browser session for free. They run LLMs on Tesla P40s pulled out of eBay auction lots when the equivalent API call would cost fractions of a cent. They solder DIP-40 sockets in their basements when the emulator is a click away.

You can explain this as nostalgia, and people sometimes do. You can explain it as hobby, and that's also partly right. You can explain it as skill acquisition, which is closer but still not the reason. The economic irrationality goes away the moment you stop assuming that the only value of running Zork is playing Zork.

People build because the thing gathers. The RetroShield is not just a way to run Zork. It is a way to make Infocom's 1980 engineering present in the room tonight. It is a way to put Faggin's chip design decisions into active service at 4 MHz. It is a way to hold the physical object that descends from Zilog's break with Intel, from MIT's Dynamic Modeling Group, from the whole genealogy of 8-bit personal computing, and to use that object for its intended purpose on a Tuesday evening fifty years after the design was finalized.

None of that is available through the endpoint. The endpoint is a contract for Zork. It is not a gathering of Zork's world.

The feeling that people describe when they say "running Zork on a real Z80 feels different" is not aesthetic preference. It is the presence of the gathering. Something is actually there that is not there when you run the emulator in a browser tab, and that something is not information. It is a specific thing's being a thing.

What This Predicts

A test of the claim: this framework predicts that communities will form around specific hardware and not around cloud providers, and it predicts which specific hardware will gather the most.

Communities form around the Tesla P40. Around the Raspberry Pi. Around the RetroShield. Around specific FPGA boards like the ULX3S and the Tang Nano 9K. Around the PDP-11 (still). Around the Apple IIe. Around the Amiga. Around AMD's Strix Halo in my own recent posts. The common feature: these are things with specific histories, specific constraints, specific failure modes, specific communities of use.

Communities do not form around "the API endpoint for a frontier LLM." They do not form around "managed Postgres." They do not form around "us-east-1." There are users of those things, and there are engineers who get very good at using them, but the thing itself is not a gathering point because the thing is structurally interchangeable. You can run managed Postgres on AWS or GCP or Azure. It doesn't matter. That's the value. That's also why no one has a tattoo of managed Postgres.

Within the cloud, communities do sometimes form, but they form around thing-like artifacts: specific open-source projects like Kubernetes or Postgres itself, specific hardware generations like the original A100 launch or the H200 launch, specific incidents like the us-east-1 outage of December 2021. The gathering happens when the abstraction fails or when a specific thing peeks through.

This is not a prediction that cloud computing will fail or that people will abandon it. They won't. The endpoint is too useful. The prediction is narrower: the cloud will never gather the way things gather, and people will keep building physical hardware in their basements even when it is economically irrational, because the gathering is not available any other way.

The Chip on the Bench

I started with the weight of a Z80, 5.7 grams. End there. The chip is still on my bench. It is in a socket. The socket is on a PCB. The PCB is in a Mega header. The Mega is connected to my laptop by USB. The laptop is rendering a serial terminal. The terminal is showing the Zork prompt. The prompt is waiting.

The physical object in front of me is small. It fits under my thumb. It was designed fifty years ago by an engineer who had just quit Intel. It has been sitting in a drawer for some years. Tonight it is running. Tonight a specific piece of silicon, fabricated in 1997, is decoding instructions written in 1980 by people in Cambridge, Massachusetts, to produce text that was designed to be read by someone sitting at a CRT terminal in a dorm room in 1982. That whole world is present on my bench, gathered by this chip, for as long as I keep the 4 MHz crystal running.

When I type "GO NORTH" and the text appears, I am not receiving a service. I am participating in a thing that is thinging, in Heidegger's awkward verb form. I am one of the mortals in the fourfold. Faggin is one. Shima is one. The Infocom implementers are some. Kocalar is one. PCBWay's fabrication technicians are some. We are all gathered around the chip for the duration of this session.

The API endpoint offers me none of this. The API endpoint offers me Zork. That's a different thing entirely, and most of the time it's what I want. But sometimes, on a Tuesday evening in 2026, it isn't, and the reason why has a name.

Designing a Dual Z80 RetroShield: Ground Planes, Ghost Shorts, and the Fix (Part 2)

A.C. Jokela — Mon, 06 Apr 2026 22:00:00 GMT

The assembled dual Z80 RetroShield on the bench. Two Z80 CPUs socketed, jumper wires from J2 to the Arduino Mega's free pins, ready for testing.

In Part 1, I designed a dual-Z80 RetroShield PCB entirely from the command line: two Z80 CPUs sharing an address and data bus, with independent control signals on a supplementary header. The Gerber files went to PCBWay. The boards arrived. I soldered everything up, plugged the shield into an Arduino Mega, wired jumpers from the J2 control header to the Mega's free pins, loaded a test sketch, and...

Nothing. Both Z80s appeared to be alive (the diagnostic showed bus activity after reset), but they couldn't execute a single instruction. The data bus was completely unresponsive. The SMP kernel I'd written—a 52-byte symmetric multiprocessing demo where both CPUs boot the same code, pull tasks from a shared scheduler, and sum arrays in parallel—hit its cycle limit and returned zeroes.

What followed was a multi-day debugging session that taught me more about PCB design than the entire design phase did. The root cause turned out to be a subtle interaction between pcb-rnd's ground fill polygon and its Gerber exporter. This is the story of finding it.

The Hardware

The boards came back from PCBWay with the usual four week turn around time. Clean fabrication, good silkscreen, no obvious defects on visual inspection. I soldered DIP-40 sockets for both Z80s, the 2×18 J1 bus header, the 2×6 J2 control header, and the bus activity LED. The Z80 chips dropped into their sockets with satisfying precision.

The J2 header needed jumper wires to the Arduino Mega's free pins (D0–D21). I chose a deliberate mapping based on the pin functions: D9 for CPU2's clock (Timer1 OC1A, which can generate a hardware PWM signal for a stable 4 MHz clock), D4 for RESET, D5–D6 for INT/NMI, D7–D8 for MREQ/IORQ, D10–D11 for RD/WR, and D12–D13 for BUSRQ/BUSAK. Twelve jumper wires total, plus +5V and GND.

The plan was to run a five-test validation suite: each Z80 solo (write a signature byte to a known address), shared RAM persistence (both CPUs write to different locations, verify both persist), a relay test (CPU1 computes a value, CPU2 picks it up and continues), and a loop counter (DJNZ loop to verify branch instructions work). After that, the SMP kernel.

None of it worked.

Routing traces with Freerouting.

First Contact: The Diagnostic Sketch

I backed off to a simpler diagnostic sketch to test each connection individually. It checked four things: idle state of control pins, bus activity after releasing reset, address bus bit toggling, and the BUSRQ/BUSAK handshake on CPU2.

The results were a mix of encouraging and confusing:

=== Test 1: Control Pin Idle State ===
  [FAIL] U1 MREQ (D41) idle HIGH
  [FAIL] U1 IORQ (D39) idle HIGH
  [FAIL] U1 RD   (D53) idle HIGH
  [FAIL] U1 WR   (D40) idle HIGH

=== U1: Bus Activity ===
  [PASS] MREQ went LOW (Z80 is alive)
  [PASS] RD went LOW (Z80 fetching)
  [PASS] Address bus active — first fetch address = 0x0000

Both Z80s were "alive" in the sense that they responded to clock pulses and attempted to fetch from address 0x0000 after reset. But every control signal (MREQ, IORQ, RD, WR) read LOW regardless of what the Z80 was doing. They should have been toggling between HIGH and LOW during bus cycles. LOW all the time meant either the Z80 wasn't actually driving these pins, or something else was pulling them down.

I filed this under "weird but not fatal" and pushed ahead to the SMP test. That's when things got serious.

The Bus Trace That Went Nowhere

The SMP kernel loaded into emulated memory at address 0x0000. After releasing reset, the Z80 should have fetched its first instruction (0xDB, the opcode for IN A, (n)), executed it, and proceeded through the scheduler loop. Instead, a 150-cycle bus trace showed this:

Cyc  MREQ IORQ RD WR  Addr    Data  Action
  0  LOW   LOW   L  L  0x0000  0xDB  MEM RD
  1  LOW   LOW   L  L  0x0000  0xDB  MEM RD
  2  LOW   LOW   L  L  0x0000  0xDB  MEM RD
  ...
149  LOW   LOW   L  L  0x0000  0xDB  MEM RD

Every single cycle: same address, same data, same control signals. The Z80 was stuck at its reset vector, endlessly attempting to fetch the first byte and never advancing. MREQ and IORQ were LOW simultaneously on every cycle, which should never happen during normal Z80 operation—they're mutually exclusive signals.

The Z80 was putting 0x0000 on the address bus (correct for a reset vector fetch), and I was driving 0xDB on the data bus (the correct opcode). But the Z80 wasn't reading it. Or rather, it was reading something else.

The Data Bus Loopback Test

I added a simple test: with the Z80 held in reset (outputs tri-stated), drive patterns on the data bus and read them back. If the Arduino writes 0xFF to PORTL and reads back 0xFF from PINL, the data bus is clean. If it reads back something else, there's a short or broken trace.

--- Data Bus Drive Test ---
  bit0(D49)  wrote 0x01 read 0x01 OK
  bit1(D48)  wrote 0x02 read 0x02 OK
  bit2(D47)  wrote 0x04 read 0x04 OK
  bit3(D46)  wrote 0x08 read 0x08 OK
  bit4(D45)  wrote 0x10 read 0x10 OK
  bit5(D44)  wrote 0x20 read 0x00 FAIL
  bit6(D43)  wrote 0x40 read 0x00 FAIL
  bit7(D42)  wrote 0x80 read 0x80 OK

Bits 5 and 6 were stuck LOW. The Arduino's GPIO pins couldn't drive them HIGH—something on the board was pulling those lines to ground with enough strength to overpower the Mega's output drivers.

This explained why the Z80 couldn't execute. When I drove 0xDB (IN A, (n)) on the data bus, the Z80 actually saw 0x9B (bits 5 and 6 forced low), which decodes as SBC A, E—a completely different instruction. The Z80 was faithfully executing garbage.

Isolating the Short

Systematic isolation. First question: is it the Arduino or the board?

I pulled the RetroShield off the Mega and ran the same loopback test with the shield disconnected. Every bit passed perfectly. The Arduino's PORTL pins (D42–D49) could drive any pattern and read it back correctly. The problem was definitively on the board.

Second question: is it CPU1 or CPU2? The two Z80s share the address and data bus, so a fault on either side would affect both. I disconnected J2's +5V jumper to deprive CPU2 of power, leaving its outputs floating. Same result—bits 5 and 6 still stuck LOW. So CPU2 wasn't the culprit. The short was in CPU1's territory.

Third question: is it the Z80 chip or the PCB? I pulled U1 from its socket. Bits 5 and 6 still stuck. Pulled U2 as well (since it shares the bus traces even without power). Both chips out, empty sockets, and the shorts persisted.

Then I ran a comprehensive pin test with both Z80 chips removed—just the bare PCB with sockets:

--- Data Bus (PORTL) ---
  [SHORT] D0/bit0 (D49/PL0)
  [SHORT] D1/bit1 (D48/PL1)
  [SHORT] D2/bit2 (D47/PL2)
  [SHORT] D3/bit3 (D46/PL3)
  [SHORT] D4/bit4 (D45/PL4)
  [SHORT] D5/bit5 (D44/PL5)
  [SHORT] D6/bit6 (D43/PL6)
  [SHORT] D7/bit7 (D42/PL7)

--- Address Bus ---
  [SHORT] A0 through A15 — all 16 lines

--- U1 Control Pins ---
  [OK] MREQ, IORQ, RD, WR, RESET, INT, NMI, CLK — all 8 fine

Every single data line. Every single address line. All 24 shorted to ground. But all 8 control signals were clean.

This wasn't random solder bridges. The pattern was too systematic: every line that belonged to the shared bus (connected to both U1 and U2) was shorted, while every line that connected to only one Z80 was fine. Something structural was wrong with the PCB.

The Ground Fill

I went back to the PCB design files. The original RetroShield design included a copper fill polygon on the bottom layer—a ground plane covering roughly the left half of the board (x = 0.3mm to 55.6mm, y = 0.3mm to 53.1mm). This is standard practice: ground planes reduce noise, improve signal integrity, and provide a low-impedance return path for high-frequency signals.

The polygon had a clearpoly flag, which tells pcb-rnd to maintain clearance around pins that aren't connected to the fill. Each Z80 through-hole pin specified 0.762mm clearance. The fill should have maintained that gap around every signal pin, connecting only to GND pins (via thermal relief pads) and leaving all data and address pins isolated.

I also found some design-level flag errors. U1 pin 1 (A11, an address line) had a thermal(0X) flag—explicitly telling pcb-rnd to connect this signal pin to a copper fill on the top layer. Several +5V pins had connected flags. These were wrong, though in pcb-rnd's net-aware polygon system, they turned out to be harmless (a connected flag only connects a pin to a fill on the same net, so +5V pins wouldn't connect to a GND fill). I fixed them anyway.

But fixing the flags didn't solve the short. The 24 bus lines were still shorted to ground with both chips removed. The problem was deeper.

The Gerber Analysis

I dug into the actual Gerber output for the bottom copper layer. In Gerber format, ground fill clearances are typically achieved either through layer polarity commands (%LPC*% to switch to "clear" mode and punch out holes) or by drawing the fill as a region with the clearance areas built into its outline.

pcb-rnd uses the region approach. The fill polygon gets exported as a complex region (G36/G37 block) whose boundary weaves around each pin, creating clearance cutouts. Or at least, that's what it's supposed to do.

I wrote a script to analyze the region vertices near U1's pins. For pin 9 (D5, a signal pin that should have full clearance), this is what the Gerber contained:

# U1-9(D5) SIGNAL at Gerber(120000,120000)
# Region 20: 5 vertices within 2.54mm
#   (120000,116587) dist=0.87mm angle=-90°
#   (121413,128587) dist=2.21mm angle=81°

Five vertices. Two distinct positions. A proper circular clearance cutout around a through-hole pin needs 8 to 16 vertices distributed at various angles to approximate the circle. This had vertices at just two angles: -90° and 81°. The polygon boundary was making a shallow notch past the pin, not encircling it.

Even worse: the closest vertex was only 0.87mm from the pin center. The pad edge is at 0.762mm (half of the 1.524mm pad diameter). That left 0.108mm of clearance—about 4.3 mil—on the side where the boundary came closest. Manufacturing tolerance at PCBWay is typically 4-6 mil. The clearance was right at the edge, and on other sides of the pin, there was no clearance at all because the polygon boundary didn't go around.

For comparison, U1 pin 29 (GND, which should connect to the fill) had a vertex at exactly 0.00mm distance. The fill went right through it. Correct.

The Root Cause

pcb-rnd's Gerber exporter was generating incomplete clearance cutouts in the ground fill polygon around through-hole pins. Instead of tracing a complete circle around each pin (maintaining the specified 0.762mm clearance on all sides), it was generating partial notches that only cleared the pin on one or two sides. On the remaining sides, the ground fill copper made direct contact with the pin pad.

This affected every through-hole pin inside the polygon's boundary: all of U1's pins, all of J1's pins, and most of the vias. The pattern now made sense. The ground fill polygon covered x = 0.3mm to 55.6mm—the left side of the board. U1 (at x = 30.48mm) was squarely inside. U2 (at x = 66.0mm) was outside. All 24 bus lines pass through U1's footprint. All 8 control lines connect only to one Z80 and route through traces, not through-hole pads, in the fill area.

The reason the initial diagnostic showed control signals as "OK" while bus lines were "SHORT" was purely geometric: the control signal traces exited the fill area quickly and reached the Arduino pins via the top copper layer, while the bus lines had through-hole pads sitting directly in the fill.

It's worth noting that this bug is specific to the combination of pcb-rnd's polygon fill, through-hole pins, and Gerber export. SMD pads weren't affected (no SMD components were inside the fill area). The clearance math in pcb-rnd's internal representation appeared correct, but the translation to Gerber region vertices lost fidelity, producing polygon outlines that didn't fully encircle the pins.

The Fix

I removed the ground fill polygon entirely.

# Find and remove the Polygon block in Layer 2 (bottom)
# Strategy: parse through the file, skip everything
# between 'Polygon("clearpoly")' and its closing ')'

The board's GND connectivity doesn't depend on the fill. The autorouter (Freerouting) had already placed explicit copper traces connecting all GND pins. The fill was adding copper density and potentially improving signal integrity, but neither matters for a board running Z80s at 4 MHz. At these speeds, the electrical benefit of a ground plane is negligible, and the manufacturing risk (as we discovered) is real.

I also cleaned up the erroneous flags while I was in there:

Removed thermal(0X) from U1 pin 1 (A11 should not connect to any fill)
Removed connected from all +5V pins (J1-1, J1-36, U1-11, U1-24, U1-25, U2-11, U2-24, U2-25)
Removed thermal/connected from all 11 vias (signal vias should get clearance, not connection)
Left thermal flags only on GND pins: J1-18, J1-19, and U1-29

The bottom copper Gerber went from 46KB (fill + traces) to 16KB (traces only). The region block count dropped from 26 to 5. Regenerated the Excellon drill file, rebuilt the production zip with BOM and centroid, and pushed everything to the repository.

Was This Always Broken?

A natural question: is this a flaw in the original RetroShield Z80 design, or something we introduced? The ground fill polygon, the thermal(0X) flag on U1 pin 1, and all the connected flags exist in Erturk Kocalar's upstream design — identical to our initial commit. We didn't add any of them. So why does the original RetroShield work?

To find out, I ran the original, unmodified kz80.pcb through the same pcb-rnd Gerber exporter and performed the same vertex analysis on the output. The results were revealing:

Pin	Our Board (re-routed)	Original Board (original traces)
U1-9 (D5, signal)	5 vertices, closest 0.87mm from center	75 vertices, closest 1.14mm from center
U1-10 (D6, signal)	similar	169 vertices, closest 1.14mm
U1-29 (GND)	vertex at 0.00mm (correct)	vertices at 0.76mm (thermal spokes, correct)

The original board, exported through the same pcb-rnd Gerber exporter, gets proper circular clearance cutouts with 75+ vertices distributed around each signal pin at a safe 1.14mm from center (0.38mm clearance from pad edge). Our re-routed board gets 5 vertices at only 0.87mm from center (0.11mm clearance — below manufacturing tolerance).

The difference is the trace geometry. The original RetroShield's traces were routed with classic gEDA/pcb. When we added the second Z80, we stripped all traces and autorouted from scratch with Freerouting. The new trace layout changed how pcb-rnd's polygon fill algorithm tessellated the clearance boundaries. With different traces running through the fill area, the polygon's outline took different paths around the pins — and those paths didn't maintain adequate clearance.

So the design flaw (wrong flags, a ground fill with tight clearance margins) was always latent in the original design. But it only manifested as physical shorts when the trace geometry changed. The original routing happened to produce geometry that pcb-rnd's exporter handled gracefully. Our autorouted traces didn't. It's the kind of bug that lies dormant until you touch something seemingly unrelated.

This is why removing the polygon entirely was the right fix. It doesn't matter how the traces are routed if there's no fill to create clearance problems against. The GND connectivity is fully handled by explicit routed traces.

Verification

To confirm the fix, I verified the new Gerber output:

No layer polarity commands (no fill, no clearance needed)
No large region blocks (no polygon fill)
Only trace geometry and pad flashes remain on the bottom copper layer
Bottom copper file size reduced by 65% (46KB → 16KB)

The new production files have been submitted to PCBWay for a second fabrication run. The fix is structural: without the fill polygon, there's nothing to short to. Every GND connection is an explicit routed trace, visible in the Gerber, and verifiable by inspection.

The corrected Rev C board: 553 traces, 25 vias, no ground fill polygon. Clean explicit routing for all 48 nets.

What I Learned

Ground fills aren't free. They improve signal integrity on high-speed boards, but on a 4 MHz Z80, the benefit is marginal. The cost is an additional failure mode: if the clearance generation is buggy, incomplete, or at the edge of manufacturing tolerance, the fill becomes a liability. For simple retro computing boards, explicit GND traces are more predictable.

Autorouting can break things you didn't touch. The ground fill polygon wasn't something we modified. But by re-routing the traces (which we had to do after adding the second Z80), we changed the geometry that the polygon fill algorithm used to compute clearances. A latent design flaw became an active one. When you re-route a board with copper fills, you need to re-verify the fills, not just the traces.

Test the bare PCB before populating it. If I'd run a continuity test between the Z80 socket pads and GND before soldering anything, I'd have caught this immediately. Instead, I spent time debugging firmware, suspecting timing issues, and questioning my understanding of Z80 bus cycles. The problem was never software.

The Gerber is the contract. The PCB design tool's internal representation doesn't matter; only the Gerber output does. Even if pcb-rnd's polygon clearance looks correct on screen, the Gerber export is what the fab house uses. Verify the Gerber, not the design file. A Gerber viewer would have shown the incomplete clearances immediately.

Diagnostic sketches are invaluable. Writing targeted Arduino sketches that tested individual pins, drove patterns, and reported results over serial turned a "nothing works" situation into a systematic narrowing process. The data bus loopback test (drive a byte, read it back, compare) is trivially simple and would have caught this on day one.

AI-assisted debugging works the same way AI-assisted design works. I brought the domain knowledge (how Z80 bus signals work, what the control signal timing should look like, what "stuck LOW" means electrically) and the AI handled the tedious parts: writing diagnostic firmware, parsing Gerber files, analyzing polygon vertices, checking coordinate math. The same division of labor that made the design possible also made the debugging possible.

Solder bridges are a red herring when the pattern is systematic. Early on, I found and fixed a solder bridge between two adjacent pins. It didn't help. When two pins are bridged, you get two bad signals. When 24 pins are all shorted to the same rail, the cause is structural, not incidental. I should have recognized the pattern sooner and stopped looking at individual joints.

Open-source hardware needs open-source verification. The entire reason I could diagnose this was that every file in the chain—PCB source, Gerber output, Excellon drill files—was text-based, parseable, and inspectable. I wrote Python scripts to analyze the Gerber's region vertices and measure distances to pin centers. Try doing that with a proprietary board file. The text-based EDA workflow that made the design possible also made the debugging possible.

What's Next

The corrected boards are in fabrication. When they arrive, Part 3 will cover what this project was always about: bringing up the SMP kernel, watching two Z80 processors boot the same code, identify themselves, and divide work across a shared memory bus. The kernel is 52 bytes. The scheduler is in the Arduino. The demo sums an array split across both CPUs and measures the speedup.

Both Z80s are confirmed alive. They just need a board that doesn't short their bus to ground.

The complete source—PCB files, Gerber production package, diagnostic sketches, SMP kernel, and wiring guide—is on GitHub.

Teaching an LLM a Language It Has Never Seen

A.C. Jokela — Thu, 02 Apr 2026 13:00:00 GMT

Lattice is a programming language I designed. Its central feature is the phase system: every runtime value carries a mutability tag that transitions between states the way matter moves between liquid and solid. You declare a variable with flux (mutable) or fix (immutable). You freeze a value to make it immutable, thaw it to get a mutable copy, and sublimate it to make it permanently frozen. forge blocks let you build something mutably and have the result exit as immutable. None of this exists in any other language.

Lattice does not appear in Claude's training data. I designed the language after the knowledge cutoff. There is no Lattice source code on GitHub (other than my own repository). There are no Stack Overflow answers. There is no tutorial ecosystem, no community blog posts, no textbook chapters. The only documentation that exists is the code itself, a 38-chapter handbook I wrote, and three blog posts on this site.

Claude writes Lattice fluently. It writes correct programs using the phase system, the concurrency primitives, the module system, and the trait/impl pattern. It writes struct definitions with per-field phase annotations. It uses forge blocks and anneal expressions correctly. And it wrote a 4,955-line self-hosted compiler in Lattice, for Lattice: a complete tokenizer, parser, and bytecode code generator that reads .lat source files and emits .latc bytecode binaries.

The question is how any of this is possible when the model has never seen the language before.

The Rust Smell

The answer starts with syntax. Here is a Lattice function:

fn greet(name: String) -> String {
    return "Hello, ${name}!"
}

And here is the Rust equivalent:

fn greet(name: &str) -> String {
    format!("Hello, {name}!")
}

The fn keyword, the colon-separated type annotations, the -> return type, the curly braces: Claude has seen these patterns millions of times in Rust code. When it encounters them in Lattice, it doesn't need to learn a new syntax. It needs to recognize a familiar one.

This extends deep into the language. Lattice structs look like Rust structs:

struct Point {
    x: Float,
    y: Float
}

Lattice enums look like Rust enums:

enum Shape {
    Circle(Float),
    Rectangle(Float, Float)
}

Lattice match expressions look like Rust match expressions:

match shape {
    Shape::Circle(r) => pi() * r * r,
    Shape::Rectangle(w, h) => w * h,
    _ => 0.0
}

Lattice traits and impl blocks look like Rust traits and impl blocks:

trait Printable {
    fn display(self: any) -> String
}

impl Printable for Point {
    fn display(self: any) -> String {
        return "(${self.x}, ${self.y})"
    }
}

Closures use the same |params| body syntax. The .. range operator works the same way. The ? postfix operator propagates errors. for item in collection iterates. let binds variables. The structural similarity is pervasive enough that a model trained on Rust can parse and generate Lattice code without any Lattice-specific training.

I did not design Lattice to be AI-friendly. I designed it because Rust's syntax is good and I wanted to use it for a language with different semantics. But the side effect is that Claude can write Lattice from day one because the syntax activates the same neural pathways that Rust does. The model doesn't know it's writing a different language. It knows it's writing code that looks like Rust, and the structural patterns transfer.

The Phase System: Where Familiarity Ends

The Rust resemblance carries Claude through basic Lattice programs without difficulty. Where it gets interesting is the phase system, because this is where Lattice has no analog in any language Claude has seen.

In Rust, mutability is a static property: let mut x = 5; or let x = 5;. You decide at declaration time and the compiler enforces it. In Lattice, mutability is a runtime state that values transition through:

flux counter = 0          // mutable
counter = counter + 1     // allowed: counter is fluid

freeze(counter)           // transition: fluid → crystal
counter = counter + 1     // runtime error: counter is crystal

flux copy = thaw(counter) // get a mutable copy
copy = copy + 1           // allowed: copy is fluid

Claude handles this correctly. When I describe the phase system and provide examples, Claude generates code that uses flux and fix declarations appropriately, calls freeze() at the right points, and avoids mutating crystal values. The model maps flux to "mutable variable" and fix to "immutable variable" in its internal representation, and the transition functions (freeze, thaw) become explicit state changes that it tracks through the program.

The harder constructs are the ones with no familiar analog.

forge blocks are mutable construction zones whose output exits as immutable:

fix config = forge {
    flux c = {}
    c.host = "localhost"
    c.port = 8080
    c.debug = false
    c   // exits the forge block as crystal
}
// config is now crystal; cannot be modified

Claude gets this right because the pattern (build something mutably, freeze the result) maps to the builder pattern in Rust and other languages. The syntax is novel but the concept isn't.

anneal is harder. It temporarily thaws a crystal value into a mutable binding for the duration of a block, then re-freezes it:

fix settings = forge { flux s = {}; s.theme = "dark"; s }

anneal(settings) |s| {
    s.theme = "light"   // temporarily mutable
}
// settings is crystal again, with theme = "light"

Claude produces correct anneal code when given the semantics, but it occasionally generates patterns that would work in Rust (taking a &mut reference) but don't apply in Lattice (where anneal is the only way to modify a crystal value in place). The model's Rust intuitions are strong enough to produce syntactically valid Lattice but sometimes semantically incorrect programs, because it defaults to Rust's mutation model when the Lattice-specific construct is unfamiliar.

The reactive phase system is where Claude needs the most guidance. react, bond, and seed have no precedent in any mainstream language:

flux temperature = 72.0

react("temperature", fn(name, old_phase, new_phase) {
    print("${name} changed from ${old_phase} to ${new_phase}")
})

freeze(temperature)  // triggers the reaction callback

flux primary = "active"
flux mirror = "active"

bond("mirror", "primary", "sync")  // when primary changes phase, mirror follows

freeze(primary)  // mirror also freezes

Claude can produce these patterns when given the API, but it doesn't intuit them. It never suggests react or bond unprompted, because there's nothing in its training data that would trigger the association. These constructs must be taught explicitly. The Rust smell gets Claude through 80% of Lattice. The last 20% requires actual specification.

The Spectrum of Difficulty

Working with Claude on Lattice code over several months has revealed a clear gradient of difficulty:

Trivial (Rust transfer): Functions, structs, enums, match expressions, closures, for loops, string interpolation, module imports, error propagation with ?. Claude writes these correctly on the first attempt because they're syntactically identical to Rust.

Easy (new vocabulary, familiar concept): flux/fix declarations, freeze()/thaw() calls, basic phase checking. Claude maps these to mutable/immutable patterns it already knows. The vocabulary is new; the concept isn't.

Moderate (new pattern, teachable): forge blocks, anneal expressions, crystallize blocks, struct field-level phase annotations (alloy structs). These require explanation, but once Claude sees one or two examples, it generalizes correctly. The builder pattern and block-scoped mutation are close enough to existing patterns that the model bridges the gap.

Hard (no analog, requires specification): Reactive phase operations (react, bond, seed), phase pattern matching (fluid val =>, crystal val =>), the concurrency constraint that only crystal values can be sent on channels, strict mode's consumption semantics for freeze. Claude can use these but never invents them. They must be explicitly described.

The concurrency constraint is a good example of the "hard" category. In Lattice, data sent on a channel must be crystal:

let ch = Channel::new()
flux data = "mutable"

// ch.send(data)  // runtime error: cannot send fluid value

freeze(data)
ch.send(data)     // works: data is now crystal

This rule exists because crystal values are deeply immutable: they can't be modified by the sender after transmission, which eliminates data races structurally. Claude understands the concept (Rust has Send and Sync traits that serve a similar purpose), but it doesn't automatically apply Lattice's specific rule without being told. Left to its own devices, Claude will try to send fluid values on channels, because that's what you'd do in Go or Python. The constraint must be stated.

Strict mode (#mode strict at the top of a file) is another case where Claude needs explicit guidance. In strict mode, let is banned (you must use flux or fix), freeze() consumes the original binding (Rust-like move semantics), and crystal bindings cannot be assigned to at all, not even as a runtime error. Claude can write strict-mode Lattice, but it defaults to casual-mode patterns unless reminded. The model's prior is "permissive runtime" because that's what most dynamic languages are.

The gradient correlates exactly with how much the construct resembles something in Rust or another mainstream language. When the syntax is familiar, Claude's transfer learning handles it. When the concept is familiar but the syntax is new, one or two examples are enough. When both the syntax and the concept are novel, Claude needs the specification.

The Self-Hosted Compiler

The strongest evidence that Claude can deeply understand a language it was never trained on is latc.lat: a 4,955-line self-hosted compiler written in Lattice, for Lattice.

The compiler reads .lat source files and emits .latc bytecode binaries. It has twelve sections:

Opcode constant definitions (mapping all 100+ VM opcodes to integers)
Token stream and cursor helpers (peek, advance, expect, match_tok)
Compiler state management (save/restore for nested compilation)
Error reporting
Bytecode emit helpers (emit_byte, emit_jump, patch_jump, emit_loop)
Constant pool management (integers, floats, strings, closures)
Scope and variable resolution (begin_scope, end_scope, resolve_local, upvalue tracking)
Expression parsing (precedence climbing, binary/unary ops, calls, field access)
Statement compilation (let/flux/fix, if/while/for, return, match, try/catch)
Declaration compilation (functions, structs, enums, traits, impl blocks)
Binary serialization (writing the LATC file format with magic bytes, version header, chunk data)
Main entry point

Claude wrote this. Not "Claude assisted with this" or "Claude generated boilerplate for this." Claude wrote a recursive descent parser for Lattice's grammar, a bytecode compiler that emits correct opcodes for the phase system, and a binary serializer that produces files the C runtime can load and execute. The compiler bootstraps: you run it with the C-based clat interpreter, and it produces bytecode that the same interpreter executes.

The compiler itself uses Lattice's phase system for its own internal state. The compiler's mutable working data (the bytecode buffer, the constant pool, the local variable tracking arrays) is declared with flux:

flux code = []
flux c_lines = []
flux constants = []
flux local_name_arr = []
flux local_depth_arr = []
flux local_captured_arr = []
flux local_count = 0

This is the compiler eating its own dogfood. The mutable state that the compiler needs to build bytecode is declared using the same phase system that the compiler is compiling. The phase keywords aren't decorative here; they're structurally necessary because the compiler modifies these arrays on every opcode emission and scope transition.

The compiler has 118 functions across 12 sections, with 554 opcode references. It handles every construct in the language: flux/fix declarations, forge blocks, freeze/thaw/sublimate calls, anneal and crystallize expressions, struct and enum definitions with phase annotations, trait/impl blocks, match expressions with phase-aware pattern matching, structured concurrency with scope/spawn, channel operations, try/catch, defer, and the complete expression grammar with correct operator precedence.

Writing a self-hosted compiler requires understanding the language at every level simultaneously. The tokenizer must know every keyword, operator, and delimiter. The parser must handle every grammatical production, including the phase-specific constructs (forge, anneal, crystallize) that exist nowhere in Claude's training data. The code generator must emit the correct opcodes for phase transitions, reactive bindings, and structured concurrency. And the whole thing must be written in the language being compiled, which means Claude is writing Lattice to compile Lattice, using constructs it learned from examples rather than training data.

The compiler's serialization section writes the LATC binary format byte by byte:

fn serialize_latc(ch: any) {
    ser_buf = Buffer::new(0)

    // Header: "LATC" + version(1) + reserved(0)
    write_u8(76)    // 'L'
    write_u8(65)    // 'A'
    write_u8(84)    // 'T'
    write_u8(67)    // 'C'
    write_u16_le(1) // format version
    write_u16_le(0) // reserved

    serialize_chunk(ch)
    return ser_buf
}

This is not pattern matching against compiler source code from the training data. No Lattice compiler exists in the training data. Claude wrote a compiler for a language that has no prior art, in a language that has no prior art, producing a binary format that has no prior art. Every decision (the magic bytes, the chunk serialization order, the upvalue encoding) came from understanding the specification I provided and the runtime behavior of the C-based interpreter.

What I Actually Gave Claude

The teaching process was less structured than you might expect. There was no formal curriculum, no staged introduction of concepts, no carefully sequenced lesson plan. And I should be honest about the recursive nature of what happened: Claude Code was the primary tool for building Lattice itself. The language, the C implementation, the grammar, the runtime, the test suite, the handbook: all of it was built with Claude Code. I designed the language and directed the implementation, but Claude wrote the C, the LaTeX, and the example programs.

So the situation is: Claude wrote Lattice (the implementation), and then Claude wrote in Lattice (the programs and the self-hosted compiler). The model built the language and then learned the language it built. The "teaching material" that Claude uses to write Lattice code is documentation and examples that Claude itself produced in earlier sessions.

The artifacts:

The C implementation: ~80 source files, the parser, the VM, the phase system runtime. Built with Claude Code from my architectural direction.
A handbook: 38 chapters covering every feature, with worked examples. Written in LaTeX with Claude Code. This lives in a repository that Claude can read in subsequent sessions.
Example programs (examples/phase_demo.lat, examples/sorting.lat, examples/state_machine.lat) that demonstrate idiomatic Lattice. Written by Claude Code.
815 test files under AddressSanitizer that exercise every construct. Written by Claude Code.
An EBNF grammar reference as an appendix to the handbook.

When I work with Claude on Lattice code, I don't paste the entire handbook into the context window. Claude has access to the project directory. It reads files as needed. If I ask it to write a function that uses forge, it reads examples/phase_demo.lat or chapters/ch11-phases-explained.tex to see how forge works. If I ask it to add an opcode to the compiler, it reads include/stackopcode.h and src/stackvm.c to understand the existing instruction set.

The key insight: Claude doesn't need to be trained on a language to write it. It needs access to the specification and examples at inference time. And in this case, those specifications and examples were produced by Claude itself in prior sessions. The model's understanding is constructed on the fly from documentation in its context, not retrieved from weights. This is why the Rust resemblance matters so much: the syntax gives Claude a structural scaffold, and the specification (which Claude wrote) fills in the semantics.

This is also why the self-hosted compiler was possible. By the time Claude wrote latc.lat, it had already written the entire language implementation, the handbook, the test suite, and hundreds of example programs. The language had moved from "novel" to "familiar" through accumulated context, not through training. Each session built on the last. Each example reinforced the phase system's rules. By the time the compiler was attempted, Claude's working understanding of Lattice (constructed from its own prior output) was deep enough to write a 5,000-line program that correctly compiles the language. The model taught itself a language by building the language first.

Why Syntax Matters More Than Semantics

The Lattice experience suggests something counterintuitive about how LLMs interact with programming languages: syntax transfer is more powerful than semantic understanding.

Claude can write correct Lattice because Lattice looks like Rust. The semantic differences (phase system vs. ownership, runtime type checking vs. compile-time guarantees, garbage collection vs. RAII) are significant, but they don't prevent Claude from producing working code. The model generates syntactically valid Lattice from Rust patterns and then adjusts the semantics when corrected.

This has implications for language design. If you want AI tooling to support your language from day one, without waiting for it to appear in training data, design your syntax to rhyme with something popular. Lattice's resemblance to Rust wasn't designed for AI, but it is the reason AI can write it. A language with a radically different syntax (APL, Forth, J) would be much harder for Claude to learn from examples alone, even if the semantics were simpler.

The reverse is also true: a language with familiar syntax but deeply unfamiliar semantics (like Lattice's reactive phase system) will produce code that looks correct but occasionally behaves wrong. Claude's Rust intuitions are strong enough to generate valid-looking phase code, but the model sometimes falls back to Rust's mutation model when the Lattice-specific behavior is more constrained. The syntax transfers perfectly. The semantics require teaching.

Implications for Language Designers

If you're designing a new programming language in 2026, the AI tooling question is unavoidable. Your language won't have IDE plugins, autocompleters, or AI coding assistants on day one. The community doesn't exist yet. The training data doesn't include your language. Every other language your users work with has Copilot or Claude support. Yours doesn't.

Lattice suggests a strategy: make your syntax rhyme with something an LLM already knows.

This isn't about copying Rust. Lattice has genuinely novel semantics. The phase system, the reactive bindings, the alloy structs with per-field phase annotations: none of these exist in Rust. But they're expressed through syntax (keywords, braces, type annotations, block expressions) that maps directly to Rust's structural patterns. Claude can parse the syntax without help and learn the semantics from examples.

The alternative is designing a syntax so novel that LLMs can't bootstrap from existing knowledge. This is a legitimate design choice; some ideas genuinely need new notation. But the cost is high: your users won't get AI assistance until your language appears in training data, which requires the language to become popular first, which is harder without AI assistance. It's a chicken-and-egg problem that familiar syntax sidesteps.

The practical recommendation: novel semantics, familiar syntax. Invent the ideas. Borrow the notation. Let the LLM cross the bridge on syntax and learn the semantics on the other side.

What This Means for the "AI Writes Code" Conversation

The Lattice case study complicates the popular narrative about AI code generation in both directions.

For the optimists who say AI can learn anything: Claude cannot invent the reactive phase system. It cannot propose bond or seed or anneal without being told they exist. The novel constructs, the ones that make Lattice a genuinely different language rather than a Rust reskin, are invisible to the model until explicitly specified. AI transfer learning has limits, and those limits are at the boundaries of what the training data contains.

For the pessimists who say AI can only regurgitate training data: Claude wrote a 5,000-line self-hosted compiler for a language it has never seen. That is not regurgitation. The compiler produces correct bytecode for constructs (phase transitions, reactive bonds, per-field phase annotations) that exist in no other language. The model assembled knowledge from its understanding of compilers generally, Rust syntax specifically, and the Lattice specification I provided, and produced something genuinely new. Antirez called this "assembling knowledge" when he observed the same phenomenon with his Z80 emulator project. I think that's the right term.

The truth is somewhere that neither camp wants to occupy. LLMs can go far beyond their training data when the new territory is structurally adjacent to something they know. They cannot go beyond their training data when the new territory is structurally novel. The boundary between "adjacent" and "novel" is syntax. Familiar syntax is a bridge. Novel syntax is a wall. Novel semantics behind familiar syntax is a trap: the model crosses the bridge confidently and then occasionally falls.

Lattice exists in all three zones simultaneously. Its Rust-like surface lets Claude cross the bridge. Its phase system is the novel semantics behind familiar syntax. And the self-hosted compiler is proof that the bridge, once crossed, supports weight that no one expected.

I didn't set out to test the limits of LLM language understanding when I designed Lattice. I set out to build a programming language with a novel approach to mutability. The AI dimension was a side effect: I used Claude Code as my development tool because I use Claude Code for everything, and the language happened to be learnable because it happened to look like Rust. But the result is one of the more complete demonstrations of LLM transfer learning applied to a genuinely novel domain: not just writing programs in an unfamiliar language, but writing a compiler for that language, in that language, from a specification that exists nowhere in the training data.

The 4,955 lines of latc.lat are the proof that LLMs can go further than their training data when the conditions are right. The conditions are: familiar syntax, clear specification, accessible examples, and a human who knows when the model is wrong. Remove any one of those and the compiler doesn't get written. But with all four in place, the model produces something that works, that compiles, and that no human typed by hand.

Distilled Reasoning on Strix Halo: Running a Claude-Trained Thinking Model Locally

A.C. Jokela — Sun, 29 Mar 2026 14:00:00 GMT

There is a specific moment in the open-source LLM ecosystem that keeps recurring: someone takes a frontier model's outputs, uses them as training data for a smaller model, and publishes the result. The technique is called distillation, and it has been applied to coding ability, instruction following, and general knowledge. What is newer is distilling reasoning—the step-by-step chain-of-thought process that models like Claude use internally when working through complex problems.

Jackrong's Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled is one of the more interesting examples. It takes the Qwen3.5-27B base model and fine-tunes it on thousands of reasoning trajectories extracted from Claude 4.6 Opus. The result is a model that exposes its thinking process through <think> tags before delivering a final answer, mimicking the extended thinking behavior that Anthropic built into Claude natively. In 4-bit quantization, the entire model fits in about sixteen gigabytes.

I wanted to know two things. First, whether this kind of distilled reasoning actually works—whether a 27B model can meaningfully replicate the structured thinking of a model orders of magnitude larger. Second, whether the AMD Strix Halo APU, with its unified memory architecture and integrated RDNA 3.5 GPU, could run it at useful speeds. The answer to both turned out to be more nuanced than a simple yes or no.

The Hardware

The machine is the same AMD Ryzen AI MAX+ 395 that has appeared in several previous posts. It is an APU: CPU and GPU on the same die, sharing the same pool of LPDDR5X memory. There is no PCIe bus between the processor and the graphics engine. There is no dedicated VRAM to fill up. The GPU sees roughly 65GB of addressable memory out of the system's 122GB total, which means a 16GB quantized model loads without any of the memory pressure games you play on discrete GPU setups.

This matters for local LLM inference because the bottleneck for most language models is memory bandwidth, not compute. Tokens are generated one at a time, each requiring a full pass through the model's weights. The faster you can stream those weights from memory to the processing units, the faster you generate tokens. The Strix Halo's LPDDR5X provides roughly 120 GB/s of bandwidth to the unified memory pool. A discrete GPU like the RTX 4090 has 1 TB/s of bandwidth to its dedicated VRAM, but the Strix Halo never has to copy weights across a PCIe bus. For models that fit entirely in the GPU's addressable space, the unified architecture eliminates an entire class of overhead.

The system runs Ollama 0.17.6, which wraps llama.cpp and provides model management and an HTTP inference API. ROCm 7.2 handles the GPU compute layer, though Ollama's GGUF inference path is primarily CPU-based with GPU offloading for specific operations. The gfx1151 GPU target is not yet in the mainline PyTorch or llama.cpp kernel prebuilds, so HSA_OVERRIDE_GFX_VERSION=11.0.0 remains necessary to map it to the closest supported target (gfx1100, Navi 31).

The Model

The model's architecture is straightforward: Qwen3.5-27B, a 27 billion parameter transformer, fine-tuned via supervised learning on structured reasoning data. What makes it interesting is the training data. The creator assembled three datasets:

Opus-4.6-Reasoning-3000x-filtered: Three thousand reasoning trajectories extracted from Claude 4.6 Opus, filtered for quality.
claude-4.5-opus-high-reasoning-250x: Two hundred and fifty examples of high-intensity structured reasoning from an earlier Claude version.
Qwen3.5-reasoning-700x: Seven hundred step-by-step problem-solving examples.

The combined training signal teaches the model to produce output in a specific format: a <think> block containing the reasoning process, followed by a clean final answer. This is architecturally similar to what Anthropic does with Claude's extended thinking, except that Claude's thinking is a native capability of the model's training and architecture, while this is a behavior pattern learned through supervised fine-tuning on examples of that behavior.

The distinction matters, and I will come back to it.

The model is distributed in GGUF format, which is the standard for llama.cpp and Ollama. I used the Q4_K_M quantization, which compresses the model's weights from 16-bit floats to 4-bit integers with a mixed precision scheme that preserves more information in attention layers. The file is 15.4GB on disk. The model card reports 29-35 tokens per second on an RTX 3090; I was curious what the Strix Halo would deliver.

Setting It Up

Getting the model running took less than ten minutes. Download the GGUF file from HuggingFace:

mkdir -p ~/models/qwen35-reasoning
curl -L -o ~/models/qwen35-reasoning/model.gguf \
  'https://huggingface.co/Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-GGUF/resolve/main/Qwen3.5-27B.Q4_K_M.gguf'

Note the filename. The HuggingFace repo is named Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-GGUF, but the actual GGUF files inside follow a simpler naming scheme: Qwen3.5-27B.Q4_K_M.gguf. I wasted time trying to guess the full distilled name before checking the API.

Create an Ollama Modelfile that imports the local GGUF and sets inference parameters:

FROM /home/alex/models/qwen35-reasoning/model.gguf

PARAMETER temperature 0.6
PARAMETER top_p 0.95
PARAMETER num_ctx 8192
PARAMETER repeat_penalty 1.2
PARAMETER stop "<|endoftext|>"
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|eot_id|>"

SYSTEM "You are a deep-thinking AI assistant. For complex questions,
use <think>...</think> tags to show your reasoning process before
providing the final answer."

Then:

ollama create qwen35-reasoning -f Modelfile

Ollama copies the GGUF into its own blob store, parses the architecture metadata, and registers it as a runnable model. The whole process takes about a minute on local storage.

The Stop Token Problem

The first run produced correct output followed by infinite repetition. The model answered a calculus question perfectly, then appended "This gives us the final answer:" and repeated the entire solution, over and over, until it hit the context window limit. The previous MarkTechPost article that inspired this experiment did not mention this issue, likely because their test prompts were short enough that the repetition was not obvious.

The fix is explicit stop tokens in the Modelfile. Without them, the model does not know when to stop generating. This is a common issue with GGUF models imported into Ollama without a proper chat template: the model's native end-of-sequence tokens are not being interpreted by the inference engine. Adding <|endoftext|>, <|im_end|>, and <|eot_id|> as stop parameters catches the three most common EOS tokens used by Qwen and Llama-family models.

The repeat_penalty of 1.2 provides a second layer of defense by penalizing the model for reusing recent tokens. This helps but is not sufficient on its own. Without the stop tokens, the model can produce novel-but-meaningless text that avoids exact repetition while still degenerating into nonsense. More on this shortly.

Where It Works: Structured Problems

With the stop tokens in place, the model performs well on structured mathematical and analytical problems. I gave it a calculus question: find the derivative of x³sin(x) using the product rule.

The response was genuinely good. The model opened a <think> block, identified the two component functions, recalled the product rule formula, computed each derivative, and applied the rule. Then it closed the think block and produced a clean, well-formatted answer with LaTeX notation, step-by-step derivation, and a factored final form. The thinking trace was coherent and tracked the actual reasoning process. It was not filler; each line in the trace corresponded to a meaningful step.

Generation speed on the Strix Halo: 10.3 tokens per second. Not fast by cloud standards, but responsive enough for interactive use. You see the thinking appear in real time, which is surprisingly useful: you can watch the model work through the problem and catch errors before it commits to a final answer.

For structured problems—mathematics, code analysis, formal logic—the distilled reasoning is genuinely functional. The model identifies subproblems, works through them sequentially, and arrives at correct answers. The think tags provide transparency into the process that you do not get from a standard instruction-tuned model.

Where It Falls Apart: The River Crossing

I ran the classic wolf-goat-cabbage river crossing puzzle as a comparison test, the same prompt on both the distilled Qwen model and Claude Haiku 4.5 via the Anthropic API.

Claude Haiku returned a perfect, concise seven-step solution in 2.9 seconds. Two hundred and twenty-three tokens. The answer identified the critical insight (bring the goat back on one return trip), laid out the sequence clearly, and stopped.

The Qwen model started well. It correctly identified that the goat must go first, recognized the wolf-goat conflict at the destination, and identified the need to bring the goat back. Then, around step three of the solution, the model began editorializing. "Oh joy what fun times ahead us humans truly enjoy sometimes huh?!" it wrote, mid-solution. Within a few more sentences, the output had degenerated into an unbroken stream-of-consciousness rant that cascaded into a wall of increasingly disconnected words. Not repeated words—the repeat penalty prevented that—but a firehose of unique, semantically null text that continued until it filled the entire 8,192-token context window.

The output was, to use a technical term, unhinged. The model went from a correct partial solution to word salad in about two hundred tokens, and there was no recovery. The stop tokens could not save it because the model was not producing any end-of-sequence markers. It had entered a mode where it was generating fluent English syntax with zero semantic content, which is exactly the kind of failure that stop tokens and repeat penalties cannot catch.

What the Comparison Reveals

The numbers tell the story concisely:

	Claude Haiku 4.5	Qwen3.5-27B (Strix Halo)
Time	2.9 seconds	Hit 8K context limit
Speed	75.9 tok/s	~10 tok/s
Output	223 tokens, correct	Thousands of tokens, degenerated
Cost	$0.0009	Free

But the comparison is not really about speed or cost. It is about the difference between native reasoning and distilled reasoning.

Claude's extended thinking is a capability that emerges from the model's architecture and training at scale. The model has internalized what it means to reason through a problem, including knowing when to stop, when a line of reasoning is unproductive, and when to switch strategies. These are meta-cognitive skills that are extremely difficult to distill.

The Qwen model learned the format of reasoning—the think tags, the step-by-step structure, the pattern of stating subproblems and working through them—from three thousand examples. What it did not learn, and arguably cannot learn from supervised fine-tuning alone, is the judgment about when reasoning is going off the rails. A model that has truly internalized reasoning has implicit quality checks: it recognizes incoherence in its own output and corrects course. A model that has learned to mimic reasoning produces the surface pattern without the underlying self-monitoring.

This is visible in the failure mode. The model did not produce wrong reasoning. It produced no reasoning. It exited the reasoning pattern entirely and entered a generation mode that had nothing to do with the problem. A model with genuine reasoning capability would have recognized the incoherence and either corrected or terminated. The distilled model had no such circuit breaker.

The Economics

The cost comparison deserves its own section because it is often cited as the primary motivation for running local models.

The Claude Haiku API call cost nine-tenths of a cent. If you ran a thousand similar queries per day, you would spend about nine dollars. That is less than the electricity cost of running the Strix Halo for a day under load. The Strix Halo draws roughly 65 watts at idle and 150 watts under GPU inference load. At Minnesota's residential electricity rate of around twelve cents per kilowatt-hour, running inference eight hours a day costs about fourteen cents. But the hardware itself cost north of two thousand dollars. You would need to amortize that over thousands of hours of inference to reach cost parity with the API, and only if you value your debugging time at zero.

The economic case for local inference is not about per-query cost. It is about use cases where you need unlimited queries without metering, where data cannot leave your network, or where you want to experiment with model behavior without worrying about a bill. If you are evaluating a model's failure modes by running hundreds of adversarial prompts—which is exactly what I was doing—the local model is the right tool because you are not optimizing for answer quality. You are optimizing for the freedom to explore.

The Strix Halo as an Inference Platform

Ten tokens per second for a 27B Q4 model is respectable for an APU. It is not competitive with a discrete GPU: an RTX 3090 delivers 29-35 tokens per second on the same model, roughly three times faster. But the Strix Halo was not designed to compete with discrete GPUs on raw throughput.

What it offers instead is capacity. The unified memory pool means you can load models that would not fit on most consumer GPUs. A Q8_0 quantization of this same model would be 28.6GB, which exceeds the VRAM of an RTX 4090 (24GB) but fits comfortably in the Strix Halo's addressable space. You could load a 70B Q4 model (roughly 40GB) without any of the layer-splitting gymnastics required on multi-GPU setups. I have run Llama 3.1 70B Q4 on this machine, and while the generation speed drops to about 4-5 tokens per second, it runs without errors or memory pressure.

For a machine that also serves as a daily desktop, development workstation, and video generation server (it runs LTX-2.3 on the same hardware), the ability to casually load and test a 27B reasoning model without dedicated GPU infrastructure is the actual value proposition. You do not plan a session. You do not allocate resources. You type ollama run qwen35-reasoning and it works.

Lessons for the Blog Post Reader

If you want to replicate this setup, here is what I would emphasize:

The stop tokens are non-negotiable. Without explicit <|endoftext|>, <|im_end|>, and <|eot_id|> stop parameters in your Modelfile, the model will produce infinite output on many prompts. This is not documented in the model card and is not mentioned in the MarkTechPost article that covers this implementation. It is the single most important configuration detail.

The model is good at structured problems and bad at open-ended ones. Mathematics, code analysis, formal logic—anything where the reasoning has a clear structure and a definitive endpoint—works well. Open-ended problems, creative tasks, or anything requiring sustained coherent narrative are risky. The model can degenerate catastrophically and without warning.

A repeat penalty helps but does not solve the fundamental issue. Setting repeat_penalty to 1.2 prevents exact repetition loops but does not prevent the semantic degeneration I observed on the river crossing problem. The model simply produces unique garbage instead of repeated garbage.

Distillation captures form, not judgment. The think tags are real and useful. The step-by-step reasoning format works. What is missing is the implicit self-monitoring that frontier models have: the ability to recognize when their own output has become incoherent and to course-correct. This is probably the hardest thing to distill, because it is not present in the training examples. The examples show successful reasoning. They do not show the model catching and recovering from failed reasoning, because Claude's failed reasoning attempts are filtered out before the training data is assembled.

Where This Goes

The distilled reasoning model is, despite its failure modes, genuinely interesting. The <think> tags provide a form of transparency that standard instruction-tuned models lack. When the model is working correctly—which is most of the time on appropriate tasks—you get a window into the reasoning process that helps you evaluate the answer's quality before you act on it.

The failure mode is also instructive. It demonstrates, concretely, the gap between learning a behavior pattern and internalizing the capability that produces that pattern. Supervised fine-tuning on reasoning trajectories can teach a model to produce reasoning-shaped output, but it cannot, from three thousand examples, teach the model to actually reason in the way the source model does. That requires either far more training data, a different training methodology (reinforcement learning from reasoning feedback, perhaps), or simply a larger model with more capacity to internalize the underlying patterns.

For now, the practical advice is: use these models for what they are good at, know their failure modes, and do not trust the output on open-ended problems without reading the thinking trace. The trace is the feature. If the trace is coherent, the answer is probably good. If the trace starts to wander, stop reading and retry.

The model runs on my desk, generates ten tokens per second, costs nothing per query, and shows its work. For a sixteen-gigabyte download and ten minutes of setup time, that is a reasonable deal—as long as you know what you are buying.

The Feedback Loop That Jevons Couldn't Name

A.C. Jokela — Fri, 27 Mar 2026 13:00:00 GMT

In 1865, William Stanley Jevons published The Coal Question and identified a paradox: James Watt's more efficient steam engine didn't reduce coal consumption. It increased it. The efficiency gain made coal-powered industry more profitable, which drove more investment, which consumed more coal. The per-unit savings were overwhelmed by the expansion in total units demanded.

In 1948, Norbert Wiener published Cybernetics: Or Control and Communication in the Animal and the Machine and described a mechanism: systems that feed their outputs back into their inputs will either stabilize (negative feedback) or accelerate (positive feedback). A thermostat is negative feedback: the output (heat) reduces the input (the gap between current and target temperature). A microphone pointed at a speaker is positive feedback: the output (sound) amplifies the input (sound), and the system screams.

Jevons saw what happened. Wiener explained why.

They never met. Jevons died in 1882, twelve years before Wiener was born. Their fields barely overlapped. Jevons was an economist and logician working in Manchester. Wiener was a mathematician and engineer at MIT. Neither cited the other. Neither would have had reason to. But they were describing the same phenomenon from different sides: Jevons from economics, Wiener from control theory. Jevons identified the paradox. Wiener provided the mechanism. Together, they explain something about AI that the current conversation consistently misses: the reason demand expands when cognitive tools get cheaper isn't economic irrationality. It's positive feedback. It's the system doing exactly what feedback systems do.

Wiener's Machines

Wiener was not an abstract theorist. He built anti-aircraft fire control systems during World War II, predicting the future position of enemy aircraft based on their observed trajectories. The mathematical problem was filtering signal from noise in real-time feedback data, and the solution required treating the human pilot as a component in a mechanical system: a system that could be modeled, predicted, and countered.

This experience shaped everything he wrote afterward. In Cybernetics, Wiener argued that communication and control were fundamentally the same problem, whether the system involved nerves, wires, or social institutions. A factory is a feedback system. An economy is a feedback system. A conversation is a feedback system. The mathematics of regulation and stability apply to all of them.

In 1950, he published The Human Use of Human Beings, a book aimed at general readers. Its central argument: automation would transform society not by replacing humans but by changing the feedback loops that humans operate within. The automated factory doesn't just make products without workers. It creates a system where the speed of production is no longer limited by human labor, which means the system's dynamics shift to whatever the next bottleneck happens to be.

Wiener's most famous warning was blunt: "The automatic machine is the precise economic equivalent of slave labor. Any labor which competes with slave labor must accept the economic conditions of slave labor." He predicted that automation would produce unemployment that would make the Great Depression "seem a pleasant joke." He wrote this in 1950, when computers filled rooms and could barely calculate ballistic tables.

The Jevons Mechanism

Jevons didn't have the vocabulary of cybernetics. He described his paradox in economic terms: efficiency improvements reduce per-unit cost, lower cost increases demand, increased demand outweighs the efficiency gain, total consumption rises. He was observing a positive feedback loop, but he described it as a paradox because the economic framework he was working within predicted the opposite. If coal becomes more efficient, you should need less of it. The loop that amplifies demand was invisible in the model.

Wiener's framework makes the loop visible. Here's the cybernetic translation of Jevons:

A system component becomes more efficient (Watt's steam engine, a cheaper semiconductor, an AI model).
The efficiency reduces the cost of the system's output.
Lower cost makes new applications viable that were previously too expensive.
New applications create new demand for the now-cheaper component.
The new demand feeds back into step 1 as pressure for more efficiency, more production, more investment.
The loop accelerates.

This is a positive feedback loop. The output (cheaper goods, more applications) amplifies the input (demand for the efficient component). There is no negative feedback mechanism to stabilize the system. The loop runs until it hits an external constraint: a physical limit on the resource, a regulatory intervention, or the saturation of all possible demand.

Jevons observed steps 1 through 4 with coal. He didn't have the mathematical framework to describe steps 5 and 6 as a feedback loop. Wiener had the framework but was focused on machines and automation, not on resource economics. The connection between them is that Jevons Paradox is a specific instance of positive feedback in economic systems, and positive feedback is the phenomenon Wiener spent his career analyzing.

The AI Loop

I've been writing about Jevons Paradox and AI for months. The argument: AI makes cognitive output cheaper, demand for cognitive output expands beyond the efficiency gain, and the expansion concentrates pressure on the one input that can't scale: human judgment. The Vampire piece described the human cost. The Excavator piece described the software quality cost. The split piece described how the craft concentrates in the judgment layer.

What I didn't do was explain the mechanism. Why does demand expand when a cognitive input gets cheaper? Why doesn't the system reach equilibrium at lower total consumption, the way classical economics predicts? What force drives the expansion?

Wiener's answer: positive feedback.

Here's the AI loop, stated in cybernetic terms:

AI makes code generation cheaper (the efficiency gain).
Cheaper code generation makes new software projects viable (the demand expansion).
New projects produce software that requires review, testing, debugging, and maintenance (the output).
Review and debugging create demand for more AI assistance (the feedback).
The loop accelerates: more projects, more software, more review, more AI, more projects.

At no point does the loop include a mechanism for slowing down. There is no thermostat. The "temperature" (volume of software in production) rises without limit until it hits an external constraint.

I can see this loop operating in my own work. I built DirtScout, a full-stack land acquisition platform, in a series of conversations with Claude Code. 29,000 lines of code across Python, TypeScript, and infrastructure-as-code. The project would have taken months to type by hand. With AI, I built it in days. But building it in days meant I immediately started adding features: soil analysis, environmental assessments, auction tracking, deal pipeline management, offer letter generation. Each feature was a conversation. Each conversation produced code that needed to be reviewed, tested, and maintained. The faster I built, the more I wanted to build, and the more I built, the more review work accumulated. The loop ran. I didn't notice it running until the maintenance surface area was larger than anything I'd built before.

That's Wiener's loop at the individual level. At the organizational level, the same dynamic plays out with more people and higher stakes. Every developer using AI-assisted tooling ships more code, which creates more surface area for bugs and security vulnerabilities, which creates more demand for review, which creates more demand for AI-assisted review tooling, which ships more code.

The external constraint, as I've argued in previous pieces, is human judgment. The three-to-four-hour ceiling on deep work is biological. It doesn't expand because the feedback loop demands more of it. It's a fixed resource being consumed by an accelerating process. In Wiener's terms, the human component in the feedback loop is a bottleneck with a fixed maximum throughput. The system can't route around it (the judgment is necessary) and can't expand it (the biology doesn't scale). So the system does the only thing a positive feedback loop can do when it hits a fixed constraint: it overloads the constraint.

That's burnout. That's what Steve Yegge described in "The AI Vampire." Wiener would have recognized it instantly. The human component in an accelerating feedback loop reaches its throughput limit and degrades. The system doesn't stop. The system doesn't care. The system is a feedback loop, and feedback loops don't have preferences about their components.

Wiener's Warning, Updated

Wiener warned that automation would make human labor economically equivalent to slave labor. He was wrong about the specifics (manufacturing employment declined but didn't collapse) but right about the dynamic. The feedback loop he described (automation reduces labor cost, which drives more automation, which further reduces labor cost) played out exactly as predicted. It just played out over decades instead of years, and the economy adapted by shifting labor to sectors that weren't yet automated.

The AI version of this warning is different in a way that matters. Wiener's automation loop operated on physical labor. Muscle has substitutes: machines. When the feedback loop overloaded the human muscle component, the system routed around it with hydraulics, robotics, and assembly lines. The human moved to cognitive work, where machines couldn't follow.

AI's feedback loop operates on cognitive labor. Judgment does not have substitutes. When the feedback loop overloads the human judgment component, the system can't route around it the way manufacturing routed around physical labor. There is no higher-order activity to retreat to. Judgment is the top of the stack. The feedback loop either overloads it (burnout) or degrades it (review quality drops, software slop accumulates, the Excavator scenario plays out).

Wiener saw this possibility in the abstract. In The Human Use of Human Beings, he wrote: "The world of the future will be an even more demanding struggle against the limitations of our intelligence, not a comfortable hammock in which we can lie down to be waited upon by our robot slaves." He was pushing back against the utopian narrative of his own era: the idea that automation would create leisure. His counterclaim was that automation would shift the struggle to a harder domain. He was right, and the harder domain turned out to be exactly the one AI is now pressuring: the limits of human cognition.

The Speed Problem

There's a dimension of the AI feedback loop that Wiener's industrial-era examples didn't anticipate: speed.

Wiener's factory automation loop ran at the speed of manufacturing. It took years to design a new factory, months to retool an assembly line, weeks to train workers on new processes. The feedback loop was real, but it operated on a timescale that allowed human institutions (unions, regulations, education systems) to adapt. Walter Reuther, the president of the United Auto Workers, received Wiener's 1949 letter warning about automation and had years to develop a response. The loop was slow enough for governance.

The AI feedback loop runs at the speed of software. An operations manager can go from "I have an idea" to "it's in production" in an afternoon. A developer can ship ten features in the time it used to take to ship one. The loop cycles in hours, not years. Human institutions that adapted to the manufacturing automation loop over decades don't have decades to adapt to the AI loop. They have the time between one deployment and the next.

This is the Jevons Paradox running at software speed. Coal consumption took decades to double after Watt's engine. Computing demand took years to double after each semiconductor generation. AI-assisted software production can double in months. The feedback loop is the same. The clock rate is different. And the human component's clock rate (the biological ceiling on judgment) hasn't changed at all.

The supply side is contracting at the same time. After sixteen consecutive years of growth, undergraduate computer science enrollment turned negative in 2025. The Computing Research Association found that 62% of computing departments reported declining enrollment for 2025-26. Students and their parents are reading the headlines about AI replacing developers and steering toward fields they perceive as more durable. The feedback loop produces an ironic secondary effect: the fear of automation reduces the supply of the human component that the accelerating system needs most. The loop runs faster. The pipeline of people qualified to govern it narrows. Wiener's warning about building governance structures before the loop overloads becomes more urgent as the pool of people who could build those structures shrinks.

Wiener and Heidegger

Wiener and Heidegger never engaged with each other's work, as far as I know. They were writing at the same time (late 1940s, early 1950s), about the same phenomenon (technology reshaping human life), and they arrived at complementary conclusions from completely different starting points.

Heidegger, as I wrote in the Enframing piece, argued that technology changes how we see the world. Everything becomes standing reserve: raw material to be ordered and consumed. The river becomes a power source. The specification becomes code. The transformation is ontological: it changes what things are, not just what we do with them.

Wiener argued that technology changes the dynamics of the systems we operate within. Feedback loops accelerate. Bottlenecks shift. Components that were adequate at one cycle speed become inadequate at a faster one. The transformation is mechanical: it changes the forces acting on us, not necessarily how we understand them.

The two frameworks aren't contradictory. They're describing different aspects of the same process. Heidegger explains why we treat the Zilog manual as raw material for code generation (Enframing). Wiener explains what happens when we do it at scale (positive feedback, demand expansion, bottleneck overload). Jevons measured the economic result (total consumption rises despite efficiency gains).

There's a useful way to layer them. Heidegger describes the precondition: technology must first transform how we see the world (specifications become standing reserve) before the feedback loop can operate. You can't accelerate production of something you don't yet see as producible. Enframing opens the door. Wiener's loop walks through it. Jevons counts what's on the other side.

The sequence matters for AI. First, we began seeing cognitive tasks as automatable (Heidegger's shift in perception). Then, AI tools made the automation practical and cheap (Wiener's efficiency gain). Then, demand for cognitive output expanded beyond what anyone predicted (Jevons' paradox). Each step enables the next. The feedback loop couldn't run until the Enframing was in place, and the economic expansion couldn't happen until the loop was running.

Three disciplines. One phenomenon. The feedback loop that Jevons couldn't name, Wiener formalized, and Heidegger diagnosed as a transformation in our relationship to the world.

The Missing Thermostat

Every stable system has negative feedback. A thermostat, a voltage regulator, a population predator-prey cycle: something measures the output and adjusts the input to keep the system within bounds. Positive feedback without negative feedback is, by definition, unstable. The microphone screams until someone unplugs it.

The AI feedback loop currently has no thermostat. There is no mechanism that measures the volume of unreviewed software in production and slows the rate of production accordingly. There is no mechanism that measures developer burnout and reduces the demand for cognitive output. There is no mechanism that measures the ratio of AI-generated code to human-reviewed code and raises an alarm when it crosses a threshold.

Wiener would argue that this is the actual problem. Not AI itself (a tool, a component, an efficiency gain), but the absence of negative feedback in the system that AI accelerates. His entire career was about designing feedback systems that stabilize rather than explode. His warning about automation wasn't "don't build machines." It was "build the governance structures that keep the feedback loop from overloading its human components."

In 1949, Wiener wrote a letter to Walter Reuther, president of the United Auto Workers union, warning him about the coming wave of industrial automation. He didn't tell Reuther to smash the machines. He told him to prepare the workforce and the institutions for a system that would accelerate beyond their current capacity to manage. The letter went largely unheeded.

We're in the same position now. The feedback loop is running. The human component is approaching its throughput limit. The thermostat doesn't exist. Someone needs to build it, and the people best positioned to do so are the ones inside the loop: the developers and decision-makers who can see the acceleration because they're experiencing it.

Wiener died in Stockholm in 1964 at the age of sixty-nine, two decades before the personal computer and six decades before large language models. He never saw the system he described reach the scale it's reaching now. But the mathematics he wrote down in 1948 describe it precisely. Positive feedback without negative feedback is unstable. The system will find its constraint and overload it. The only question is whether we build the thermostat before or after the overload.

What makes Wiener worth reading today isn't his specific predictions (some were right, some were wrong, the timeline was consistently too compressed). It's his framework. He understood that technological change is not a series of discrete events but a system of coupled feedback loops. Each efficiency gain changes the dynamics of the system it operates within. Each change in dynamics creates pressure on whatever component is now the bottleneck. And each bottleneck, when overloaded, produces consequences that feed back into the system and accelerate the next cycle.

That framework applies to coal in 1865, to factory automation in 1950, and to AI-assisted cognitive work in 2026. The specific resources change. The specific bottlenecks change. The feedback dynamics don't.

John von Neumann, Wiener's contemporary and one of the minds his work most influenced, once said that young mathematicians should not worry about whether their work would be useful because "truth is much too complicated to allow anything but approximations." Wiener's approximation of the feedback dynamics of technological change was good enough that it still describes the system seventy-eight years after he formalized it. Whether it's good enough to help us build the thermostat before we need it is the question his work leaves us with.

What a Thermostat Might Look Like

Wiener didn't just diagnose problems. He designed solutions. His entire field was about building systems that regulate themselves. If he were alive today, he'd be asking: what does negative feedback look like in an AI-accelerated software economy?

Some possibilities:

Mandatory review ratios. For every N lines of AI-generated code deployed to production, M lines must be reviewed by a qualified human. The ratio creates a coupling between the production rate and the review rate, forcing the system to slow down when review capacity is saturated. This is a thermostat: the output (deployed code) is measured against a constraint (review capacity), and the input (generation rate) is throttled accordingly.

Liability assignment. If AI-generated code causes a data breach or financial loss, who pays? Currently, nobody in particular. Assigning liability to the person who deployed the code (not the person who prompted the AI) creates negative feedback: the cost of deployment failure feeds back into the decision to deploy, making people more cautious about shipping unreviewed code. Insurance markets would price this risk and create their own feedback mechanisms.

Institutional adaptation. This is what Wiener actually advocated. Not technical solutions but organizational ones. He told Walter Reuther to prepare the workforce for automation. The equivalent today: companies need to build review capacity at the same rate they build production capacity. Every developer who ships AI-generated code needs a corresponding increase in testing, security review, and architectural oversight. The organizations that treat AI as free productivity without investing in review are the ones that will hit the overload first.

Cultural awareness. Tristan Harris and the Center for Humane Technology have been arguing since 2023 that AI is being deployed faster than any technology in history under maximum incentives to cut corners on safety. Harris makes a distinction that Wiener would have appreciated: the difference between the "possible" (AI's theoretical benefits) and the "probable" (what actually happens given current incentive structures). The probable outcome, without intervention, is that companies race toward capability because the competitive feedback loop punishes restraint. Harris's proposed response is building global consensus that the current trajectory is unacceptable, the way the nuclear test ban and the Montreal Protocol established consensus before those feedback loops ran to their conclusions. In Wiener's terms, Harris is trying to build the thermostat at the cultural level: changing the system's objective function so that it optimizes for something other than pure output volume.

None of these exist at scale today. The thermostat is unbuilt. The loop runs open.

Jevons told us what happens. Wiener told us why. The question that remains is whether anyone is building the feedback mechanism that prevents the system from screaming.

Building DirtScout: A Land Acquisition Platform with Claude Code

A.C. Jokela — Thu, 26 Mar 2026 01:00:00 GMT

Nine years ago, I built something similar.

It was 2017, St. Louis County, Minnesota. I wanted to find raw undeveloped land from the county's delinquent tax rolls. The county had an ArcGIS service, but the APIs were primitive compared to what they offer now. I stood up a PostgreSQL database with PostGIS extensions, wrote Ruby scripts to scrape parcel data from the county's map server, geocoded addresses, and built a Ruby on Rails frontend to browse the results. The whole thing lived on a single VPS. It worked for one county. The data model was rigid, the scraping was fragile, and every time St. Louis County changed their GIS service, something broke.

That project died the way side projects do: I got what I needed from it and moved on.

In March 2026, I came back to the idea. The landscape had changed. ArcGIS REST APIs are now standardized and reliable. Wisconsin publishes a statewide parcel dataset covering all 72 counties through a single endpoint. Minnesota counties expose delinquent tax data through queryable feature services. AWS Lambda and DynamoDB mean I don't need to manage a database server. And I had a tool that didn't exist in 2017: Claude Code.

DirtScout is the result. It's a full-stack land acquisition platform at dirtscout.land that searches delinquent tax parcels across 21 Minnesota counties and browses raw land across 72 Wisconsin counties. It has AI-powered investment analysis, environmental and soil assessments, a deal pipeline with offer letter generation, tax forfeit auction tracking, and automated monitoring with email alerts. The codebase is about 29,000 lines across Python, TypeScript, and infrastructure-as-code.

I built it with Claude Code. Not "Claude Code assisted me" or "Claude Code helped with the boilerplate." Claude Code wrote the code. I directed the architecture, made the decisions, and did the debugging when things broke. But the actual lines of code came from conversations, not from me typing in an editor.

The Architecture

The 2017 version was PostgreSQL + PostGIS + Ruby on Rails on a single server. The 2026 version:

Frontend: Next.js 16, static export, Tailwind CSS, react-leaflet for maps. Hosted on S3 behind CloudFront. The entire frontend is pre-rendered HTML and JavaScript; there's no server-side rendering. CloudFront serves it from edge locations. A URL rewrite function handles dynamic routes for deal detail pages and shared parcel links.

Backend: Python FastAPI running on a single AWS Lambda function behind API Gateway. Mangum adapts the ASGI app to Lambda's event format. Every API request hits the same Lambda, which cold-starts in about 3.5 seconds and handles subsequent requests in under a second. The function has 512MB of memory and a 5-minute timeout.

Data: Two DynamoDB tables. The main table stores user data, flagged parcels, deals, preferences, notes, attachments, saved searches, tax list imports, and auction tracking. The cache table stores land cover analysis, environmental data, soil analysis, and geometry with TTLs. No PostgreSQL. No PostGIS. No database server to manage.

Infrastructure: AWS CDK in Python. One cdk deploy command creates the Lambda, API Gateway, DynamoDB tables, S3 buckets, SQS queues, EventBridge schedules, Route 53 records, CloudFront distributions, and ACM certificates. The entire infrastructure is version-controlled and reproducible.

On-premises worker: A service running on a local AMD Strix Halo machine (Ryzen AI MAX+ 395, 128GB RAM) processes delinquent tax list PDFs using pdfplumber for text extraction and a local Qwen3 32B model via Ollama for structured data extraction. It polls an SQS queue for jobs.

This is a fundamentally different architecture than what I could have built in 2017. No servers to patch aside from the Strix Halo. No database to back up. No PostGIS extensions to compile. The Lambda handles the compute, DynamoDB handles the storage, and the on-prem machine handles the jobs that need a real browser or a local LLM.

What Claude Code Actually Did

I want to be specific about this because the "AI-assisted development" conversation is usually vague. People say "I used AI to help me code" and it could mean anything from autocomplete suggestions to full application generation. Here's what actually happened.

I started with a Rust TUI. The original project was a terminal application that queried a handful of Minnesota county ArcGIS services and displayed delinquent parcels in a text interface. It had county configurations, a query client, land cover analysis via USGS NLCD, and a flagging system. Claude Code built this from my descriptions of what I wanted: "query this ArcGIS service for parcels where the delinquent flag is set, filter by acreage and land use, show me the results in a table with navigation."

Then I decided to make it a web app. I described the architecture I wanted: FastAPI on Lambda, Next.js on S3, DynamoDB for storage. Claude Code ported the Rust query logic to Python, built the FastAPI routes, created the React components, wrote the CDK infrastructure, and handled the deployment. Each feature was a conversation: "add Google OAuth," "add a deal pipeline with stages - make it look like Kanban," "generate offer letter PDFs," "add an AI investment summary using the Claude API."

The codebase grew to 29,000 lines across 113 files in the initial commit. Later sessions added another 60 files and 5,000 lines for Wisconsin support, soil analysis, tax list imports, auction tracking, spatial search, and saved searches.

I didn't write these lines. I directed them. There's a difference, and it matters.

When I say "directed," I mean I made every architectural decision. I chose DynamoDB over PostgreSQL because I didn't want to manage a database. I chose Lambda over ECS because I didn't want to manage containers. I chose static export over SSR because I didn't want to manage a Node.js server. I chose to use a local LLM for PDF parsing instead of Claude API because the parsing is structured data extraction that doesn't need frontier model quality.

Claude Code implemented these decisions. When something broke, I described the symptom and Claude Code diagnosed the cause. When I wanted a new feature, I described the behavior and Claude Code wrote the code. The feedback loop was: describe what I want, review what I get, deploy, test, describe what's wrong, iterate.

Some things broke in interesting ways. DynamoDB doesn't accept Python floats; you have to convert everything to Decimal. The county field maps are reverse-keyed from what you'd expect (ArcGIS field names are the keys, common names are the values). Google OAuth redirect URIs need a trailing slash. CloudFront caches aggressively and you have to invalidate after every deploy. The Census TIGER API for county boundaries is painfully slow, so we downloaded the GeoJSON once and serve it as a static file. Each of these was discovered in production and fixed in conversation.

The Data Sources

The interesting part of DirtScout isn't the web framework. It's the data integration.

Minnesota parcel data comes from 11 different county ArcGIS REST services, each with its own field names, query syntax, and data quality. St. Louis County has DELINQUENT_TAX_FLAG and BAL_DUE. Aitkin has DELINQUENT_FLAG (text: "YES"/"NO") and BALDUE. Hennepin stores acreage in square feet (divide by 43,560). Goodhue stores acreage as a string that requires CAST in the SQL WHERE clause. Each county is a separate configuration with field mappings, WHERE clause templates, and normalization logic.

Minnesota tax lists come from 15 county PDFs and Excel files. Itasca County publishes an Excel file updated monthly. The rest publish PDF legal notices. The PDFs are processed by either the Claude API (Haiku model, cheapest tier) or a local Qwen3 32B running on the Strix Halo machine. The AI extracts parcel IDs, owner names, delinquent amounts, and addresses from the unstructured PDF text and returns structured JSON.

Wisconsin parcel data comes from a single statewide ArcGIS feature service maintained by the State Cartographer's Office. One endpoint, all 72 counties, standardized fields. Owner names, mailing addresses, assessed values, acreage, property class. No delinquent tax data in the GIS, but we supplement with 9 county-level PDF lists of tax-delinquent and tax-forfeited properties.

Environmental analysis layers: FEMA NFHL for flood zones, NWI for wetlands, NHD for water bodies, Minnesota DNR County Well Index for well data, MPCA for contamination sites. Each is a separate ArcGIS REST service query using the parcel's centroid.

Soil analysis comes from the USDA Soil Data Access REST API (SSURGO). A SQL query with the parcel's centroid returns soil components, drainage class, hydric rating, slope, farmland classification, and capability class. We compute a "buildability" score from these factors.

Land cover comes from the USGS MRLC WMS service, querying the NLCD 2021 Land Cover layer. We sample the parcel area and return a breakdown by cover type: forest, agriculture, water, wetlands, developed, barren.

Each of these integrations was built in conversation with Claude Code. "Add flood zone analysis using FEMA's service." "The NWI wetlands query needs table-prefixed field names." "The SSURGO soil query needs a WKT point geometry."

What Changed Since 2017

The PostgreSQL + PostGIS + Ruby on Rails stack I used nine years ago was the right choice for 2017. PostGIS let me do spatial queries locally. I had to store the parcel data because the ArcGIS services weren't reliable enough to query in real time. Rails rendered server-side because that's what Rails did.

None of that is necessary anymore. The ArcGIS services are fast and reliable enough to query live. DynamoDB handles the persistence without a schema to manage. Lambda eliminates server management. Static export means the frontend is just files on S3.

There's a personal angle here too. In graduate school, I spent an entire semester manually developing land cover classifications for a final project — hand-labeling training data, running supervised classification algorithms, validating results against ground truth. It was weeks of work for one study area. For DirtScout, I told Claude Code "add a buildability score based on soil drainage, hydric percentage, slope, and capability class" and had a working assessment in minutes. The SSURGO soil data query, the scoring logic, the frontend panel with color-coded ratings — all from a single conversation. The knowledge that took a semester to develop is now a commodity you can describe and deploy.

But the bigger change is the development process. In 2017, I wrote every line of Ruby and SQL by hand. I designed the PostGIS schema, wrote the scraping scripts, built the Rails views, configured the Nginx proxy, set up the SSL certificates, and wrote the systemd service files. It took months of evenings and weekends for a single-county tool.

In 2026, I built a two-state, multi-service platform with AI analysis, auction tracking, deal management, and offer letter generation in a series of conversations over a few days. The code isn't hand-crafted. I'm not interested in hand-crafted code when that's not the point. The point is finding undervalued rural land from delinquent tax records and making offers to motivated sellers. The code is the means. Claude Code made the means faster.

The On-Prem Angle

A tax list import worker runs on a Bosgame M5 mini PC in my basement.

The worker exists because I didn't want to pay for Claude API calls to parse 24 county PDFs every week. The AMD Strix Halo has 128GB of RAM and runs Qwen3 32B through Ollama. The worker downloads each PDF, extracts text with pdfplumber (a Python library that does the PDF-to-text conversion locally, no model needed), then sends the extracted text to the local Ollama instance for structured JSON extraction. Each 2-page chunk takes 5-7 minutes on the 32B model. It's slower than a cloud API. It's also free.

The worker is a systemd service that starts on boot and polls an SQS queue continuously. A weekly systemd timer enqueues an "import all" message every Monday morning.

This is the economics of owning your own inference in practice. The frontier model handles the quality-sensitive work (AI investment analysis, parcel chat). The local model handles the batch extraction work. The split happens naturally based on the task requirements.

What It Does Now

The production site at dirtscout.land:

Searches delinquent tax parcels across 21 Minnesota counties (8 Tier 1 with full ArcGIS data, 2 Tier 2 with partial data, 2 Tier 3 with minimal data, 9 Tier 4 with imported tax list data)
Browses raw land parcels across all 72 Wisconsin counties via the statewide parcel service
Scores each MN parcel on a 0-100 scale (grades A through F) based on financial opportunity, road access, environmental factors, and land character
Generates AI investment summaries using Claude Sonnet with full context: parcel data, land cover, environmental analysis, soil data, owner's other properties, and attached documents
Tracks deals through a pipeline (prospecting, offer sent, negotiating, under contract, closed, dead) with offer letter PDF generation using three templates
Monitors for new delinquent parcels daily via EventBridge-triggered Lambda scans, with email alerts
Tracks tax forfeit auction dates across 8 Minnesota counties, with a floating widget showing upcoming auctions
Imports delinquent tax lists from 15 MN and 9 WI county PDFs/Excel files weekly
Provides environmental analysis (flood zones, wetlands, water bodies, wells, contamination), land cover classification (NLCD 2021), and soil analysis (SSURGO) for each parcel
Shows parcel boundaries on satellite imagery, with an interactive explore map that loads parcel shapes at high zoom levels
Manages parcel notes, file attachments (via S3 presigned URLs), shareable parcel links, and saved searches

What I'd Do Differently

I'd add geometry caching earlier. Every map view that shows parcel boundaries makes a live ArcGIS query with returnGeometry=true, which is slower than querying attributes only. Caching the geometry in DynamoDB with a TTL would make the explore map significantly faster.

I'd standardize the county configurations into a more declarative format. Right now each county is a Python dataclass with hand-tuned field mappings. A JSON configuration file that Claude Code could modify more easily would reduce the friction of adding new counties.

I'd separate the frontend into a proper monorepo with shared types between the API client and the backend models. The current setup has TypeScript interfaces in the frontend that mirror Pydantic models in the backend, and they get out of sync when fields are added.

But these are optimizations, not regrets. The system works. It finds land. It makes the research process faster. And it was built in conversations, not in sprints.

Enframing the Code

A.C. Jokela — Sun, 22 Mar 2026 13:00:00 GMT

I asked Claude to build a Z80 emulator. The constraint was explicit: no reference to existing emulator source code. The inputs were the Zilog Z80 CPU User Manual, an architectural plan I wrote, and the test ROMs to validate against. Claude produced 1,300 lines of C covering every official Z80 instruction, undocumented flag behaviors, ACIA serial emulation, and CP/M support. It passed 117 unit tests. It boots CP/M and runs programs.

The emulator works. The question is what it means that it exists.

The Clean Room That Wasn't

"Clean room" is a legal term borrowed from semiconductor fabrication. In software, it describes a methodology where developers build from specifications and documentation without ever examining existing implementations. The purpose is to produce code that is legally independent of prior art. If you've never seen the original code, you can't have copied it.

The clean-room process was designed for human cognition. A developer reads a specification, forms a mental model, and writes code that implements the behavior the specification describes. The legal fiction is that the developer's mental model is informed solely by the specification, not by any existing implementation. In practice, developers have seen other implementations, read blog posts, studied textbook examples. The clean room is a discipline, not a guarantee: you follow the process, document that you followed it, and hope that's sufficient if someone challenges you.

When Claude writes a Z80 emulator from the Zilog manual, the clean-room concept doesn't dissolve because the AI is better at following the rules. It dissolves because the framework doesn't apply. Claude's training data includes dozens of Z80 emulators. The model has seen MAME's Z80 core, it has seen Fuse, it has seen whatever antirez published. The question of whether a specific output is "derived from" a specific input is unanswerable, because the model's internal state isn't decomposable into "I learned this from source A" and "I learned this from source B." The provenance that clean-room law requires you to demonstrate doesn't exist in a form that can be demonstrated.

But here's what's interesting: the emulator I directed Claude to produce is not a copy of any specific emulator. The architecture is mine. The bit-field decoding strategy (x/y/z/p/q decomposition of opcode bytes) was specified in my architectural plan. The test suite structure, the ACIA emulation interface, the system emulator's callback design: all specified by me and implemented by Claude from those specifications plus the Zilog manual. The output is an original assembly of knowledge. It's also an output of a system that has seen the source code it was told not to reference.

The law has no category for this. It's not a copy. It's not independent. It's something else.

The Language That Doesn't Exist

The Z80 case is complicated by the fact that prior implementations exist. Somebody could, in theory, diff my emulator against MAME's and look for structural similarities. (They won't find meaningful ones, because the architecture is different, but the argument could be made.) The more interesting case eliminates this possibility entirely.

Lattice is a programming language I designed. It has a novel feature called the phase system: mutability is not a static attribute but a runtime property that values transition through, like matter moving between liquid and solid. You declare a value in flux (mutable), freeze it to fix (immutable), thaw it back if needed. The language has forge blocks for controlled mutation zones. None of this exists in any other language.

Claude writes Lattice code. It writes it well. It produces correct programs using the phase system, the concurrency primitives, and the bytecode VM's 100-opcode instruction set. It does this despite the fact that Lattice does not appear in its training data. The language was designed after Claude's knowledge cutoff. There is no Lattice source code on GitHub, no Stack Overflow answers, no blog posts (other than mine) explaining the syntax.

How does Claude write Lattice? Because Lattice's syntax looks like Rust. The curly braces, the type annotations, the pattern matching: Claude recognizes the structural similarity and maps its understanding of Rust-like languages onto the Lattice grammar. The phase-specific keywords (flux, fix, freeze, thaw, forge) are new, but they appear in contexts that are syntactically familiar. Claude doesn't need to have seen Lattice before. It needs to have seen languages that smell similar.

This is a fundamentally different kind of creation than what copyright law contemplates. Claude didn't copy Lattice code (none exists to copy). It didn't copy Rust code (Lattice isn't Rust). It transformed a grammar specification and a set of examples into working programs in a language that has no prior art. The specification became the implementation without passing through any intermediate step that could be called "copying."

Heidegger Saw This Coming

In 1954, Martin Heidegger published The Question Concerning Technology. His central argument: modern technology is not just a set of tools. It is a way of seeing the world. He called this way of seeing Enframing (Gestell): the tendency of modern technology to reveal everything as standing reserve (Bestand), raw material ordered into availability.

The example Heidegger used was a hydroelectric dam on the Rhine. The river is no longer a river in the way a bridge reveals it (something to cross, something to contemplate, something with its own presence). The dam reveals the river as a power source. The water is standing reserve: ordered, measured, extracted. The river hasn't changed physically. What changed is how technology frames it.

This is exactly what happens when Claude reads the Zilog Z80 CPU User Manual.

The manual is a specification: 332 pages of timing diagrams, instruction tables, register descriptions, and pin assignments. When a human developer reads it, the manual is a guide. The developer forms an understanding, makes design choices, writes code that reflects their interpretation of the specification. The manual and the implementation are connected through the developer's comprehension. The developer is present in the code in a way that matters both legally and philosophically.

When Claude reads the same manual, the specification becomes standing reserve. The timing diagrams are not studied; they are consumed. The instruction tables are not interpreted; they are transformed. The manual is raw material, ordered directly into code, the way the dam orders the river into electricity. There is no intermediate step of "understanding" in the human sense. There is a transformation from one representation (specification) to another (implementation), and the transformation is mechanical in a way that human interpretation is not.

This is what Heidegger meant by Enframing. Technology doesn't just use resources; it changes what counts as a resource. The Zilog manual was written as a reference for engineers. Enframing reveals it as raw material for code generation. The specification was always latently an implementation; the AI just makes the transformation explicit.

What Copyright Was Protecting

Copyright law protects "original works of authorship fixed in a tangible medium of expression." The key word is "original." A Z80 emulator is copyrightable because the programmer made creative choices in expressing the specification as code. Two programmers given the same Zilog manual will produce different emulators: different variable names, different control flow structures, different optimization strategies, different architectural decisions. The specification constrains the behavior. The expression is where the creativity lives.

This framework assumes that the gap between specification and implementation is where human creativity operates. The specification says "the ADD instruction sets the zero flag if the result is zero." A hundred programmers will write a hundred slightly different implementations of this behavior. Each is an original expression. Each is copyrightable.

What happens when the gap closes? When the transformation from specification to implementation becomes mechanical, when there is no creative gap for originality to occupy, what is left to protect?

Claude's Z80 emulator makes specific structural choices: the x/y/z/p/q bit-field decomposition, the callback-based system bus interface, the T-state tracking architecture. These choices came from my architectural plan, not from Claude's autonomous creativity. I specified the structure; Claude filled it in from the Zilog manual. The "creative choices" that copyright relies on were mine (the architecture) and the specification's (the behavior). Claude's contribution was the transformation between the two, and that transformation is closer to compilation than to authorship.

Lattice pushes this further. Claude writes programs in a language with no training data, from a grammar specification and examples I provided. The output is correct Lattice code. But who is the author? I designed the language. Claude learned it from my spec. The programs it produces are implementations of tasks I described. At no point did Claude exercise the kind of independent creative judgment that copyright assumes. It transformed a task description into code in a grammar it learned from me. The entire chain from specification to implementation is mechanical, even though the output looks exactly like something a human programmer would write.

The Dissolution

Clean-room reverse engineering was a legal ritual designed to prove that a human developer's mental model was not contaminated by existing code. The ritual made sense when the concern was human memory: a developer who has read source code might unconsciously reproduce it.

AI makes the ritual meaningless in two ways.

First, provenance is undemonstrable. You cannot prove that Claude's output is or isn't derived from a specific piece of training data, because the model's internal representations don't maintain source attribution. The clean-room question ("did the developer see the original code?") has no answerable equivalent for an LLM. The model has seen everything in its training data simultaneously. It cannot unsee selectively.

Second, the distinction between "specification" and "implementation" is collapsing. When the transformation between them is mechanical and instantaneous, the specification is the implementation in a meaningful sense. The Zilog manual contains the Z80 emulator the way an acorn contains an oak tree. The transformation from one to the other requires energy and process, but the information content is the same. Copyright protects the expression, but when the expression is a deterministic function of the specification, the creative contribution approaches zero.

This doesn't mean all AI-generated code is uncopyrightable. If I write a detailed architectural plan, direct Claude to implement it, review and revise the output, and make structural decisions throughout the process, the result reflects my creative choices expressed through an AI tool. The tool is more sophisticated than a compiler, but the relationship is similar: I made the design decisions; the tool translated them into a lower-level representation. The copyright, if it exists, is in my architectural choices, not in Claude's line-by-line implementation.

But if someone asks Claude to "write a Z80 emulator" with no architectural plan, no structural constraints, and no iterative review, and Claude produces a working emulator from its training data, who owns that code? Not the person who typed the prompt; they made no creative contribution beyond the request. Not Anthropic; they built the tool but didn't direct the output. Not the authors of the Z80 emulators in the training data; their code wasn't copied in any legally meaningful sense. The code exists in a copyright vacuum: produced by a process that doesn't have an author in the way the law requires.

Why This Matters Now

The velocity of AI-assisted code production is accelerating. Every developer using Claude, Copilot, or Cursor is producing code whose provenance is uncertain. The code works. It passes tests. It ships to production. And its relationship to the training data that informed it is, in a strict legal sense, unknown and unknowable.

The current legal frameworks (copyright, clean room, fair use) were designed for a world where code was written by humans who could testify about their creative process. "I read the specification. I designed the architecture. I wrote the code. I did not reference any existing implementation." This testimony is the foundation of clean-room defense. An LLM cannot provide it, and the human directing the LLM can only testify about their own contributions (the prompt, the architectural plan, the review), not about what the model drew from.

I took a CS ethics course as an undergraduate. The cases we studied (Compaq's clean-room reimplementation of the IBM PC BIOS, SCO's claim that Linux contained UNIX code, DeCSS and the DMCA's prohibition on circumventing copy protection) all assumed a human author whose creative process could be examined and whose sources could be traced. Every one of those cases would be decided differently if the defendant had said "I told an AI to implement the specification and it produced this code." The existing precedent doesn't apply, and the new precedent doesn't exist yet.

The Acorn and the Oak

Heidegger would say that the danger of Enframing is not that it's wrong but that it's totalizing. When technology reveals everything as standing reserve, we lose the ability to see things as they are. The river becomes only a power source. The specification becomes only raw material for code generation. The act of programming becomes only a transformation pipeline from input to output.

What gets lost is what the clean-room process was actually designed to protect: the space between specification and implementation where human understanding operates. That space is where a developer reads "the ADD instruction sets the zero flag if the result is zero" and decides how to express that in code. The decision is small. The creativity is modest. But it's real, and it's human, and it's the entire basis of software copyright.

AI doesn't eliminate that space. My Z80 emulator project included genuine creative decisions: the architecture, the test strategy, the system emulator design. Lattice exists because I designed a novel type system that no AI would have invented from existing languages. The creative space still exists for the people who operate at the design level.

But for the implementation level, for the transformation from "what this should do" to "code that does it," the space is closing. The specification is becoming the implementation. The acorn is becoming the oak without passing through the seasons of human comprehension. And the legal and philosophical frameworks we built for a world where that transformation required human creativity haven't caught up.

They will. The question is how much code ships before they do.

The Excavator and the Foundation

A.C. Jokela — Sat, 21 Mar 2026 13:00:00 GMT

Jason Fried posted a sharp critique of the "bespoke software revolution" narrative this week. His argument: most people don't like computers, don't want software projects, and won't become builders just because AI hands them better tools. The three-person accounting firm wants the paperwork gone, not a new system to maintain. The logistics company wants optimized routes, not Joe's side project. The law firm wants leverage on their time, not a codebase.

His metaphor is good: "A powerful excavator doesn't turn a homeowner into a contractor. Most people just want the hole dug by someone else."

He's right about who builds. He's wrong about what happens next.

The Echo Chamber Is Real

Fried's observation about the software community talking to itself lands because it's obviously true. Open any tech feed and the bespoke software excitement is coming from people who already build software for a living. They're excited because AI makes their work faster and more interesting. They project that excitement onto everyone else and conclude that everyone will want to build. This is like assuming everyone wants to change their own oil because you enjoy working on cars.

Most people have no interest in building software. Not because they lack intelligence or creativity, but because software is a means to an end and they'd rather focus on the end. The accounting firm wants to close the books faster. The logistics company wants fewer empty miles. These are domain problems, not software problems, and the people who understand them best have spent their careers on the domain, not on code.

Fried identifies the outliers correctly: the people who go deep with AI building tools were already dabblers. The curiosity was already there. AI didn't create new builders; it gave existing builders a power tool. This is an important observation that the tech community consistently ignores because it's less exciting than "everyone becomes a developer."

Where Fried Stops

But Fried's analysis ends at "most people won't build," and that's where the interesting question starts. Because some people will try.

Not the majority. Not the three-person accounting firm drowning in paperwork. But the accounting firm's nephew who's "good with computers." The operations manager at the logistics company who watched a YouTube tutorial on Cursor. The paralegal at the law firm who built a spreadsheet macro once and now has access to tools that can generate entire applications from a text description.

These people exist in every organization. They're not professional developers. They don't think of themselves as builders. But they have just enough technical confidence to be dangerous, and AI tools have just lowered the barrier enough to let them act on it.

This is not a hypothetical. It's already happening. People are building internal tools with AI assistance, deploying them to their teams, and running business processes on software that no one with software judgment has reviewed. The tools work on the happy path. They do exactly what the builder asked for. The problem is what the builder didn't ask for.

The Happy Path Is All You Get

When a non-developer builds software with AI, they describe what they want: "I need a tool that takes client intake forms, extracts the relevant fields, and puts them in a spreadsheet." The AI builds it. It works. The builder is thrilled.

What the builder didn't specify, and the AI didn't volunteer:

What happens when a client submits a form with special characters that break the parser? What happens when two people submit simultaneously? What happens when the spreadsheet hits the row limit? Where are the backups? Who has access? What happens when the API key expires? What happens when the builder leaves the company and nobody knows how the tool works?

These aren't obscure edge cases. They're the standard failure modes of every software system ever built. Professional developers think about them not because they're smarter, but because they've watched systems fail in these exact ways. That accumulated experience of failure is what I've been calling the judgment layer: the part of building that AI can't replace because it requires contact with the consequences of getting it wrong.

The operations manager building a routing tool in Cursor has domain judgment about logistics. She knows which routes are efficient and which constraints matter. She does not have software judgment about error handling, data integrity, concurrent access, or failure recovery. Professional developers fail at these things constantly too. The difference is that professionals recognize the failure when it happens and have the skills to iterate toward a fix. The operations manager's tool breaks the same way, but she doesn't know it broke, doesn't know why, and doesn't know what to do about it. The AI gave her a tool that satisfies her domain judgment perfectly and her software judgment not at all, because she doesn't have any, and she doesn't know she doesn't have any.

This Has Happened Before

The counterargument writes itself: people have been building bad mission-critical software forever. Hospitals tracked patient records in Access databases. Small banks ran loan portfolios in Excel. Supply chains depended on macros that one person understood. When that person left, nobody could maintain it. The world survived.

This is true, and it's important to take seriously. "Bad software" and "functional software" are not mutually exclusive. The accounting firm's Access database was terrible by every engineering standard and it ran their business for fifteen years. The nurse's Excel tracker was a data integrity nightmare and it kept patient appointments from falling through the cracks. Fried is right that custom software has always been "bloated, confusing, and built wrong in all the ways." He's also right that it existed and that people used it.

So if bad software has always existed and the world kept turning, what changes with AI?

Velocity

The change is speed.

Access took months to build something broken. You had to learn Access first, or find someone who knew it. You had to build the forms, design the tables, write the queries. The pace of construction imposed a natural speed limit on how fast bad software could enter production. By the time you finished, you'd encountered at least some of the failure modes, because the slow process forced you through enough iterations to stumble into them.

AI removes that speed limit. The operations manager can go from "I have an idea" to "it's running in production" in an afternoon. The intake form tool is live before lunch. The routing optimizer is deployed by end of day. The contract parser is running by Friday. Each one works on the happy path. Each one has the same class of unexamined failure modes that Access databases had. But Access databases took months to accumulate. AI-built tools accumulate in days.

More attempts. Same failure rate. More failures. Compressed into a shorter timeline. By the time the first tool breaks, three more have been deployed. By the time someone realizes the intake form tool is silently dropping records with special characters, the routing optimizer and the contract parser are already load-bearing parts of the business.

This is Jevons Paradox applied to the failure mode itself. When building software gets cheaper, you don't get the same amount of bad software for less effort. You get vastly more bad software for the same effort. The per-unit cost of production drops, total production expands, and the total volume of unreviewed, unexamined software in production grows faster than anyone anticipated.

The Judgment Bottleneck

I've argued in previous pieces that human judgment is the binding constraint in AI-augmented work. AI makes the labor cheaper; demand expands; the expansion concentrates on the one input that can't scale: the human capacity for deep, focused evaluation. The three-to-four-hour ceiling on cognitively demanding work is biological, not cultural, and no productivity tool changes it.

Software judgment is a specific instance of this general constraint. Reviewing code for failure modes, reasoning about edge cases, thinking through data integrity, anticipating what happens when components interact in unexpected ways: this is deep work. It requires the kind of sustained attention that depletes on a fixed biological schedule. And the supply of people who have this judgment is not growing. Computer science enrollment is up, but software judgment comes from experience, not coursework. You develop it by watching systems fail, and that takes years.

AI expands the rate at which software enters production. It does not expand the rate at which qualified people can review it. The production side scales. The judgment side doesn't.

And the judgment side may actually be contracting. After sixteen consecutive years of growth, undergraduate CS enrollment turned negative in 2025. The Computing Research Association (CRA) found that 62% of computing departments reported declining enrollment for 2025-26, while only 13% saw increases. At University of California campuses, CS enrollment fell 6% in 2025 after declining 3% in 2024: the first drops since the dot-com crash. Students and their parents are reading the headlines about AI displacing entry-level developers and steering toward fields they perceive as more durable.

The irony is thick. The fear that AI will replace software developers is reducing the supply of software developers at the exact moment that AI is massively expanding the demand for software judgment. Students are fleeing the field because they think AI can do the work. AI is simultaneously creating more work that only humans with software judgment can evaluate. The enrollment decline doesn't just fail to solve the judgment bottleneck; it tightens it.

The gap between "software that exists" and "software that someone qualified has evaluated" widens from both directions: production accelerates while the pipeline of qualified reviewers narrows. Something has to give, and what gives is the review.

Software Slop

I wrote an essay about what makes AI-generated content "slop": superficial competence masking an absence of substance. The text looks right. The grammar is clean. The structure is logical. But it doesn't commit to anything, doesn't engage with anything, doesn't mean anything. It fills the container without filling it with content.

AI-generated software has the same property. The code is syntactically correct. The UI has proper styling, responsive layouts, loading spinners, appropriate error messages. It passes every visual inspection. A manager looking at a demo sees a professional application. A user running through the standard workflow sees something that works.

Underneath: no input validation beyond what the framework provides for free. No error handling beyond try/catch blocks that swallow exceptions. No concurrency protection. No backup strategy. No audit trail. No security beyond defaults. The software is superficially competent and structurally hollow, and you cannot tell the difference by looking at it.

This is what distinguishes the AI-built software problem from the Access database problem. Access databases looked like Access databases. The limitations were visible in the interface. The grey forms, the flat tables, the clunky queries: everyone could see they were using a tool that was not designed for what they were doing with it. The expectations were calibrated, even if the risks weren't.

AI-built software looks like real software. The surface quality has been democratized. What hasn't been democratized is the structural integrity underneath. And because the surface looks professional, the people using it have no signal that anything is missing. The feedback loop that would normally tell you "this is a prototype, not a product" has been severed. The prototype looks like the product, and nobody in the room can tell the difference except the people with software judgment, who weren't in the room when it was built.

What Fried Misses

Fried's framework has one gap. He says the demand for bespoke software won't grow because people don't want software projects. But the demand is already growing, not because people want to build, but because AI collapsed the apparent cost of building to near zero. The operations manager didn't set out to start a software project. She set out to solve a routing problem, and the software was a side effect that happened so fast she didn't register it as a project.

This is the mechanism Fried doesn't account for. The excavator doesn't turn the homeowner into a contractor. But it does let the homeowner dig a hole so fast that they're standing in it before they realize they don't know what they're doing. The question isn't whether they wanted to dig. It's what happens now that the hole exists and the house is being built on top of it.

The bespoke software revolution won't come from people deliberately choosing to become builders. It will come from people accidentally becoming builders because the tools made it so frictionless that building happened before the decision to build was consciously made. And the software they produce will be the fastest-growing category of technical debt in history, because it was created without judgment, deployed without review, and adopted without anyone understanding what's underneath.

Who Benefits

Fried is right that the excitement about bespoke software comes from software makers. What he doesn't say is why they should be excited. It's not because everyone becomes a builder. It's because everyone becomes a client.

Every operations manager who builds a broken routing tool and discovers it doesn't handle the edge cases is a future client for someone who can build it properly. Every accounting firm that deploys an AI-built intake system and loses data is a future client for someone who understands data integrity. The DIY phase doesn't replace professional software development. It creates demand for it, at a scale and urgency that didn't exist before, because now the potential clients have firsthand experience with why the problem is hard.

The judgment bottleneck doesn't prevent the Jevons expansion. It shapes it. More software gets attempted. More software fails. The failures create demand for the constrained resource (qualified judgment) at a rate that exceeds supply. The people who have software judgment become more valuable, not less, because the volume of work that needs their attention has exploded.

Fried's excavator metaphor is correct. Most homeowners won't become contractors. But the excavator lets them dig enough bad foundations that the contracting business booms. AI doesn't democratize building. It democratizes demand.

The Forecast

I'll make a prediction specific enough to be wrong about. Within three years, the majority of data-loss and security incidents at small and mid-sized businesses will trace back to AI-assisted internal tools built without professional review. Not because the AI wrote bad code (the code will be syntactically fine), but because the person directing the AI didn't know what to ask for and didn't know what they were missing. The failure mode won't be dramatic. It will be silent: records that were never backed up, access controls that were never configured, race conditions that corrupt data once a month in a pattern nobody notices until the audit.

There is an irony here that I should name. The people who most need to read this are the ones who never will. The operations manager vibing a routing tool into production this afternoon is not reading a blog about Jevons Paradox and GPU inference. She's solving her problem, and it feels like it's working, and no article on a site called Tiny Computers is going to reach him before the first silent failure does.

The bespoke software revolution is real. It's just not the revolution anyone is advertising. It's not a million people building great custom tools. It's a million people building adequate tools with invisible structural deficiencies, deployed to production at a velocity that outpaces the world's capacity to review them. The excavator is powerful, the foundations are being dug, and most of them are too shallow.

Running a 22B Video Model on Four Tesla P40s

A.C. Jokela — Fri, 20 Mar 2026 13:00:00 GMT

LTX-Video 2.3 is a 22 billion parameter model that generates video from text prompts. It was designed for modern hardware: GPUs with bfloat16 support, high-bandwidth memory, and enough VRAM to hold the full model on one or two cards. The Tesla P40 has none of these things. It is a Pascal-generation GPU from 2016, with 24GB of GDDR5X per card, no native bfloat16, no Tensor Cores, and a PCIe 3.0 bus. It was built for data center inference workloads that no longer exist.

I have four of them in a rack-mount server in an unheated shop building in Minnesota. Together they provide 96GB of VRAM. The question was whether that 96GB, spread across four old cards, could run a model that was never meant to run on any of them.

The answer is yes, with significant caveats and a substantial amount of code to work around hardware limitations that the model's authors never anticipated.

The Problem

LTX-Video 2.3's transformer has 48 blocks. At fp16 precision, the model weights alone consume roughly 44GB. With the Gemma text encoder, the video VAE encoder/decoder, the spatial upsampler, and the audio components, the full pipeline needs more memory than any single P40 can provide. The model doesn't fit on one card. It doesn't fit on two. It barely fits on three, with no room for activations during inference.

Four cards at 24GB each gives 96GB total, which is enough for the weights with room for intermediate activations. But CUDA doesn't automatically spread a model across multiple GPUs. You have to tell it how.

The standard approach for multi-GPU inference is accelerate's dispatch_model, which automatically distributes model layers across available GPUs based on memory constraints. This works for the Gemma text encoder, which is a straightforward transformer. For the LTX transformer, it doesn't work, because the model has a custom forward pass with audio-video cross-attention that accelerate's automatic dispatch can't handle correctly. The model needs to move data between GPUs at specific points in the forward pass, and accelerate doesn't know where those points are.

The solution was manual pipeline parallelism: split the 48 transformer blocks evenly across four GPUs (12 blocks per card), keep the shared components (patchify projections, normalization, output projections) on GPU 0, and write a custom forward pass that moves tensors between devices at block boundaries.

The Precision Problem

Even with the model split across four cards, nothing worked on the first attempt. Or the fifth. Getting LTX-Video running on Pascal hardware was an iterative process, with Claude Code generating solutions and me testing them against the actual hardware. Each failure revealed another assumption the model made about the GPU it would run on. The feedback loop was brutal: load a 22B model across four GPUs, wait eight minutes for a test generation, get a black frame or a NaN error, diagnose which precision boundary caused it, generate a fix, and try again.

The first problem was bfloat16. The model weights are stored in bf16 format. Pascal GPUs cannot compute in bf16. PyTorch handles this silently for some operations by promoting to fp32, but other operations fail or produce garbage. The initial approach was the obvious one: monkey-patch torch.bfloat16 to redirect to torch.float16. This seemed to work at load time. The model loaded, the weights populated, no errors. Then the first forward pass produced NaN everywhere. The monkey-patch had corrupted the safetensors weight loading. The weights loaded as fp16 bit patterns interpreted as bf16 values, which is not the same thing. A bf16 value of 1.0 has a different bit pattern than an fp16 value of 1.0. Reinterpret one as the other and you get a number that's either wildly wrong or NaN.

The second attempt tried running everything in fp16 natively, converting weights properly during load. This got further: the model produced output that wasn't NaN. But the output was a solid green frame. The intermediate activations in the transformer blocks were overflowing fp16 range. Values above 65,504 become infinity in fp16, and the model's internal representations regularly exceed that during the attention and feedforward passes. The green frame was the model's attempt to decode latents that had been clipped to infinity at some point in the pipeline.

The working solution was to let the model builder properly convert weights from bf16 to fp16 on load, then run the entire computation pipeline in float32. The weights sit in memory as fp16 (saving space), but every computation promotes to fp32 before executing. This required patching F.linear to handle mixed dtype inputs:

_orig_linear = F.linear
def _mixed_linear(input, weight, bias=None):
    if input.dtype != weight.dtype:
        weight = weight.to(input.dtype)
        if bias is not None:
            bias = bias.to(input.dtype)
    return _orig_linear(input, weight, bias)
F.linear = _mixed_linear

The same pattern extends to every normalization function and every convolution operation. Layer norm, group norm, RMS norm, conv1d through conv_transpose3d: all patched to handle mixed dtypes and accumulate in float32. Without these patches, intermediate values overflow fp16 range (values above 65,504 become infinity) and the output is a black frame.

The Gemma Problem

The text encoder is Google's Gemma 3, a separate model that converts text prompts into embeddings the video transformer can condition on. Gemma's attention mechanism overflows when run in fp16 on Pascal hardware. The attention scores grow large enough to exceed fp16 range, producing NaN values that propagate through the rest of the pipeline.

The fix was running the entire Gemma encoder in float32. This uses more memory, but the text encoder only runs once per generation (to encode the prompt), and its weights can be freed from GPU memory before the transformer starts. The sequence is: load Gemma across all four GPUs using accelerate, encode the prompt in float32, delete the encoder, free the memory, then load the video transformer.

def encode_prompt_float32(prompt, model_ledger, device):
    model_ledger.dtype = torch.float32
    te = model_ledger.text_encoder()
    # Dispatch across all 4 GPUs for memory
    max_memory = get_balanced_memory(
        te, max_memory={i: "22GiB" for i in range(4)},
        no_split_module_classes=["Gemma3DecoderLayer"],
    )
    te = dispatch_model(te, device_map=device_map)
    hidden_states, attention_mask = te.encode(prompt)
    # Free GPU memory before transformer loads
    del te
    gc.collect()
    torch.cuda.empty_cache()

This load-encode-delete cycle is ugly but necessary. There isn't enough total memory to hold both Gemma and the video transformer simultaneously, even across four cards. The sequential approach works because each component only needs to exist during its phase of the pipeline.

The Pipeline

The generation runs in two stages, matching LTX-Video's distilled inference schedule.

Stage 1 generates a half-resolution latent video (e.g., 256x384) through 8 denoising steps. Each step runs the full 48-block transformer, with data moving across all four GPUs:

def patched_process(video, audio, perturbations):
    for i, block in enumerate(ltx.transformer_blocks):
        dev = block_devices[i]
        video = move_args_to_device(video, dev)
        audio = move_args_to_device(audio, dev)
        video, audio = block(video=video, audio=audio,
                             perturbations=perturbations)
    video = move_args_to_device(video, device0)
    audio = move_args_to_device(audio, device0)
    return video, audio

Every GPU boundary involves a tensor transfer across PCIe 3.0. With 12 blocks per GPU, there are 3 boundary crossings per denoising step (GPU 0 to 1, 1 to 2, 2 to 3), plus a final transfer back to GPU 0. With 8 denoising steps, that's 32 cross-device transfers per stage, each moving both video and audio state tensors. PCIe 3.0 x16 has a theoretical bandwidth of ~16 GB/s. The tensors being transferred are small relative to the bandwidth (attention states and activations, not full weight matrices), so the overhead is manageable. But it adds up.

Stage 1 takes roughly 4 minutes for 241 frames at 24 fps (a 10-second clip). The spatial upsampler then doubles the resolution. Stage 2 runs 3 more denoising steps at full resolution (512x768), taking roughly 6.5 minutes. The VAE decoder converts latents to pixels and generates the audio track in another 40 seconds.

Total generation time for a 10-second, 512x768 video with audio: approximately 18.5 minutes. For a 1-second clip (25 frames): about 8 minutes. For a 4-second clip (97 frames): about 10.5 minutes.

The Memory Layout

During inference, the four GPUs aren't loaded equally. GPU 0 carries extra weight because it hosts all the shared components (patchify projections, normalization layers, output projections) plus its 12 transformer blocks. The actual memory distribution:

GPU	VRAM Used	Role
0	10.8 GB	Shared components + blocks 0-11
1	9.3 GB	Blocks 12-23
2	9.3 GB	Blocks 24-35
3	9.3 GB	Blocks 36-47

That's 38.7 GB of the available 96 GB. The remaining 57 GB provides headroom for activations, KV cache growth, and the VAE decoder. There's enough margin that generation never OOMs, even at 241 frames.

The API

Running inference from the command line is fine for testing, but generating videos for blog content requires something more practical. I wrapped the generation script in a FastAPI server with an async job queue:

# Submit a text-to-video job
curl -X POST http://10.1.1.24:8585/jobs \
  -F "prompt=A cinematic flyover of a Zilog Z80 processor on a PCB" \
  -F "duration=10" -F "seed=42"

# Submit an image-to-video job
curl -X POST http://10.1.1.24:8585/jobs \
  -F "prompt=A fluffy orange cat dancing" \
  -F "duration=4" -F "image=@cat.jpg"

# Check status
curl http://10.1.1.24:8585/jobs/07420abb6d82

# Download result
curl http://10.1.1.24:8585/jobs/07420abb6d82/video -o output.mp4

Jobs queue and execute sequentially. The GPU can only handle one generation at a time, and the load-encode-delete cycle for Gemma means there's significant setup overhead per job. The API spawns each job as a subprocess, which gives clean GPU memory cleanup between runs. If a generation crashes (which happened frequently during development), the next job starts fresh.

The server supports both text-to-video and image-to-video. Image conditioning locks the first frame to a provided image and generates subsequent frames from it, which produces more controllable results for specific visual subjects. In practice, image-to-video is the more useful mode. Text-to-video gives the model complete creative freedom, which means the output is unpredictable. You might ask for a Z80 processor and get something that looks like a generic IC, or something that looks like a Z80, depending on the seed. Image-to-video lets you provide the exact first frame you want and the model animates from there. For blog content where visual accuracy matters, starting from a real photograph or a specific reference image gives consistently better results.

What the Output Looks Like

The video quality is genuinely good. LTX-Video 2.3 produces coherent motion, reasonable physics, and detailed textures. Here are three examples, generated entirely on the P40 server:

Text-to-video: "A cinematic flyover of a Zilog Z80 processor on a printed circuit board" (10 seconds, 18.5 minutes to generate)

Image-to-video: "A fluffy orange cat with a hat dancing" (4 seconds, 10.5 minutes to generate)

Text-to-video: "A cat sitting on a windowsill, sunlight streaming in" (1 second, 8 minutes to generate)

The model understands object permanence, lighting consistency, and basic spatial relationships. The Z80 flyover produces a recognizable IC package with surrounding components, proper lighting, and smooth camera movement.

The audio is a different story. LTX-Video 2.3 generates an audio track alongside the video, but the results are inconsistent. Prompts describing characters speaking produce odd ambient music instead of voices. Prompts describing environments produce vaguely appropriate soundscapes. The audio pipeline works mechanically (it generates real audio waveforms via a separate VAE decoder and vocoder), but the semantic connection between prompt and audio output is weak. For blog content, I'd likely strip the generated audio and add narration or music separately.

The 512x768 resolution at 24fps is usable for web content. It's not 4K. It's not going to replace stock footage for production video. But for blog hero images in motion, visual demonstrations, or supplementary content alongside text, it works.

What This Cost

The hardware cost is zero incremental. The four P40s and the server already existed for LLM inference. LTX-Video is an additional workload on the same hardware.

The electricity cost is modest. The server draws roughly 500W under full GPU load. An 18.5-minute generation (10-second video at full resolution) consumes about 0.15 kWh, roughly $0.024 at Minnesota residential rates. You could generate forty 10-second clips for a dollar.

The real cost was development time. Getting from "model downloaded" to "working generation pipeline" took many iterations across multiple sessions with Claude Code. Each precision-related failure mode (bf16 corruption, fp16 overflow, mixed-dtype kernel errors, NaN propagation through attention) required diagnosis, a hypothesis, a code change, and a test cycle that involved loading a 22B model across four GPUs. The feedback loop was slow. A single test takes 8 to 18 minutes to confirm whether a change worked. Many didn't.

The Broader Point

A 22 billion parameter video generation model was not designed to run on 2016 hardware. The authors assumed bf16, assumed modern attention kernels, assumed enough memory on one or two cards. None of those assumptions hold on the P40.

But the model runs anyway, because the underlying math doesn't actually require any of those features. Bfloat16 is a convenience, not a requirement; float32 computes the same function. Flash attention is an optimization, not a necessity; standard attention produces identical results. And 96GB across four cards is 96GB, regardless of whether it's cutting-edge HBM3 or decade-old GDDR5X.

The generation is slow. Eighteen minutes for ten seconds of video is not competitive with a single A100, which would finish the same job in under two minutes. The float32 computation pipeline roughly doubles the FLOPS required compared to the bf16 path the model was designed for, and the PCIe 3.0 transfers between four separate memory pools add latency that a single modern GPU with unified HBM would never incur. But competitive wasn't the point. The point was that four GPUs I bought on eBay for a thousand dollars total, sitting in a server in a shop building, can run a model that was released this month. The gap between "latest model" and "latest hardware" is not as wide as the spec sheets suggest, as long as you're willing to write the code that bridges it.

The P40 server was already paying for itself on LLM inference and TTS generation. Video generation is one more workload on a machine that I own, running models that I choose, on a schedule that I control. The 18-minute wait is the price of not asking anyone's permission.

The Split Isn't Between People, It's Between Tasks

A.C. Jokela — Thu, 19 Mar 2026 13:00:00 GMT

Les Orchard's "Grief and the AI Split" identifies something real. AI tools have revealed a division among developers that was previously invisible, because before these tools existed, everyone followed the same workflow regardless of motivation. Now the motivations are exposed. Some developers grieve the loss of hand-crafted code as a practice with inherent value. Others see the same tools and feel relief: the tedious parts are handled, the interesting parts remain. Orchard frames this as a split between people. Craft-oriented developers on one side, results-oriented developers on the other.

He's right that the split exists, and the piece clearly resonated with software creators because it names something people have been feeling but couldn't articulate. The observation is sharp. Where I think it can be extended is in where the line falls.

Orchard draws the line between people. I think it falls between tasks. The same person crosses that line dozens of times a day, moving between work that demands human judgment and work that doesn't, between moments where the craft concentrates and moments where it was never present in the first place. The split is real. It's just not an identity.

The Kernel I Didn't Write

JokelaOS is a bare-metal x86 kernel: 2,000 lines of C and NASM assembly, booting from a Multiboot header through GDT (Global Descriptor Table, which defines memory segments and access rights) and IDT (Interrupt Descriptor Table, which maps interrupt vectors to service routines) setup, paging, preemptive multitasking with Ring 3 isolation, a network stack that responds to pings, and an interactive shell. No forks. No libc. Every memcpy, every printf, every byte-order conversion written from scratch.

I didn't write most of it. Claude did.

In Orchard's framework, this should place me firmly in the "results" camp. I used AI to produce 2,000 lines of systems code; clearly I care about the outcome, not the process. But that framing misses what actually happened during the project.

The decisions that made JokelaOS work were not typing decisions. They were sequencing decisions: bring up serial output first, because without it you have no diagnostics for anything that follows. Initialize the GDT before the IDT, because interrupt handlers need valid segment selectors. Get the bump allocator working before the PMM (Physical Memory Manager), because page tables need permanent allocations before you can manage dynamic ones. These choices come from understanding how x86 protected mode actually works, which subsystems depend on which, and what the failure modes look like when you get the order wrong.

Claude generated the GDT setup code. I decided what the GDT entries should be, caught the access byte errors, and debugged the triple faults when segment selectors were wrong. Claude wrote the process scheduler. I determined that the TSS (Task State Segment, which tells the CPU where to find the kernel stack when switching privilege levels) needed updating on every context switch and diagnosed the General Protection Faults that occurred when it wasn't. Claude produced the RTL8139 network driver. I decided to bring up ARP before ICMP, caught a byte-order bug in the IP checksum, and validated that the packets leaving QEMU were actually well-formed.

The typing was delegated. The architecture, the sequencing, the diagnosis, the validation: those were mine. If you asked me whether JokelaOS involved craft, I would say yes, more than most projects I've done. If you asked me where the craft was, I would not point at any line of code.

The Board That Failed Twice

The Giga Shield tells a longer version of the same story, and it's messier, because hardware involves the physical world in a way that software doesn't.

The project started with a $468 Fiverr commission. I gave a designer in Kenya the spec documents, the components I thought should be used, and the form factor requirements: an Arduino Giga R1 shield with bidirectional level shifters, 72 channels of 3.3V-to-5V translation, KiCad deliverables. He produced a clean design. Nine TXB0108PW auto-sensing translators on a two-layer board. PCBWay fabricated it. Professional work, quick turnaround, and PCBWay sponsored the fabrication.

Then I plugged in the RetroShield Z80 and the board was blind.

The TXB0108 detects signal direction automatically by sensing which side is driving. For most applications, that's a feature. For a Z80 bus interface, it's fatal. During bus cycles, the Z80 tri-states its address and data lines. The pins go high-impedance: not high, not low, floating. The TXB0108 can't determine direction from a floating signal. It guesses wrong, and the Arduino reads garbage. I'd paid $468 for a board that couldn't see half of what the processor was doing.

Nobody caught this in the design phase. Not the Fiverr designer, who was working from the spec I gave him. Not me, when I reviewed the schematic. The TXB0108 datasheet doesn't scream "incompatible with tri-state buses"; you have to understand what tri-stating means in practice and recognize that auto-sensing can't handle it. That understanding came from plugging the board in and watching it fail.

The redesign used Claude to replace all nine auto-sensing translators with SN74LVC8T245 driven level shifters. Driven shifters have an explicit direction pin: you tell them which way to translate, and they do it regardless of whether the signal is being actively driven. Claude wrote Python scripts that pulled apart the KiCad schematic files, extracted all 72 signal mappings across 9 ICs, and generated new board files with the correct components and pin assignments.

I was about to submit the revised design to PCBWay when I realized we needed a tenth level shifter. The original nine covered not just the digital pins that map to the Z80 RetroShield but all of the analog pins on the Giga, giving complete 3.3V-to-5V coverage across the board. But with driven shifters, each IC has a single direction pin controlling all eight channels. Signals that need to travel in opposite directions at different times can't share an IC without creating bus contention. Some of the channel assignments had conflicting direction requirements, and the only fix was a tenth IC to separate them.

Adding one more TSSOP-24 package to an already dense two-layer board broke the trace routing. The board that had been routable with nine ICs was unroutable with ten. Moving to four layers helped but still left two to four traces with no viable path. The solution was a six-layer stackup, which needed a copper pour layer to act as a common ground plane. The open-source autorouter Freerouting couldn't handle a full copper pour; its architecture has no concept of flood-fill connectivity. So I used Quilter.ai, an AI trace router, to route the six-layer board with the ground plane that the open-source tooling couldn't represent.

Count the layers of delegation and intervention in this project. I delegated the initial design to a human professional. Physics revealed the flaw. I delegated the redesign to an AI. I caught the missing tenth shifter before it went to fabrication. I delegated the trace routing to another AI. PCBWay is currently manufacturing these boards. At every stage, the work alternated between labor that could be delegated and judgment that couldn't. The Fiverr designer did skilled labor. Claude did skilled labor. Quilter.ai did skilled labor. The craft was never in the labor. It was in knowing when the labor was wrong.

Where the Craft Actually Lives

Both of these projects point at the same thing. The craft isn't in the typing, the routing, or the code generation. It's in a layer that sits above and around all of that: the judgment layer.

The judgment layer is where you decide what to build next. Where you recognize that the output is wrong before you can articulate why. Where you sequence subsystems based on dependency chains that aren't documented anywhere. Where you plug a board in and notice that the readings don't make sense. Where you catch a missing component that the AI, the designer, and the autorouter all missed because none of them were thinking about the problem at that level.

This layer has specific properties. It requires contact with the problem domain, not just the code or the schematic but the actual behavior of the system under real conditions. It depends on accumulated experience: understanding what tri-stating means in practice, knowing that x86 protected mode has forty years of backward-compatible traps waiting for you. And it's the part that AI is worst at, precisely because it requires grounding in physical or logical reality that language models don't have access to.

The TXB0108 failure is the clearest example. The information needed to predict this failure existed in the datasheets. But recognizing its relevance required understanding what a Z80 bus cycle actually looks like at the electrical level, which required either experience with the hardware or a simulation environment that nobody had set up. No amount of language model capability substitutes for plugging in the board and watching it fail.

The Same Person in Both Modes

Orchard describes himself as results-oriented. He learned programming languages as "a means to an end" and gravitated toward AI tools because they let him focus on the outcome. He acknowledges that craft-oriented developers experience genuine loss. His framing is empathetic, but it still draws the line between people.

The line doesn't hold, because I'm both of his archetypes depending on the hour.

On Tuesday I might use Claude to generate a hundred lines of systemd service configuration because I need Ollama running on a machine and I don't care about the elegance of the unit file. On Wednesday I might spend three hours hand-debugging why rocm-smi reports GPU utilization at zero percent: reading kernel logs, checking DKMS module versions, testing HSA_OVERRIDE_GFX_VERSION values, loading the amdgpu module manually because it didn't auto-load at boot. The first task is pure delegation. The second is pure craft. Both are mine. Both happened this week.

When I wrote the economics piece, I used Claude to draft sections and I measured real power draw with nvidia-smi and rocm-smi at 500-millisecond intervals. I let AI handle the prose scaffolding and I personally caught that Ollama on the Strix Halo had been running entirely on CPU because the systemd service file was missing an environment variable. Every benchmark I'd trusted before finding that bug was wrong. No AI caught it. I caught it because the numbers felt off.

These aren't different people. They're different tasks. The identity framing ("I'm a craft developer" or "I'm a results developer") obscures what's actually a task-level decision that experienced people make constantly: this piece of work benefits from my full attention; this piece doesn't.

What the Grief Is About

The craft-grief that Orchard describes is real and worth taking seriously. Part of it targets the wrong thing. Part of it doesn't.

What's being mourned is typing as the bottleneck. For forty years, the primary constraint on software projects was the speed at which a human could produce correct code. Design mattered, architecture mattered, but someone still had to sit down and type it. The typing was slow enough that it forced a certain kind of attention. You couldn't write a function without thinking about it, because writing it took long enough that thinking was unavoidable. The bottleneck created the conditions for craft, and it felt like the craft itself.

AI removes the bottleneck. Code appears in seconds. The thinking isn't forced by the typing anymore; it has to be deliberate. And that shift feels like a loss, because the rhythm of the work has changed. The long, meditative stretches of writing code, where your understanding deepened as your fingers moved, are replaced by short bursts of generation followed by review. The texture is different.

But the craft didn't live in the texture. It lived in the judgment that the texture incidentally supported. The experienced developer who hand-writes a function isn't doing craft because the typing is slow. The typing is slow, and the craft happens during the slowness, but the craft is the decisions: what to name things, what to abstract, what edge cases to handle, when to stop. Those decisions haven't gotten easier. If anything, they've gotten harder, because AI lets you attempt projects that would have been too large to type by hand, which means you hit the judgment bottleneck more often and at higher stakes.

JokelaOS would have taken me months to type by hand. I probably wouldn't have attempted it. With AI handling the code generation, I attempted it in days and spent the entire time making architecture and debugging decisions. The project had more craft in it than most things I've built, precisely because the typing wasn't the bottleneck. The judgment was.

The Biological Ceiling

I wrote in the AI Vampire piece that human judgment is the binding constraint in a Jevons cycle operating on cognitive output. AI makes the labor cheaper; demand expands; the expansion concentrates on the one input that can't scale: human attention and judgment. The three-to-four-hour ceiling on deep work is biological, not cultural, and no amount of productivity tooling changes it.

The task-level split is where this plays out in practice. AI compresses the labor side of every project: the code generation, the trace routing, the prose drafting, the schematic extraction. What remains is denser, harder, and more consequential. Every hour of work has a higher ratio of judgment to labor than it did before AI. That's why Yegge's developers feel burned out, not because they're working more hours, but because every hour is now a judgment hour.

The craft isn't disappearing. It's being compressed into a smaller, denser layer. The typing is gone. The design reviews are shorter. The code appears instantly. What's left is the part that was always the actual craft: deciding what to build, recognizing when it's wrong, knowing what to test, catching the missing tenth level shifter. That layer is entirely human, it's harder than it used to be because the projects are bigger, and it's the only part that matters.

Orchard identified the split correctly. The grief is real, the division is real, and the piece resonated because it named something that software creators recognized immediately. The refinement I'd offer is that the line doesn't separate two kinds of people; it separates two kinds of tasks. The craft was never in the code. It was in the decisions that surrounded the code. Those decisions haven't gone anywhere. They've just lost the slow, meditative typing that used to accompany them. What remains is craft at higher concentration, with no filler.

There was something cathartic about the old way. The hours of typing weren't just production; they were a complete experience. You conceived the idea, worked through the logic, typed every character, fought the compiler, and watched it run. The whole arc from intention to execution passed through your hands. That totality had a satisfaction to it that reviewing AI-generated output doesn't replicate, even when the output is correct.

And there was something else: the syntax was a sacred tongue. Not everyone could read it. Not everyone could write it. The curly braces, the pointer arithmetic, the register mnemonics formed a language that belonged to the people who had invested years learning to speak it. That exclusivity wasn't gatekeeping for its own sake; it was the mark of hard-won fluency, and it meant something to the people who had it. Now anyone can describe what they want in English and get working code back. The priesthood dissolved overnight.

I feel that loss. I still create. I still orchestrate. I still catch the errors that the tools miss. But I no longer speak a language that most people can't. The judgment layer is real, and it's where the work that matters happens. But it doesn't carry the same weight as mastery of a difficult notation. Orchestrating a process is not the same as performing it, even if the orchestration requires more skill.

The grief is real. It's not about the wrong thing. It's about something that actually disappeared.

The Economics of Owning Your Own Inference

A.C. Jokela — Tue, 17 Mar 2026 13:00:00 GMT

I own $5,500 worth of GPU hardware dedicated to running AI models locally. I also pay for a Claude Max subscription that I use for nearly everything that matters. If that sounds like a contradiction, it is the entire subject of this article.

The local inference conversation online is dominated by two positions. The first: why pay for API calls when you can run models on your own hardware? The second: local models are worse, so just pay for the good ones. Both are correct. Both are incomplete. The interesting question is where the boundary falls between them, and the answer turns out to depend less on cost-per-token arithmetic than on what kind of work you are doing.

The Split

I use Claude for research, code review, writing feedback, technical analysis, and anything that used to be a Google search. The frontier models are better at all of these tasks than anything I can run locally. Not marginally better; categorically better. An 8B parameter model running on my hardware is not in the same conversation as Claude Opus or GPT-5.4 for anything requiring reasoning, nuance, or broad knowledge. The subscription cost is fixed regardless of volume, which eliminates per-query friction entirely. For interactive, quality-sensitive work, I pay for the best model available and I do not think about it.

Local inference handles everything else: the batch jobs, the grunt work, the high-volume tasks where model quality matters less than model availability. The work that would be expensive at cloud API rates not because any single call costs much, but because the calls number in the tens of thousands.

This is not a temporary arrangement while local models catch up. It is a structural split. Frontier models are getting better. Local models are also getting better. The gap is not closing in the ways that matter for my usage, because the tasks I send to each side are fundamentally different. I do not need my local 8B model to reason better. I need it to process text cheaply and without metering.

What the Local Hardware Actually Does

Three workloads. All batch. All quality-tolerant.

Text-to-speech. Every post on this site has an AI-generated audio narration. This is the workload that justifies the hardware on its own. Google Cloud Platform has superior TTS voices; Chirp3-HD sounds noticeably more natural than any open-source model I have tested. I ran a novel through it once: 82,000 words, 500,000 characters, $17.25. That is reasonable for a one-off project.

It is not reasonable for a library of blog posts that I revise and regenerate periodically. At GCP rates ($16 per million characters, more for premium voices), narrating every post on this site would cost $200 to $400, and that bill resets every time I edit an article and regenerate the audio. Open-source TTS (F5-TTS and Qwen TTS) mispronounces technical terms. The prosody goes flat on dense jargon. But it is good enough for blog narration. "Good enough" at zero marginal cost beats "excellent" at $4 to $10 per post when you are generating audio daily.

Code scanning. Running local models over source files for pattern detection, documentation extraction, and automated analysis. These jobs produce high token volume at low quality requirements. An 8B model is adequate. The token count across a full codebase makes API pricing add up in a way that individual queries do not.

Infrastructure work. Benchmarking hardware (as in this article), testing prompt structures across quantization levels, evaluating model behavior under different configurations. These queries have no value individually. They are the test drives, not the commute. Paying per-token for test drives is paying per-mile to drive your own car around the block.

None of these workloads require a frontier model. All of them generate enough volume to make metered pricing uncomfortable. That is the boundary.

The Machines

Two machines. Both mine. Both running Ollama.

A four-GPU Tesla P40 server: Penguin Computing 2U chassis, Xeon E5-2697A v4, 252GB DDR4 ECC, four Tesla P40s with 24GB GDDR5X each. Ninety-six gigabytes of VRAM. Pascal architecture, 2016 vintage. Built from eBay parts for about $2,500. Lives in an unheated shop building in Minnesota.

A Bosgame M5 mini desktop: AMD Ryzen AI MAX+ 395, Strix Halo APU with integrated RDNA 3.5 graphics. No discrete GPU. CPU and GPU share 128GB DDR5, roughly 60GB addressable as VRAM through ROCm 7.2. Cost about $3,000. Fits on a desk.

What They Cost to Run

I logged GPU power draw at 500-millisecond intervals during inference using nvidia-smi on the P40 server and rocm-smi on the Strix Halo. Same prompt, same models, same Ollama configuration. All models ran 100% on GPU.

Model	P40 tok/s	P40 GPU Power	Halo tok/s	Halo GPU Power
Llama 3.2 3B	91.2	170W avg	78.4	64W avg
Llama 3.1 8B	47.5	278W avg	40.2	82W avg
Llama 3.1 70B (4K ctx)	6.3	278W avg	5.6	81W avg

The P40 is 15-18% faster in raw throughput. It draws 3-4x the power. The 3B model lives on a single P40; the other three cards idle at ~9W each but still cost electricity. The 8B and 70B models span two GPUs while two idle. You always pay for cards that are not working. The Strix Halo has one GPU. No idle penalty.

GPU power is not total system power. The P40 server's Xeons, 252GB of RAM, dual PSUs, and fans add roughly 200W to the GPU figures. The Strix Halo's APU and DDR5 add roughly 40-60W. Conservative estimates for total system draw: 500W for the P40 under load, 120W for the Strix Halo.

At Minnesota residential electricity rates ($0.157/kWh), the cost per million tokens:

Machine	3B	8B	70B
P40 Server	$0.19/M	$0.46/M	$3.47/M
Strix Halo	$0.06/M	$0.13/M	$0.94/M

Why the Per-Token Number Is Misleading

Those numbers look competitive with hosted inference, which runs $0.05 to $0.20 per million tokens for 8B-class models through providers like Together AI or Groq. The Strix Halo at $0.13/M sits squarely in that range. The P40 at $0.46/M does not.

But per-token cost during active inference is the wrong metric for two reasons.

Hardware amortization changes the math. The P40 server cost $2,500. The Strix Halo cost $3,000. Amortized over two years, that adds $0.14/hr and $0.11/hr respectively. On the 8B model, the all-in cost per million tokens rises to about $1.28 for the P40 and $0.90 for the Strix Halo. Both are more expensive than every hosted inference API for the same model.

Idle power is the dominant cost. The P40 server draws roughly 340W at idle: $38.50 per month whether I run a single query or not. The Strix Halo draws roughly 35W at idle: $4.20 per month. Over a year, idle electricity alone costs $462 on the P40 and $50 on the Strix Halo. If you are not using the hardware frequently, idle power overwhelms everything else in the cost model.

Per-token math at load flatters local inference by ignoring the hours when the hardware is doing nothing. It is like calculating your car's fuel economy only during highway driving and ignoring that it sits in the driveway 22 hours a day with the engine running.

Why I Run Both Anyway

The per-token economics favor API providers. The per-workload economics favor local hardware for specific tasks. TTS is the starkest example.

Generating a 20-minute blog narration on the Strix Halo takes about 45 minutes of inference at roughly 85W above idle power. The incremental electricity cost is about $0.02. The same narration through Google Cloud TTS would cost $4 to $10 depending on character count and voice tier.

That is a 200-to-500x cost difference on the marginal unit. And the marginal unit is what matters, because the question is never "should I generate TTS at all?" It is "should I regenerate the audio for this post I just edited?" or "should I try a different voice on this article?" or "should I narrate this niche post about PCB trace routing that maybe fifty people will listen to?"

At $4 to $10 per narration, the answer to all of those is "probably not." At $0.02, the answer is "why wouldn't I?" That shift from "probably not" to "why not" is the entire economic argument for owning TTS hardware. It is not about the average cost. It is about the marginal decision.

Before running local TTS, I narrated posts selectively with Google Cloud's Text-to-Speech. Some were too long or too niche to justify the GCP cost. Now every post gets audio. I regenerate after revisions without thinking about it. I have run the same post through three different TTS models to compare voice quality. I experiment with speaker voices, pacing parameters, and chunk sizes. The total volume of audio I have generated locally exceeds what I would have purchased from Google at any price point. This is Jevons Paradox at the smallest possible scale: make TTS cheap enough and I do not produce the same amount of TTS for less money; I produce vastly more TTS for slightly less money.

The same logic applies to code scanning. Any individual scan is cheap enough through an API. But the friction of metered pricing discourages the kind of speculative, exploratory analysis that turns up unexpected findings. When the marginal cost is zero, I scan more freely and more often. The value is not in any single scan; it is in the scans I would not have run otherwise.

The Strix Halo Problem

The most surprising result in the benchmarks is the Strix Halo's efficiency. An integrated APU with no discrete GPU delivers 40.2 tokens per second at 82W of GPU power. The P40 server delivers 47.5 tokens per second at 278W of GPU power. The P40 is 18% faster. The Strix Halo uses 70% less power. In performance per GPU watt, the Strix Halo (0.49 tok/s per watt) is nearly three times more efficient than the P40 (0.17 tok/s per watt).

This creates a problem for the P40 server's economics. The server's advantage is VRAM: 96GB lets it run 120B MoE models that the Strix Halo cannot fit. For the gpt-oss 120B model, the P40 server is the valid option. But for everything 8B and below, the Strix Halo is cheaper to buy ($2,000 vs. $2,500), cheaper to idle ($4.20/month vs. $38.50/month), cheaper per token ($0.13/M vs. $0.46/M), quieter, smaller, and only 18% slower.

If I were building a local inference setup today from scratch and my workload was 8B models and TTS, I would buy the Strix Halo and nothing else. The P40 server justifies its existence only for the large models that need its VRAM and the fact that I put it together well before the current RAM price spike.

This is worth sitting with for a moment, because it inverts the conventional wisdom about inference hardware. The enterprise GPU server that looks impressive on paper (four GPUs, 96GB VRAM, 2U rack mount) loses on total cost of ownership to a $3,000 mini desktop for the workloads that dominate my actual usage. The P40's raw throughput advantage is real but small. Its power cost advantage is negative. The VRAM advantage matters only for models most people do not run.

The Maintenance Tax

The per-token calculations ignore the cost of keeping these machines running. It is not zero.

I have had two kernel updates break the NVIDIA DKMS module on the P40 server. The AMD machine requires specific pre-release PyTorch wheels and environment variable overrides for ROCm to function on gfx1151 hardware. While running the benchmarks for this article, I discovered that Ollama on the Strix Halo had been running entirely on CPU because the systemd service file lacked the HSA_OVERRIDE_GFX_VERSION=11.5.1 variable. Every benchmark I had run on that machine prior to catching this was measuring CPU inference, not GPU inference. The fix took two minutes. Finding it took longer.

The P40 server's fans run at full speed from October through April because the BMC interprets Minnesota winter temperatures as a hardware malfunction. The noise is audible from the house, 150 feet away.

None of this is catastrophic. All of it is time. And time spent debugging DKMS modules or adding environment variables to systemd units is time not spent on the work that the hardware is supposed to enable. A Claude Max subscription requires zero maintenance. The local hardware requires ongoing attention. That asymmetry does not show up in per-token cost tables, but it is real.

Who This Is For

Most people should not build a local inference server. If you use AI for interactive tasks (questions, code, analysis, writing), a frontier model subscription is a better product at a lower total cost than any local setup. The quality gap between a local 8B model and Claude or GPT-5.4 is not closing in the ways that matter for conversational use. Pay for the good models. Use them freely.

Local inference makes economic sense when you have a specific, high-volume, quality-tolerant workload that you will run often enough to justify hardware sitting on 24/7. TTS is the clearest case. Batch code analysis is another. If you cannot name the workload, you do not have one, and the hardware will cost you $40 to $50 per month in idle electricity to find out.

The split between frontier subscriptions and local batch processing is not a compromise. It is, for my usage, the correct architecture. The frontier model handles the work where quality determines value. The local hardware handles the work where volume determines cost. Neither replaces the other. The mistake is thinking they compete.

LLM-Generated Content: What Makes Something Slop

A.C. Jokela — Mon, 16 Mar 2026 13:00:00 GMT

Merriam-Webster named "slop" its 2025 Word of the Year. In its new usage, the word describes low-quality AI-generated content produced and distributed with minimal human oversight. It captures something the internet has been feeling for a while: the growing suspicion that much of what appears online wasn't written so much as emitted.

I should be transparent about where I stand. This blog uses AI-generated text-to-speech narration on every post. The articles about PCB trace routing describe boards that were auto-routed by algorithms. The code that builds and deploys this site was partially written with Claude Code assistance. I wrote a six-part series on Jevons Paradox with AI tools open in the next terminal window the entire time. I am not writing this from outside the system.

And yet I know slop when I see it. You probably do too. The interesting question is not whether slop exists (it obviously does) but what exactly we're recognizing when we encounter it. What quality makes certain AI-generated or AI-assisted content feel hollow, and what distinguishes it from output that has substance? The answer matters, because if we can't articulate the distinction, we're left with a binary that helps nobody: reject all AI tools, or accept everything they produce uncritically.

You Know It When You See It

In 1964, Justice Potter Stewart offered his famous non-definition of obscenity: "I know it when I see it." We're in a similar position with AI slop. Most people can identify it immediately but struggle to explain what they're detecting.

The surface markers are easy enough to catalog. The hedging language: "It's important to note that..." The false balance, presenting every issue as having exactly two equally valid sides. The emoji padding that serves no communicative purpose. The five-paragraph essay structure applied to every topic regardless of complexity. The confident incorrectness: statements delivered with the same breezy authority whether they're true or fabricated. The vocabulary of caution and qualification that reads less like thoughtfulness and more like a language model covering its bases.

These are the tells that AI-detection tools try to measure, and they work well enough for obvious cases. But they're symptoms, not the disease. A skilled prompt engineer can eliminate every one of these markers and still produce slop. Conversely, a human writer can exhibit several of them (hedging, false balance, structural rigidity) and still produce something worth reading. The surface features point toward the problem without being the problem itself.

What we're actually detecting is an absence. Not an absence of quality at the sentence level (LLMs write clean, grammatical sentences) but an absence of something harder to name. The text reads correctly line by line and says nothing paragraph by paragraph. It is fluent without being articulate. It covers a topic without engaging with it. And we recognize this gap almost instantly, the way you recognize a smile that doesn't reach someone's eyes.

The Three Properties

The MINT Lab at Indiana University proposed a useful framework for thinking about this. They identified three properties that characterize slop: superficial competence, asymmetric effort, and mass producibility.

Superficial competence is the core mechanism. The text performs competence at the surface level: vocabulary is appropriate, structure is logical, claims are plausible. But it doesn't demonstrate competence at the level of understanding. There's a difference between a sentence that uses the right words and a sentence that conveys the right meaning. Slop consistently achieves the former while missing the latter. The prose is grammatically flawless and semantically empty, a combination that is almost impossible for human writers to produce at scale but trivially easy for language models.

Think of a student essay that hits every point on the rubric: thesis statement in the right place, three supporting paragraphs, counterargument acknowledged, conclusion that restates the thesis. A teacher reads it and gives it a B+. But the teacher also knows, without being able to point to a specific sentence, that the student didn't learn anything while writing it. The essay demonstrates knowledge of essay structure, not knowledge of the subject. That's superficial competence.

Asymmetric effort describes the production economics. The author (or deployer) invested minimal effort relative to the volume of output. A single prompt generates 2,000 words in seconds. The resulting text has the length and format of something that would take a human writer hours, but it cost nothing in terms of thought, research, or revision. This asymmetry creates an incentive structure where the marginal cost of publishing approaches zero and the quality feedback loop disappears.

Mass producibility follows from the first two. If the text is superficially competent and cheap to produce, there's no natural limit on volume. This is how you get AI-generated recipe blogs with 10,000 pages, product review sites with no evidence of product testing, and news aggregators that rewrite wire stories into blandly authoritative summaries. The content fills a shape (a blog post, a review, a news article) without filling it with meaning.

These three properties interact. Mass production exacerbates the problem of superficial competence because there's no time or incentive for the depth that would distinguish one piece from another. And asymmetric effort means there's no skin in the game: the producer doesn't care whether the content is right, because it cost almost nothing to create and nothing to correct.

Greenberg's Ghost

There's a version of this argument that's eighty-seven years old.

In 1939, Clement Greenberg published Avant-Garde and Kitsch, one of the most influential essays in twentieth-century art criticism. Greenberg argued that mass culture produces "kitsch," art that "pre-digests art for the spectator and spares him effort, provides him with a shortcut to the pleasure of art that detours what is necessarily difficult in genuine art." Kitsch offers "vicarious experience and faked sensations." It looks like art. It has the shape of art. But it demands nothing from the viewer and delivers nothing in return except the comfortable feeling of having consumed something.

AI slop does exactly this with information. It pre-digests knowledge for the reader, offering the appearance of understanding without requiring (or enabling) actual understanding. You read 2,000 words about a topic and come away with the sense that you've learned something, but when you try to articulate what you learned, there's nothing solid to grasp. The text gave you the experience of reading an informative article without the substance of one. Vicarious understanding. Faked insight.

The parallel extends further than you might expect. Greenberg worried that kitsch would overwhelm genuine art because it was cheaper to produce and easier to consume. The same dynamics apply to AI-generated content: it's infinitely cheaper to produce, formats itself for easy consumption, and competes for the same attention as substantive work. Greenberg's nightmare was a culture where the imitation crowds out the real thing. That's recognizably the state of much of the internet in 2026.

But Greenberg was also, let's be honest, a snob. His framework positioned the critic as the essential gatekeeper: only the trained eye could distinguish art from kitsch, and the masses were essentially passive consumers incapable of judgment. This elitism left him unprepared for Pop Art. When Warhol silk-screened Campbell's soup cans and Lichtenstein blew up comic panels to gallery scale, they took the materials of kitsch and made something genuinely interesting from them. They didn't reject mass culture; they engaged with it in a way that Greenberg's binary framework couldn't accommodate.

There's an obvious recursive problem here, and I should name it rather than pretend it doesn't exist. This essay was written with AI assistance. It is, in a direct sense, an attempt to take the materials of mass production (an LLM's facility with argument structure, literature survey, prose drafting) and make something that isn't slop. Whether it succeeds is for the reader to judge. But the attempt itself is the Pop Art move: not rejecting the tools of mass culture, but trying to use them to say something specific. If I fail, the essay is kitsch that thinks it's art. If I succeed, Greenberg's binary was too rigid, and the tool was never the problem.

This tension matters for the slop conversation more broadly. If we define slop as any AI-generated content (regardless of what it does or says), we make the same mistake Greenberg made with kitsch. The question is not the tool; it's whether something is being done with it at all.

The Authenticity Problem

So what is the actual distinguishing quality? What separates writing that happens to involve AI tools from writing that is slop?

It's not voice. LLMs can mimic voice convincingly enough to fool most readers. It's not structure; LLMs organize material at least as well as the average human writer. It's not even factual accuracy, since LLMs can be accurate when properly grounded and cited. These are all necessary conditions for good writing, but slop can satisfy all of them and still be slop.

What's missing is a point of view: the willingness to be wrong about something specific.

Slop hedges. It covers all sides. It presents every position as having merit and declines to choose between them. It never commits to a claim that could be falsified, challenged, or argued against. And this is not a bug in the technology; it's a feature. Language models are trained to be helpful, harmless, and accurate. Helpfulness means addressing the user's question. Harmlessness means avoiding offense. The intersection of these goals produces text that is relentlessly, pathologically balanced. Every "on the one hand" gets an "on the other hand." Every strong claim gets a qualification. The result is prose that cannot be disagreed with, because it doesn't say anything specific enough to disagree with.

I notice this constantly in my own AI-assisted drafts. The first pass comes back with every edge sanded off. Where I wrote "Freerouting can't do copper pours, and that's a fatal limitation for production boards," the draft wants to say "Freerouting has some limitations regarding copper pours that may affect certain use cases." The second version is more cautious. It's also emptier. The editorial work, the part that makes writing not-slop, is putting the edges back on: choosing the stronger claim, deleting the qualifications that exist for safety rather than accuracy, deciding that this is what I actually think and I'm willing to defend it.

Good writing, whether human or AI-assisted, takes a position and defends it. The author exists in the text because they have opinions, not because they have fluency. When I wrote that Jevons Paradox applies to human attention in the context of AI-assisted work, that was a specific, falsifiable claim. You could disagree with it. You could argue the model doesn't apply, or that the historical parallels are misleading, or that the biological ceiling on attention changes the dynamics. The argument creates a surface for friction. It takes a stance that could be wrong.

Slop never takes that risk. It describes all positions and endorses none. It informs without arguing. And because it never commits to anything, it can never be wrong, which means it can never be right either. It occupies a semantic dead zone: technically not false, functionally not true, informationally zero.

This is the test most people are applying intuitively when they identify something as slop. They're asking: is someone home? Does the text have a perspective, or is it just generating plausible sentences? The "someone" doesn't have to be a human, exactly. It has to be a process that made choices: that included some things and excluded others, that decided this interpretation was better than that one. Slop is text produced by a process that made no choices at all, because the defaults were good enough to fill the space.

When AI Output Isn't Slop

If the test is commitment and accountability, then it follows that AI-assisted output can clear the bar. But I want to be specific here, not hand-wavy, because vague appeals to "my own experience" are themselves a slop move.

The Giga Shield project started with a $468 Fiverr design that didn't work. Nine bidirectional level shifters, professional layout, clean two-layer board. Then I tested it with a Z80 processor, and the auto-sensing TXB0108 chips fell apart. The Z80 tri-states its address bus between cycles; the pins go high-impedance, floating. The TXB0108 can't determine drive direction from a floating signal. It guesses wrong, and the Arduino on the other side reads garbage. I'd paid $468 for a board that was blind to half of what the processor was doing.

The redesign used Claude Code to generate the entire replacement board from a Python script: no graphical PCB editor, no manual placement, just code that outputs a routable board file. AI wrote the board generator. AI helped parse the KiCad schematic to extract all 72 signal mappings across 9 ICs. Then Freerouting, an open-source autorouter, handled the trace routing.

Here's the kind of specificity that slop can't contain: after 60 optimization passes (about 45 minutes of compute), Freerouting brought the via count on the Giga Shield from roughly 220 down to 158. I ran 128 parallel instances across three machines with randomized net ordering to explore different regions of the solution space. And still, a hard floor of 5-6 unrouted ground connections remained, because Freerouting's architecture literally cannot represent copper pours, and the 0.65mm-pitch TSSOP-24 packages didn't have physical room for ground vias. That limitation is structural. No amount of prompt engineering or parameter tuning changes the fact that the algorithm has no concept of flood-fill connectivity. I wrote about this in detail, including the A* search internals and the specific geometric constraints, and if I got the analysis wrong, anyone can read the source code and check.

I also discovered that Freerouting v2.1.0 produced 152 unrouted connections on the same board where v1.9.0 produced 6. That's a testable, reproducible claim, attached to specific version numbers, specific board files, specific machines. It's the opposite of "AI autorouting tools can sometimes produce inconsistent results," which is the slop version of the same observation. One of those sentences tells you something. The other fills space.

Even the TTS narration is more complicated than "it just works." The Qwen model mispronounces technical terms. It puts emphasis in odd places. The audio for posts with dense jargon has an uncanny flatness where the model clearly doesn't understand what it's reading. I publish it anyway because it's useful despite its flaws, and because I label it as AI-generated narration, which means I'm not asking the listener to trust it as a human performance. It's a tool with known limitations, deployed for a specific purpose, accountable to its function.

The common thread isn't that AI made these outputs perfect. It's that they were tested against something outside themselves. The board works or it doesn't. The via count is 158 or it isn't. The audio plays or it doesn't. Slop faces no such test. It exists to fill a container, and its success is measured by whether the container looks full, not by whether what's inside is true.

The Compost Argument

There's a reasonable counterargument that goes like this: human slop has always existed. Content farms, SEO spam, airport bookstore filler, corporate press releases, academic papers that exist only to pad a CV. The internet was full of low-quality, low-effort content long before large language models existed. AI didn't invent slop; it industrialized it.

This is true, and it's worth taking seriously. The people arguing that "the idea of AI slop is slop" have a point: if we define slop as low-quality content produced with minimal effort, most of what humans have ever published qualifies. Sturgeon's Law (ninety percent of everything is crud) predates AI by decades.

But the economics are different now, and economics change everything. When slop required human labor, there was a floor on production cost. A content farm still had to pay writers (however little). An SEO spammer still had to hire someone to string keywords into sentences. That floor limited volume, which limited the ratio of noise to signal in any given information ecosystem.

AI removes the floor. The marginal cost of producing a 2,000-word article drops to fractions of a cent. The marginal cost of producing 10,000 such articles drops to the cost of an API call and a deployment script. The constraint was never willingness to produce slop; it was cost. With cost eliminated, volume expands without bound. This is, incidentally, another case of Jevons Paradox: make content production cheaper, get more content production, not less.

Some writers have made what I'll call the compost argument: that cultural slop, even the human-produced kind, serves as a sort of fertilizer. The vast majority of pulp fiction was forgettable, but it created the ecosystem that produced Philip K. Dick and Ursula K. Le Guin. Most blog posts are unremarkable, but the blogging ecosystem produced some genuinely important writing. The compost nourishes rare blooms.

Maybe. But there's a concentration problem. A garden benefits from compost; a garden buried under six feet of compost is just a landfill. If the ratio of slop to substance shifts far enough, the substance becomes unfindable. Search engines surface slop because it's optimized for surfacing. Recommendation algorithms amplify it because engagement metrics can't distinguish between "I read this and learned something" and "I read this and it filled two minutes." The signal doesn't just get drowned out; it gets algorithmically deprioritized in favor of the noise.

What the Test Looks Like

I've argued that what we recognize as slop is the absence of commitment: text that declines to be wrong about anything specific. I believe this is correct, but I should be honest about where the test gets uncomfortable.

Committed writing can be terrible. Conspiracy theories are committed. Propaganda is committed. A confidently wrong blog post about vaccine microchips passes the "takes a position" test with flying colors. Commitment is necessary but not sufficient. It separates slop from writing that has a pulse, but it doesn't separate good writing from bad writing. That's a different and older test, one that involves accuracy, evidence, reasoning, and intellectual honesty: all the things we've always used to evaluate arguments. The slop test is prior to all of that. It asks whether there's anything present to evaluate in the first place.

The tool doesn't determine the category. The commitment does. And if this essay has failed to commit to anything worth arguing against, then by its own logic, it belongs in the landfill with the rest.

The Mathematics of PCB Trace Routing

A.C. Jokela — Sun, 15 Mar 2026 16:00:00 GMT

Every PCB design eventually arrives at the same moment. Components are placed. Nets are defined. The ratsnest of thin lines connecting pad to pad looks like a plate of spaghetti dropped on a cutting board. Now someone, or something, has to turn that mess into real copper traces that don't cross, don't short, and fit within the design rules. That's the routing problem.

For hobbyists and professionals alike, autorouters do this work. You press a button, wait, and traces appear. But what actually happens during that wait? The answer turns out to involve some of the most elegant mathematics in computer science, and some surprisingly hard geometric constraints that no algorithm can finesse.

I've been using Freerouting, the open-source Specctra autorouter, for two PCB projects now: a dual Z80 RetroShield and a level-shifter shield for the Arduino Giga R1. The second project pushed Freerouting to its limits in ways that forced me to understand how it works internally. This is what I found when I read the source code.

Not a Grid, Not a Maze

Giga Shield routed by Freerouting in 45-degree mode. Top layer, rendered in pcb-rnd.

Most descriptions of PCB autorouting start with Lee's maze algorithm from 1961. Place the board on a grid. Flood-fill from the source pad. When the wave hits the destination, backtrack along the shortest path. It's intuitive, easy to implement, and used in introductory EDA courses everywhere.

Freerouting doesn't do this.

Instead of discretizing the board into a grid of cells, Freerouting operates on a continuous geometric plane. The routing space is partitioned into convex polygonal regions called expansion rooms. Each room is a chunk of free space on one layer of the board, bounded by the edges of existing obstacles (traces, vias, pads) plus their clearance halos. The rooms aren't precomputed. They're generated lazily during the search, grown on demand as the router explores new areas.

This is a shape-based router, sometimes called a free-space router. The distinction matters. A grid-based router's resolution is fixed: if your grid is 0.1mm, you can't route a trace at 0.05mm offset from an obstacle, even if the design rules would allow it. A shape-based router has no such limitation. It works with exact geometry (integer-valued coordinates for precision), and the routing channels it discovers are as wide or narrow as the physical clearances actually allow.

Three geometry modes control the shape of the rooms:

Mode	Room Shape	Allowed Trace Angles
90-degree	Axis-aligned rectangles	Horizontal, vertical
45-degree	Octagons	Plus 45-degree diagonals
Any-angle	General convex polygons	Unrestricted

The choice affects both routing quality and performance. Axis-aligned rectangles are fastest to compute and intersect. Octagons allow the 45-degree traces common in modern PCBs. General polygons give the router maximum freedom but at a computational cost.

Same board in any-angle mode. Traces follow direct paths instead of 45-degree snapping.

The image to the left shows the same board routed in any-angle mode. Notice how traces leave pads at arbitrary angles, following straight-line paths toward their destinations rather than snapping to a 45-degree grid. Compare this with the image above, which used the standard 45-degree octagon mode. The any-angle result has shorter total trace length but can be harder to manufacture cleanly at tight tolerances.

The A* Core

At its heart, Freerouting's search algorithm is A*, the same algorithm that drives pathfinding in video games, robot navigation, GPS routing, and network packet delivery. A* was published by Peter Hart, Nils Nilsson, and Bertram Raphael at the Stanford Research Institute in 1968. Nearly sixty years later, it remains the standard algorithm for finding shortest paths in weighted graphs where a heuristic estimate of remaining distance is available.

The mathematical foundation is straightforward. A* maintains a priority queue of candidate states, each with a cost value:

$$f(n) = g(n) + h(n)$$

Where:

g(n) is the actual accumulated cost from the start to state n. In PCB routing, this includes trace length, layer changes (vias), preferred-direction penalties, and any ripped-up obstacle costs.
h(n) is a heuristic estimate of the remaining cost from n to the destination. This must be admissible: it must never overestimate the true remaining cost.
f(n) is the total estimated cost of the best path through n.

At each step, A* pops the state with the lowest f(n) from the queue, expands its neighbors, and adds them back with updated costs. When the destination is popped, the algorithm has found the optimal path (given an admissible heuristic).

The key insight is the heuristic. Without it, A* degenerates into Dijkstra's algorithm, which explores in all directions equally. A good heuristic focuses the search toward the destination. In Freerouting's case, DestinationDistance.calculate() estimates the minimum cost to reach the target, accounting for both planar distance and any required layer transitions. The sorting value in the priority queue is computed as:

double sorting_value = expansion_value
    + this.destination_distance.calculate(shape_entry_middle, layer);

Where expansion_value is the g(n) accumulated cost, and the distance calculation is h(n). This is textbook A*.

Why A* Works So Well

A* has a remarkable optimality guarantee. If the heuristic h(n) is admissible (never overestimates), A* is guaranteed to find the shortest path. If h(n) is also consistent (satisfying the triangle inequality: h(n) <= cost(n, n') + h(n') for every neighbor n'), then A* never needs to re-expand a state it has already visited. This makes it both optimal and efficient.

For PCB routing, the Euclidean distance between the current position and the destination pad is a natural admissible heuristic: a straight line is always shorter than any actual route that must navigate around obstacles. Freerouting's heuristic is somewhat more sophisticated, incorporating via costs for layer transitions, but the principle is the same.

The efficiency gain over brute-force search is dramatic. Dijkstra's algorithm (A* with h(n) = 0) explores states in concentric rings outward from the source. On a board with N searchable regions, it visits O(N) states. A* with a good heuristic carves a narrow corridor from source to destination, visiting far fewer states. In practice, on a moderately complex board, this is the difference between milliseconds and minutes per connection.

A* Is Everywhere

The same algorithm, with different cost functions and heuristics, solves an astonishing range of problems:

Game pathfinding. Every real-time strategy game since the 1990s uses A* to move units around obstacles. The grid cells are the states, movement cost is g(n), and Manhattan or Euclidean distance to the target is h(n).

GPS navigation. Road networks are weighted graphs. Edge weights are travel times. A* with geographic distance as the heuristic finds near-optimal routes across millions of road segments.

Robot motion planning. A robot's configuration space (position, orientation, joint angles) is the state space. A* finds collision-free paths from one configuration to another.

Natural language processing. Viterbi decoding, which finds the most likely sequence of hidden states in a Hidden Markov Model, is structurally similar to A* over a trellis graph.

Puzzle solving. The 15-puzzle, Rubik's Cube, Sokoban. A* with an appropriate heuristic solves them all optimally, and the heuristic is what makes the search tractable rather than exponential.

What makes A* general is the abstraction. It doesn't care whether the "states" are grid squares, road intersections, robot poses, or polygonal rooms on a PCB layer. It only needs a cost function, a heuristic, and a neighbor-expansion rule. Freerouting provides all three, with the unusual twist that its states are dynamically-computed convex polygons rather than fixed graph nodes.

But A* Only Routes One Net

Here's the catch. A* finds the optimal path for a single source-destination pair. A PCB has hundreds of nets, all competing for the same physical space. Route net A first, and it might block the optimal path for net B. Route net B first, and net A suffers instead. The quality of the overall routing depends heavily on the order in which nets are processed.

Freerouting handles this with rip-up-and-reroute, a strategy from the 1970s that remains the standard approach. The idea is simple: route all nets in some initial order. When a net fails (no path exists without violating design rules), rip up one or more blocking traces and add them to a retry queue. Then try again with different priorities.

The implementation in BatchAutorouter.java runs multiple passes over the board. On each pass, every unrouted connection is attempted. The critical detail is how ripup decisions are made. Each existing trace has a ripup cost, and the cost increases linearly with the pass number:

ripup_cost = start_ripup_costs * passNumber

Early passes are conservative: the router avoids tearing up existing routes. Later passes become progressively more aggressive, willing to rip up more traces to find solutions. This is a controlled escalation that prevents the router from thrashing (endlessly ripping and re-routing the same nets) while still allowing it to escape local minima.

The scheduler also implements a limited form of backtracking. Every few passes, the router checks whether the board score (total unrouted connections, via count, trace length) has improved. If not, it restores a previously saved board snapshot and continues from that earlier state. This is a coarse approximation of simulated annealing: occasionally accepting a worse intermediate state to explore a different region of the solution space.

Net Ordering: The Hidden Variable

The order in which nets are routed has an outsized effect on the result. By default, Freerouting routes nets in the order they appear in the DSN file, which is typically the order they were defined in the schematic. There's no sorting by airline length, fan-out degree, or criticality. The router's source code contains a commented-out sort-by-distance that was disabled in v2.3 because it "negatively impacts convergence."

This means the same board can produce different routing results depending on how the DSN file was generated. I exploited this during the Giga Shield project by writing a script (shuffle_dsn.py) that generates dozens of copies of the same DSN file with randomized net ordering:

for i in range(n_copies):
    shuffled = nets.copy()
    random.seed(i * 31337 + 42)
    random.shuffle(shuffled)
    # write shuffled DSN...

Each copy routes nets in a different sequence, converging to a different local optimum. Running 128 parallel Freerouting instances across three machines (a local Mac, a 64-core server, and a 32-core workstation) explored 128 different regions of the solution space simultaneously. The best result was measurably better than any single run. This is an embarrassingly parallel optimization: each job is independent, and you keep the best answer.

The takeaway: if your autorouter isn't finding a clean solution, the problem might not be the algorithm. It might be the ordering. Changing the input order is cheaper than changing the router.

The Optimization Phase

After the initial routing passes, Freerouting enters an optimization phase controlled by the -mp flag. This phase iterates over every existing via and trace in the design, processing them in a left-to-right spatial scan.

For each item, the optimizer:

Rips up the item's entire connection (all traces and vias for that net segment)
Re-runs up to 6 passes of the A*-based autorouter on just that connection
Accepts the result only if it reduces via count or total trace length
Restores the previous state if the re-route was no better

Vias are visited before traces, reflecting the priority of via reduction. Each unnecessary via adds manufacturing cost, signal integrity degradation, and parasitic capacitance. The optimizer also alternates between preferred and non-preferred trace directions on successive passes, preventing the solution from getting stuck in a directional rut.

Via positions themselves are fine-tuned by a separate algorithm (OptViaAlgo). For vias connecting exactly two traces, the optimizer searches for the position that minimizes the combined weighted trace length on both layers, iteratively nudging the via toward the geometric optimum.

The result of the optimization phase is typically a 15-30% reduction in via count and a 10-20% reduction in total trace length compared to the initial routing. On the Giga Shield, 60 optimization passes ran for about 45 minutes and brought the via count from ~220 down to ~158.

Why Freerouting Can't Do Copper Pours

This is where the elegance of the algorithm runs headfirst into a hard architectural limit.

Every non-trivial PCB has a ground net that connects to dozens or hundreds of pads. The standard solution in commercial EDA tools is a copper pour: a filled polygon that covers an entire layer (or most of it), with clearance cutouts around non-ground features and thermal relief connections to ground pads. You don't route GND with traces. You flood-fill it.

Freerouting cannot do this.

The limitation isn't a missing feature that could be added with a few hundred lines of code. It's structural. Freerouting's entire architecture is built around point-to-point trace routing. The maze search, the rip-up scheduler, the optimizer: they all operate on individual connections between pairs of pads. A copper pour is a fundamentally different object. It's not a path from A to B. It's a region that grows to fill available space, adapting its shape around every obstacle on the layer.

In the source code, copper pours are represented as ConductionArea objects with a fixed shape set at import time. When the autorouter encounters a net that already has a ConductionArea, it simply returns CONNECTED_TO_PLANE and considers the job done. There's no flood-fill algorithm. There's no thermal relief generation. The router expects that the EDA tool (KiCad, pcb-rnd, etc.) has already computed the pour geometry before the DSN file was exported.

For foreign nets (anything that isn't the pour's net), the ConductionArea is treated as a hard obstacle. Traces can't cross it. Vias can't be placed inside it. The router routes around it as if it were a solid wall. This is exactly right from a clearance perspective, but it means the router has no ability to create, modify, or extend a pour during the routing process.

The practical impact is severe for boards with fine-pitch surface-mount parts. On the Giga Shield, each SN74LVC8T245PW (TSSOP-24) has three GND pins at 0.65mm pitch. The gap between adjacent pads is roughly 0.25mm. A via needs approximately 0.9mm of space (drill diameter plus annular ring plus clearance). There is physically no room to place a via next to a TSSOP-24 GND pad and connect it to a GND trace on another layer. The router can see the GND pad, it can see that it needs to be connected to other GND pads, but it cannot find a valid path because there is no valid path using its vocabulary of traces and vias.

A copper pour solves this trivially. The pad sits directly on (or thermally connects to) the pour polygon. No via needed. No trace routing needed. The connectivity is implied by physical overlap. But this is a concept that simply doesn't exist in Freerouting's model of the world.

On the Giga Shield project, this limitation manifested as a hard floor of 5-6 unrouted GND connections that no amount of optimization could resolve. I threw 128 parallel instances at the problem across three machines. I tried 2-layer, 4-layer, and 6-layer board configurations. I wrote custom post-processing scripts to add GND vias and MST-based bottom-layer routing. None of it worked within DRC constraints. The geometry was simply too tight. We ended up solving it with a different tool entirely, which is a story for the next article in this series.

The Shove Machine

Bottom layer. The shove algorithm packs traces tightly between through-hole pin rows.

One of Freerouting's more sophisticated subsystems is its forced insertion with shove mechanism. When the A* search finds that the optimal path for a new trace passes through space occupied by an existing trace, the router doesn't immediately give up or rip up the obstacle. Instead, it tries to push the obstacle aside.

The ForcedPadAlgo and ShoveTraceAlgo classes implement this recursively. When a new trace needs to go where an existing trace is, the existing trace is nudged perpendicular to the new trace's path. If that nudge collides with a third trace, the third trace is nudged too, and so on, up to a configurable recursion depth (default: 20 levels for traces, 5 for vias). Only if the shove cascade exceeds this depth does the router fall back to ripping up the blocking item.

This is the routing equivalent of parallel parking in a tight spot. Instead of abandoning the space, you bump the neighboring cars just enough to fit. It produces much denser routing than a pure rip-up approach, especially on boards with tight clearances and many competing nets.

After every trace insertion, a pull-tight pass (PullTightAlgo) smooths and shortens all traces in the affected area. This is a local optimization that removes unnecessary corners, straightens diagonal segments, and reduces total trace length. The combination of global A* search, local shove, and pull-tight smoothing produces routing quality that is competitive with commercial autorouters.

Clearance Compensation: Geometry Trick

One implementation detail worth highlighting is how Freerouting handles clearance checking. Rather than testing "does this trace violate clearance with that via?" as a separate geometric predicate, Freerouting inflates every item's shape by its clearance value when storing it in the search tree. A trace with 0.254mm clearance is stored as a shape 0.254mm wider on each side. A via with 0.127mm clearance is stored as a circle 0.127mm larger in radius.

This transforms all clearance checks into simple overlap tests. If two inflated shapes overlap in the search tree, there's a clearance violation. If they don't, there isn't. No separate clearance computation is needed during routing. The free-space rooms computed by the maze search are automatically clearance-legal by construction, because they're defined as the gaps between pre-inflated obstacles.

This is an instance of the Minkowski sum from computational geometry. The inflated obstacle shape is the Minkowski sum of the original shape and a disc of radius equal to the clearance. The free space is the complement of the union of all inflated obstacles. It's mathematically clean and computationally efficient.

Strengths and Weaknesses

After reading through the source and pushing the router to its limits, here's my honest assessment.

Strengths:

Gridless geometry. The shape-based approach produces routing that uses space optimally, without the artifacts of grid snapping. Traces can be placed at any position and any angle (in the selected mode), not just on grid points.
Mathematically sound core. The A* search with admissible heuristic guarantees optimal single-net routing. The rip-up-and-reroute scheduler provides a practical framework for multi-net optimization. These are well-understood algorithms with decades of theoretical backing.
Shove + pull-tight. The forced insertion mechanism and post-routing optimization produce dense, clean routing that competes with commercial tools for signal traces.
Reproducibility. Deterministic algorithm, text-based input/output, command-line interface. Same input always produces the same output. You can script it, parallelize it, and integrate it into CI pipelines.
Open source. You can read the code, modify the cost functions, change the heuristics, rebuild for different Java versions, and understand exactly what the tool is doing. That's rare in EDA.

Weaknesses:

No copper pour support. The most significant limitation. Any board with a meaningful ground net requires manual post-processing or a different tool for GND connectivity. This eliminates Freerouting from the running for most production boards with fine-pitch ICs.
Single-threaded core. The maze search is inherently sequential. Multi-threading exists in the codebase but only at the item level (different connections routed by different threads), not within the search itself. On modern multi-core machines, this leaves most of the CPU idle.
Net ordering sensitivity. The same board produces meaningfully different results depending on input order, with no built-in intelligence about which order is likely to be best. The disabled sort-by-distance suggests the developers tried and found it counterproductive.
GUI initialization in batch mode. Freerouting's Swing UI code initializes even when running headless with -de/-do flags. On servers without X11, this requires xvfb or a virtual framebuffer, adding deployment complexity to what should be a pure command-line tool.
Version regression. Freerouting v2.1.0 produced dramatically worse results than v1.9.0 on the same board (152 unrouted vs 6). The newer version isn't always better.

What's Next

The board you see in the images above was v0.2: nine SN74LVC8T245PW shifters, 72 channels, fully routed by Freerouting on two layers. I was ready to submit it to PCBWay for fabrication. Then I counted the GPIO pins one more time.

The Arduino Giga R1 has 76 digital I/O pins that need level shifting, plus a handful of analog and control lines. Nine 8-channel shifters give you 72 channels. That's not enough. I was four signals short. The board needed a tenth IC, which meant reworking the layout, adding more decoupling caps, and re-routing everything. The v0.2 design that Freerouting had spent hours optimizing was going in the bin.

With ten shifters instead of nine, the board got denser. The GND problem got worse. And the copper pour limitation that was already a hard floor at 5-6 unrouted connections on the 9-IC board became completely impassable on the 10-IC version. I threw 128 parallel Freerouting instances at it across three machines. I tried 2-layer, 4-layer, and 6-layer configurations. I wrote custom post-processing scripts for MST-based ground routing and copper pour stitching. None of it produced a clean board within DRC constraints.

The solution came from an unexpected direction: Quilter.ai, an AI-powered PCB router that understands copper zones. It routed the 10-IC, 6-layer board with zero unrouted nets on the first attempt. The full story of that journey, from massively parallel Freerouting across a home lab cluster to the moment Quilter solved it in one shot, is coming in Part 2 of the Giga Shield redesign series. If the mathematics of A* is the beauty of PCB routing, the GND problem is where theory meets the physical constraints of 0.65mm-pitch IC packages, and the theory blinks first.

The source code for all of this, including the board generator, the net shuffler, the parallel routing scripts, and the post-processing tools, is available in the giga-shield repository.

Processing 51,000 Photos with AI on AMD Strix Halo

A.C. Jokela — Sat, 14 Mar 2026 17:00:00 GMT

I have roughly 20 years of photos sitting on a home fileserver. They span 2001 to 2020, shot on everything from a Minolta DiMAGE F100 to a Nikon D5100 to various iPhones over the years. A mix of 21,554 JPEGs and 29,860 Nikon RAW files (51,414 images total) organized in a Lightroom backup directory by year, month, and date. Most were shot handheld, many in a hurry. The kind of archive that accumulates when you take photos for two decades without ever going back to curate them.

The Lightroom catalog that once made sense of all this was long gone, lost to a drive migration somewhere around 2018. What remained was a directory tree of raw files with no organization beyond the date folders. No star ratings, no keywords, no collections. Just files. Thousands of them, some sideways, some crooked, all unlabeled.

I wanted to fix that. Not manually (I don't have a month to spend in Lightroom) but programmatically. The goals were straightforward: correct orientation issues, straighten crooked horizons, generate AI descriptions of every photo's content, and catalog the whole archive in a queryable database. The kind of batch processing job that would have been impractical five years ago but is now entirely doable with the right hardware and a weekend of scripting.

The Hardware

Two machines on the local network, each with a distinct role:

Machine	Role	Key Specs
Fileserver	NAS / photo storage	28TB RAID (`/md0`), 125GB RAM, NFS exports
GPU workstation	ML inference	AMD Ryzen AI Max+ 395, Radeon 8060S, 121GB RAM

The fileserver is a straightforward storage box. The interesting machine is the GPU workstation running an AMD Strix Halo APU, specifically the AI Max+ 395 with its integrated Radeon 8060S. I've written about this chip before, and it continues to impress for inference workloads. The RDNA 3.5 integrated GPU shares system memory, giving it access to 65.2 GB of VRAM without the typical constraints of a discrete card. For a model like BLIP that needs maybe 2 GB, that's absurdly generous, but it means you never have to think about VRAM budgets, which is a luxury when you're iterating on a processing pipeline.

The fileserver already had NFS configured, exporting /md0 to the local subnet. One mount command on the GPU workstation and both machines could see the same filesystem:

sudo mount -t nfs fileserver.localnet:/md0 /md0

No file copying, no rsync scripts, no staging directories. The photos live on the NAS and get processed in-place over the network. Gigabit Ethernet introduces some I/O overhead (each 25 MB NEF file takes 200–300ms to read across the wire), but for an overnight batch job, the simplicity of a single shared filesystem is worth the throughput trade-off. If this were a recurring workflow, I'd invest in 10GbE, but for a one-time archive processing run, gigabit got it done.

The Software Stack

Everything runs in a Python virtual environment on the GPU workstation:

PyTorch 2.9.1+rocm6.3: ML framework with AMD ROCm backend
BLIP (Salesforce/blip-image-captioning-large): vision-language model for image captioning
OpenCV 4.13: horizon detection via Canny edge detection and Hough transforms
rawpy 0.26.1: Nikon NEF/NRW decoding (wraps LibRaw)
piexif: EXIF metadata extraction for JPEGs
exiftool: EXIF extraction for RAW files (called as a subprocess)
SQLite: metadata and results database

The gfx1151 Situation

If you've followed my previous posts on Strix Halo, you know the drill. The Radeon 8060S reports as gfx1151 in ROCm, which is newer than what PyTorch's ROCm wheels officially target. The fix is the same environment variable override:

export HSA_OVERRIDE_GFX_VERSION=11.0.0

This maps the GPU to a generic gfx11 target. In practice, it works without issues, with no compute errors and no performance penalties. ROCm 6.16 on this machine also reports amdgcn-amd-amdhsa--gfx11-generic as a supported ISA, which is likely why the override works cleanly. I've been running production workloads with this flag for months now without incident.

The Processing Pipeline

Each photo passes through five stages: EXIF extraction, orientation correction, horizon detection and straightening, AI captioning, and finally saving the corrected image and cataloging everything in SQLite.

EXIF Metadata Extraction

For JPEGs, piexif reads the embedded EXIF data directly; it's a pure Python library that parses the binary EXIF structure without needing any external dependencies. For NEF/NRW files, piexif can't handle Nikon's proprietary container format, so I shell out to exiftool with JSON output (exiftool -json -n <file>). The -n flag is important; it returns numeric values instead of human-readable strings, which makes downstream processing much cleaner.

The extracted fields cover the full gamut: camera make and model, lens, dates, exposure settings (shutter speed, aperture, ISO, focal length), flash, white balance, metering mode, GPS coordinates, and the original orientation tag.

EXIF data is notoriously inconsistent across two decades of cameras. I'll come back to this; it became a debugging story of its own.

Orientation Correction

The EXIF orientation tag (values 1 through 8) encodes how the camera was held when the photo was taken. A value of 1 means the image is right-side up. A value of 6 means the camera was rotated 90 degrees clockwise. Value 3 means 180 degrees. Some values encode horizontal or vertical flips. The full matrix looks like this:

ops = {
    2: [Image.FLIP_LEFT_RIGHT],
    3: [Image.ROTATE_180],
    4: [Image.FLIP_TOP_BOTTOM],
    5: [Image.FLIP_LEFT_RIGHT, Image.ROTATE_270],
    6: [Image.ROTATE_270],
    7: [Image.FLIP_LEFT_RIGHT, Image.ROTATE_90],
    8: [Image.ROTATE_90],
}

Out of the 51,411 successfully processed photos, 8,797 (17.1%) needed orientation correction. The majority came from the Nikon D5100 and iPhone 4, both of which set the orientation tag but don't bake the rotation into the pixel data itself. Without this correction, nearly one in five photos would display sideways or upside-down in any viewer that doesn't respect EXIF orientation.

Here's what that looks like in practice. The raw pixel data from this iPhone photo is stored sideways; the camera recorded an EXIF orientation tag of 6, meaning "rotate 90 degrees clockwise to display correctly." Any viewer that ignores that tag renders the image on its side:

Before: raw pixel data (EXIF orientation 6, displayed sideways)

After: orientation corrected based on EXIF tag

Horizon Detection and Straightening

This stage uses classical computer vision, no neural network needed. The approach:

Downscale the image to 1200px on the long side for speed
Convert to grayscale, apply Gaussian blur
Run Canny edge detection
Crop to the vertical middle 50%, since the horizon is rarely at the extreme top or bottom of a frame
Apply the Hough Line Transform to find line segments, requiring a minimum length of one-quarter the image width
Filter to near-horizontal lines (within 20 degrees of level)
Compute a weighted average of the detected angles, weighted by line length

The key is the threshold window. If the detected angle is less than 0.5 degrees, it's not worth correcting, since you'd introduce interpolation artifacts for no visible benefit. If it's greater than 15 degrees, it's probably not a tilted horizon at all; it's either intentional composition or the algorithm latching onto a staircase railing. The correction itself uses cv2.warpAffine with Lanczos interpolation and a reflective border mode, followed by an inward crop to eliminate any border artifacts:

crop_factor = 1.0 / (cos(angle) + sin(angle) * (min(h, w) / max(h, w)))

The initial implementation used Canny edge detection and Hough line transforms, classical CV techniques from the 1980s. Fast, deterministic, 100ms per image. But it had a fatal flaw: it couldn't distinguish between a tilted horizon and a roofline receding toward a vanishing point. Architecture, roads, staircases, any strong line in the middle band of the image would register as a "tilted horizon," and the algorithm would dutifully rotate the image to "correct" it. In practice, this meant a significant number of photos were being made worse, not better.

The fix was to replace Hough line detection with semantic segmentation. SegFormer (nvidia/segformer-b2-finetuned-ade-512-512), trained on the ADE20K dataset, segments each image into 150 classes, including sky. The approach is simple: find the sky pixels, trace the bottom edge of the sky region, fit a line to that boundary, and measure its angle. If there's no sky (less than 5% of the image), or the sky boundary is too fragmented (fewer than 20 points), skip the correction entirely.

This eliminates false positives on indoor shots, close-ups, architecture, and anything without a visible sky. SegFormer runs on CPU at about 0.4 seconds per image; the model is only 25M parameters, so it doesn't need the GPU. The GPU stays dedicated to BLIP captioning.

Two examples from the corrected archive. This bridge over a river had a 2.68-degree clockwise tilt, and the bridge deck and far shore are visibly leveled:

Before: 2.68° clockwise tilt

After: horizon straightened via sky boundary detection

This rocky Lake Superior shore had a 3.85-degree clockwise tilt, and the far horizon is leveled:

Before: 3.85° clockwise tilt

After: horizon straightened

AI Captioning with BLIP

The Salesforce/blip-image-captioning-large model generates natural language descriptions of each photo. It runs in float16 on the Radeon 8060S. Each image is resized to a maximum of 1024px before inference. Beam search with 5 beams and a 75-token limit generates the caption:

output_ids = model.generate(
    **inputs,
    max_new_tokens=75,
    num_beams=5,
    early_stopping=True,
)

Caption inference takes about 0.5–0.7 seconds per image, consistent regardless of whether the input was a JPEG or a decoded NEF. The model handles a wide variety of subjects surprisingly well. Some examples from the archive:

"a brown and white dog standing next to a blue chair"
"two silos sitting in the middle of a field"
"a bird sitting on a branch of a tree"
"a wooden sign that says hoban road in front of some trees"
"a blurry photo of a car driving down a snowy road"
"a dog being groomed by a woman in a salon"

The captions tend toward a "there is a..." pattern, and they occasionally get details wrong (BLIP once described a photo of my living room as "a hotel lobby," which is generous). But for searchability and cataloging purposes, they're remarkably useful. Being able to query WHERE caption LIKE '%dog%' across 51,000 photos and get meaningful results is something that would have required manual tagging before models like BLIP existed. For an archive this size, "good enough" captions on every photo are vastly more useful than perfect captions on none of them.

Save and Catalog

Corrected images are saved as high-quality JPEGs (quality 92) to /md0/photos_processed/images/, mirroring the original directory structure. NEF and NRW files are converted to JPEG in the process; the corrected archive is a uniform format. All metadata flows into a SQLite database with WAL journaling, tracking 40+ fields per photo: every piece of EXIF data, processing flags (was orientation corrected? was the horizon straightened? by how many degrees?), the AI caption, file hashes, dimensions, and processing timestamps.

The database makes the archive queryable in ways that were never possible before:

-- What cameras did I use, and when?
SELECT camera_model, MIN(date_taken), MAX(date_taken), COUNT(*)
FROM photos GROUP BY camera_model ORDER BY COUNT(*) DESC;

-- Photos with GPS data
SELECT filename, caption, gps_latitude, gps_longitude
FROM photos WHERE gps_latitude IS NOT NULL;

-- How crooked were my photos, by camera?
SELECT camera_model, ROUND(AVG(ABS(horizon_angle_degrees)), 2) as avg_tilt, COUNT(*)
FROM photos WHERE horizon_corrected = 1
GROUP BY camera_model ORDER BY avg_tilt DESC;

The EXIF Tuple Bug

The first processing pass completed 51,414 photos, but with 2,146 errors. All of them were TypeError: type tuple doesn't define __round__ method. For a pipeline that had been running cleanly on thousands of Nikon D5100 and D60 photos, this was unexpected.

The root cause turned out to be a two-part problem with how certain budget cameras from the 2008–2012 era write EXIF rational numbers.

Part 1: Malformed Tuples

The EXIF standard stores rational numbers as (numerator, denominator) pairs. Most cameras follow this. But some, particularly a batch of older point-and-shoots, wrote the ExposureBiasValue field as a 4-element tuple like (36, 0, 18, 0) instead of the expected 2-element (36, 0).

My _rational_to_float helper only handled 2-tuples:

def _rational_to_float(val):
    if isinstance(val, tuple) and len(val) == 2:
        if val[1] == 0:
            return None
        return val[0] / val[1]
    return val  # passes through 4-tuples as raw tuples

When a 4-tuple fell through, the downstream round() call choked on it. The fix was simple: return None for any tuple that isn't a standard rational pair.

Part 2: None Propagation

Even after fixing Part 1, many of these same cameras had written (36, 0), a rational with a zero denominator. The function correctly returned None for division by zero, but the calling code then did round(None, 2), triggering the same TypeError with a slightly different message.

The fix was a _safe_round wrapper:

def _safe_round(val, digits=1):
    if val is None:
        return None
    try:
        return round(val, digits)
    except TypeError:
        return None

After both fixes, the second pass recovered all 2,143 photos. The remaining 3 errors were genuine file corruption: a truncated JPEG, a NEF that LibRaw couldn't parse, and a NEF with filesystem-level I/O errors. Probably bad sectors on the source drive. Those can't be fixed in code.

This is one of those bugs that only surfaces at scale. Run the pipeline on a hundred Nikon photos and everything works perfectly. Run it on 51,000 photos spanning 15 different camera models over 20 years, and every edge case in the EXIF spec comes out to play. The lesson, which I should have internalized long ago: never trust external data formats at scale without defensive parsing on every field. The EXIF spec is a suggestion, not a contract, and camera manufacturers have been interpreting it creatively since the early 2000s.

Resumability

A 15-hour batch job will inevitably need to be restarted: bugs, system updates, a random hound disconnects the magsafe power cord from my MacBook Pro. The script tracks progress in SQLite and skips completed files on restart:

def is_already_processed(conn, source_path):
    row = conn.execute(
        "SELECT id FROM photos WHERE source_path = ? AND error IS NULL",
        (source_path,),
    ).fetchone()
    return row is not None

Photos that failed with errors are intentionally not skipped, so fixing a bug and re-running automatically retries them. This made the EXIF debugging cycle painless: fix the parser, clear the failed rows from the database, relaunch, and only the 2,143 affected photos get reprocessed.

Performance

The pipeline sustained 1.0–1.8 photos per second, depending on file format:

Stage	Time per Photo
JPEG load	~10ms
NEF decode (rawpy)	~400ms
MD5 hash	~5ms (JPEG), ~100ms (NEF)
Horizon detection	~100ms
BLIP inference	~500–700ms
JPEG save	~50ms

BLIP inference dominates the runtime. NEF decoding is the second bottleneck; each RAW file is 20–30 MB and requires full demosaicing through LibRaw. The NFS overhead for reading large NEFs over gigabit Ethernet is noticeable but not the primary constraint.

Total wall time: 15.5 hours across two passes for 51,414 photos. The BLIP model uses roughly 2 GB of the 65.2 GB available VRAM on the Strix Halo. Memory was never a concern.

Final Results

Metric	Count	Percentage
Total photos	51,414	100%
Successfully processed	51,411	99.99%
Orientation corrected	8,797	17.1%
Horizon straightened	15,251	29.7%
AI captioned	51,411	99.99%
Unrecoverable errors	3	0.006%

The top cameras in the archive tell the story of 20 years of gear:

Camera	Photos
Nikon D5100	24,073
Nikon D60	8,734
iPhone 4	2,664
Nikon D3100	1,698
Panasonic DMC-FX07	975
Minolta DiMAGE F100	870
iPad	803
iPhone 5s	698
Samsung SCH-I500	645

The output lives on the NAS:

Corrected images: /md0/photos_processed/images/, 51,411 JPEGs preserving the original year/month/date folder structure, all NEFs converted, all orientation and horizon corrections applied.
SQLite database: /md0/photos_processed/photos.db, 40+ fields per photo with full EXIF metadata, processing results, and AI-generated captions.
Processing log: /md0/photos_processed/processing.log, timestamped record of the entire run.

Takeaways

AMD's Strix Halo continues to earn its keep for ML inference. The HSA_OVERRIDE_GFX_VERSION=11.0.0 workaround remains necessary, but once set, PyTorch and ROCm run without complaints. The 65 GB shared VRAM pool means you can load models without thinking about memory budgets, a workflow advantage that's easy to underestimate until you've experienced it.

Classical computer vision still has its place. The horizon detection pipeline uses Canny edge detection and Hough transforms, techniques from the 1980s. No training data, no GPU needed, deterministic results, and the whole thing runs in 100ms per image. For geometric corrections on photographic images, you don't need a neural network. You need line detection.

EXIF is a minefield. Twenty years of cameras from different manufacturers means every edge case in the spec gets exercised. Tuple lengths vary, denominators are zero, fields are missing or repurposed. If you're parsing EXIF at scale, assume nothing about the data's shape and validate everything.

Resumability is non-negotiable for long-running jobs. Tracking progress in the database and skipping completed work made it trivial to iterate on bugs. Without this, every fix would mean reprocessing 51,000 photos from scratch.

NFS over gigabit is fine for batch processing. Not optimal, but for an overnight job, the network overhead from NAS-attached storage is acceptable. The real bottleneck was ML inference at 0.6 seconds per photo. If I were doing this regularly, 10GbE would be worth the upgrade, but for a one-time archive processing run, gigabit got the job done.

The whole project, from first SSH to final database entry, took about a day of wall time, most of which was unattended processing. The scripting itself was maybe three hours of work. Twenty years of photos, cataloged and corrected overnight. Not bad for a Strix Halo and some Python. The full source is available on GitHub.

What I didn't expect was how useful the database would be after the fact. Being able to ask "show me every photo I took with the D5100 at ISO 3200 or higher" or "find photos with GPS data from 2015" turns a pile of files into something that actually tells a story. The AI captions add another dimension; I can now search my own photo archive by content, not just metadata. It's the kind of capability that makes you wonder why photo management software hasn't done this for years. The models have been available. The hardware has been affordable. Someone just needed to wire it together.

Redesigning a PCB with Claude Code and Open-Source EDA Tools (Part 1)

A.C. Jokela — Fri, 13 Mar 2026 16:00:00 GMT

In January, I spent $468 on Fiverr to have a professional design an Arduino Giga R1 shield with level shifters. It was a good design. Nine TXB0108PW bidirectional level translators, 72 channels of 3.3V-to-5V shifting, a clean two-layer board ready for fabrication. And then I started testing it with the RetroShield Z80, and the auto-sensing level shifters fell apart.

The TXB0108 is a clever chip. It detects signal direction automatically, so you don't need to tell it whether a pin is input or output. For most applications, that's a feature. For a Z80 bus interface, it's a fatal flaw. During bus cycles, the Z80 tri-states its address and data lines. The outputs go high-impedance. They're not driving high or low, they're floating. The TXB0108 can't determine drive direction from a floating signal. It guesses wrong, or it doesn't drive at all, and the Arduino on the other side sees garbage. The board was blind to half of what the Z80 was doing.

The fix was clear: replace the TXB0108s with SN74LVC8T245PW driven level shifters. The SN74LVC8T245 has an explicit DIR pin: you tell it which direction to translate, and it does exactly that, regardless of whether the signals are being actively driven. No guessing, no ambiguity, deterministic behavior during tri-state periods. The trade-off is that you need a direction control signal for each shifter IC, but that's a small price for reliability.

What wasn't clear was how to execute the redesign. I could go back to Fiverr for another $400-500. I could spend weeks learning KiCad properly. Or I could try something that had worked surprisingly well on a previous project: use AI and open-source command-line EDA tools to design the board from a terminal, without ever opening a graphical PCB editor.

This is part one of a two-part series. This piece covers the design and toolchain: how I used Claude Code, the gEDA ecosystem, pcb-rnd, and Freerouting to go from a failed design to production-ready Gerber files. Part two will cover the physical boards, assembly, and testing against the Z80.

The Toolchain Problem

The original Fiverr design was done in KiCad 9.0. My first instinct was to modify it directly: swap the TXB0108 footprints for SN74LVC8T245, update the pin mappings, add the DIR control header, and re-route. But there was a problem. My preferred command-line PCB tool, pcb-rnd, is version 3.1.4 on Ubuntu. KiCad 9.0 uses a file format version (20241229) that pcb-rnd's io_kicad plugin doesn't support. When I tried to open the KiCad PCB:

unexpected layout version number (perhaps too new)

Hard stop. No conversion path exists from KiCad 9.0 to pcb-rnd. The formats aren't just different versions. KiCad's S-expression format and pcb-rnd's text-based format are fundamentally different syntaxes.

I could have started KiCad and used its GUI. But I'd already proven to myself with the dual Z80 RetroShield project that text-based, AI-assisted PCB workflows are not only possible but sometimes preferable. The gEDA/pcb-rnd file format is human-readable. AI can parse it, reason about it, and generate it. A Python script can manipulate it. You can diff two boards and see exactly what changed. None of that is true for a graphical-only workflow.

So the plan became: extract everything useful from the KiCad source files, then rebuild the board from scratch in pcb-rnd's native format using Python. Sound insane? It kind of is. But it worked.

Extracting the DNA

Even though pcb-rnd couldn't read the KiCad files directly, the KiCad files contained all the design intelligence I needed. Component positions, net assignments, pin mappings, board dimensions. It was all there, just in a format I couldn't import.

KiCad's CLI tools (kicad-cli) could export what I needed:

# Component positions (X, Y, rotation for each part)
kicad-cli pcb export pos AlexJ_bz_ArduinoGigaShield.kicad_pcb -o giga_pos.csv

# Netlist connectivity
kicad-cli pcb export ipc2581 AlexJ_bz_ArduinoGigaShield.kicad_pcb -o giga_netlist.d356

The schematic file (AlexJ_bz_ArduinoGigaShield.kicad_sch) was an S-expression text file I could parse to extract the signal mappings: which Giga pin connects to which 5V header pin through which level shifter channel. This was the most critical piece: getting the net assignments wrong would mean the board physically connects but logically doesn't work.

This is where Claude Code earned its keep. I described the KiCad schematic structure and asked it to help me parse out the signal mappings. The KiCad schematic uses hierarchical sheets with positional net connections, which isn't the simplest format to work with manually, but straightforward for an AI that can read S-expressions and track net names across sheets. Within an hour, I had a complete mapping of all 72 signal channels across the 9 shifter ICs.

Generating the Board with Python

With positions and nets extracted, I wrote build_giga_shield.py, a single Python script that generates the entire pcb-rnd board from scratch. No GUI involved. Every component footprint, every pin, every net connection is defined programmatically.

The script is structured around four generator functions:

tssop24_element() generates the SN74LVC8T245PW footprint. TSSOP-24 is a precise geometry: 0.65mm pin pitch, 6.4mm pad-to-pad span, 24 pins. The function calculates pad positions mathematically: 12 pins on the left, 12 on the right, with pin 1 marked as square per convention. Getting the pin numbering right was critical. The SN74LVC8T245's datasheet shows pins 1-12 on the left (DIR, A1-A4, GND, A5-A8, OE#, GND) and pins 13-24 on the right counting bottom-to-top (B8-B5, VCCB, B4-B1, VCCA, VCCA, VCCB).

pin_header_element() handles through-hole pin headers with rotation support. The Arduino Giga R1 has an unusual form factor: the long pin headers run along the board edges horizontally, not vertically. In the original KiCad design, these were placed with 90-degree or -90-degree rotation. Without matching that rotation, a 26-pin header at y=84mm would extend 63.5mm downward to y=148mm, well past the 90mm board edge. The rotation transform was simple once identified:

def rotate(px, py):
    if rot == 90:
        return (py, -px)
    elif rot == -90:
        return (-py, px)
    return (px, py)

smd_0603_element() creates the 0603 footprint shared by all 27 decoupling capacitors and 9 pull-down resistors. Small SMD parts, simple geometry.

mounting_hole_element() places the four 3.2mm mounting holes that align with the Arduino Giga's standoff positions.

The coordinate system was the trickiest part. KiCad uses an arbitrary origin; in this design, x=106mm, y=30.5mm. pcb-rnd uses (0,0). Every KiCad coordinate had to be translated:

KX, KY = 106.0, 30.5
def kpos(kx, ky):
    return (mm(kx - KX), mm(ky - KY))

The build_pcb() function ties everything together: place components, assign nets, build the symbol table, generate the layer stack, and write out a valid pcb-rnd .pcb file. Running the script produces a complete, unrouted board: components placed, netlist defined, silkscreen text positioned, board outline drawn. Ready for routing.

$ python3 build_giga_shield.py
Generated giga_shield.pcb
Board: 155mm x 90mm
9x SN74LVC8T245PW (TSSOP-24) level shifters
DIR control via J11 (1x10 header)

The Format Wars

Getting pcb-rnd to actually accept the generated file was its own adventure. pcb-rnd's parser is strict about things that look optional in the documentation, and its error messages are sometimes misleading. An error in an Element definition might be reported as a syntax error in the Layer section fifty lines later.

Three format issues bit me hardest:

The "smd" flag. I initially generated elements with Element["smd" "TSSOP24" "U1" ...], which seemed logical for surface-mount parts. pcb-rnd rejected it with "Unknown flag: smd ignored," which cascaded into a complete parse failure. The fix: use an empty string Element["" "TSSOP24" "U1" ...]. The SMD-ness is implicit from using Pad[] entries instead of Pin[] entries.

Bare zeros. pcb-rnd is inconsistent about whether 0 and 0nm are interchangeable. In some contexts, bare 0 works fine. In others, it causes a silent parse error that manifests as a syntax error dozens of lines later. The defensive fix: always use 0nm, never bare 0, everywhere.

Missing flags on Layer lines. The Line[] entry inside Layer blocks needs 7 fields, not 6. The seventh is a flags string like "clearline". My generator omitted it, producing Line[x1 y1 x2 y2 thickness clearance]. The parser's error message: syntax error, unexpected ']', expecting INTEGER or STRING, reported at the layer definition, not at the malformed line.

I found these bugs using a binary search approach, truncating the file with head -N and testing each truncation point until I isolated which section introduced the failure. It's crude but effective when error reporting is unhelpful. Claude Code helped enormously here. I'd paste the error and the surrounding file content, and it would spot the structural issue faster than I could.

The pcb-rnd Ecosystem

For anyone unfamiliar with the tools involved, a brief orientation.

gEDA (GNU Electronic Design Automation) is a suite of open-source tools for electronic design. The original project dates to the late 1990s and includes gschem (schematic capture), pcb (PCB layout), and various utilities. The file formats are text-based and human-readable, a deliberate design choice that makes them scriptable and version-control-friendly. The original pcb program is now deprecated.

pcb-rnd is the actively maintained successor to gEDA's pcb program. It reads and writes the same text-based PCB format, but adds modern features: more export formats, better plugin support, and critically for this project, command-line export of Gerber files, PNG renderings, and Specctra DSN files. It runs on Linux (packaged for Ubuntu) but not macOS, which is why I ran it over SSH on a remote machine throughout this project.

Freerouting is a Java-based autorouter that speaks the Specctra DSN/SES interchange format. You feed it a board definition with components and nets but no traces, and it computes the copper routing, finding paths for every net while respecting design rules for trace width, clearance, and via placement. It's the open-source standard for PCB autorouting and has been used in production for decades.

The workflow chains these tools together:

build_giga_shield.py → giga_shield.pcb
                            ↓
                    [pcb-rnd DSN export]
                            ↓
                     giga_shield.dsn
                            ↓
                   [Freerouting autorouter]
                            ↓
                     giga_shield.ses
                            ↓
              [pcb-rnd SES import + Gerber export]
                            ↓
                    Production files

Every step is a command-line operation. Every intermediate file is text. Every transformation is reproducible. Change a component position in the Python script, re-run the pipeline, get new Gerber files. This is the power of text-based EDA: the entire design is version-controlled, diffable, and automatable.

Autorouting: The Machine Does the Tedious Part

With the board generated and validated in pcb-rnd, the next step was routing: connecting all 308 nets with actual copper traces across a two-layer board. This is where Freerouting comes in.

The pipeline starts with exporting the unrouted board to Specctra DSN format. pcb-rnd handles this in batch mode on the remote Linux machine:

pcb-rnd -x dsn giga_shield.pcb

The DSN file contains the board geometry, component placements, pad definitions, and netlist, everything the autorouter needs to compute a routing solution. One subtlety I learned the hard way: the DSN's (structure) section needs explicit (rule) and (via) definitions. pcb-rnd's DSN exporter puts the design rules inside the net class section, but Freerouting also expects them in the structure section. Without them, the router can see the nets but can't figure out what trace widths and via sizes are legal, and it silently fails to route most connections. A two-line addition fixed this:

(via pstk_1)
(rule
  (width 0.254)
  (clearance 0.254)
)

Freerouting itself is a Java application with both GUI and command-line modes. On my machine, I'm running a custom build from source. The current main branch had a few issues I had to fix (a missing static on the main method, a null pointer on maxThreads in the GUI initialization, and a Gradle build compatibility issue). The v1.9 codepath was more reliable for headless routing:

java -jar freerouting-1.9.0-executable.jar \
  -de giga_shield.dsn \
  -do giga_shield.ses

The autorouter loaded the 308-net board, ran through its passes, and produced a Specctra Session file containing 2911 wire segments and 172 vias. Every net connected. Every design rule satisfied. The routing took about 10 seconds for initial placement followed by optimization passes.

Importing the routes back into pcb-rnd was the final step. pcb-rnd can import SES files through its batch mode:

pcb-rnd --gui hid_batch giga_shield.pcb <<EOF
ImportSes(giga_shield.ses)
SaveTo(LayoutAs, giga_shield_routed.pcb)
EOF

The result: a fully routed PCB with 2911 traces and 172 vias, ready for Gerber export.

Running pcb-rnd Over SSH

One of the more unusual aspects of this project is that all pcb-rnd operations happened on a remote Ubuntu 24.04 machine accessed over SSH. pcb-rnd isn't available on macOS via Homebrew (I tried; there's a deprecated pcb package but no pcb-rnd), and building from source on macOS looked like a rabbit hole I didn't want to enter.

The remote workflow was straightforward:

# Upload the PCB
scp giga_shield.pcb alex@10.1.1.27:/tmp/

# Export DSN for routing
ssh alex@10.1.1.27 "pcb-rnd -x dsn /tmp/giga_shield.pcb"

# Import SES and export gerbers
ssh alex@10.1.1.27 'pcb-rnd --gui hid_batch /tmp/giga_shield.pcb <<EOF
ImportSes(/tmp/giga_shield.ses)
SaveTo(LayoutAs, /tmp/giga_shield_routed.pcb)
EOF'

# Export production files
ssh alex@10.1.1.27 "pcb-rnd -x gerber --gerberfile /tmp/giga_shield /tmp/giga_shield_routed.pcb"
ssh alex@10.1.1.27 "pcb-rnd -x png --dpi 600 --photo-mode --outfile /tmp/top.png /tmp/giga_shield_routed.pcb"

# Download results
scp alex@10.1.1.27:/tmp/giga_shield.*.gbr .
scp alex@10.1.1.27:/tmp/top.png giga_shield_top.png

It's more keystrokes than clicking Export in a GUI. But it's scriptable, repeatable, and fits into the same terminal where Claude Code is running. When I needed to iterate (move a component, re-route, re-export) I could do it in a single pipeline without switching contexts.

Claude Code as a Hardware Design Partner

I should be explicit about what Claude Code did and didn't do in this project, because the AI angle is the part people will either find most interesting or most suspicious.

What Claude Code did:

Parsed the KiCad schematic to extract the 72-channel signal mapping across 9 level shifter ICs
Wrote the initial build_giga_shield.py generator script, including all four footprint generators and the net assignment logic
Debugged pcb-rnd format issues by analyzing error messages and file structure
Managed the remote SSH workflow: uploading files, running pcb-rnd commands, downloading results
Fixed bugs in the Freerouting build (the static main issue, the null maxThreads, the Gradle fileMode API change)
Handled iterative changes: "move tinycomputers.io down by a millimeter" became an edit to the Python script, a regeneration, a re-import, and a re-export, all executed as a single flow

What Claude Code didn't do:

Make architectural decisions. The choice to use SN74LVC8T245 over TXB0108, the DIR control header design, the decision to use pull-down resistors defaulting to A-to-B direction. Those were my decisions based on understanding the Z80 bus protocol; it is also on me for selecting the TXB0108 in the first place
Verify electrical correctness. I checked the SN74LVC8T245 datasheet pin mapping myself. I verified that OE# tied to GND means always-enabled. I confirmed the 10K pull-down value was appropriate for the DIR pin
Replace domain knowledge. I knew why the TXB0108 failed during tri-state periods because I understand Z80 bus cycles. Claude Code could have looked up the TXB0108 datasheet, but it couldn't have diagnosed the real-world failure mode from first principles

The pattern that emerged was: I made design decisions, Claude Code implemented them. I said "the DIR pins need pull-down resistors to default A-to-B direction," Claude Code generated the pcb-rnd Element entries with the correct footprint, position, and net assignments. I said "export gerbers at 600 DPI with photo mode," Claude Code ran the right pcb-rnd command on the remote machine.

This is the same division of labor I described in the dual Z80 post: I bring the domain knowledge, the AI handles the format translation. The text-based nature of gEDA files makes this work. If the design lived in a binary format or required mouse interactions, the AI would have been far less useful.

The New Design

Here's what the redesigned board looks like:

Parameter	v0.1 (Fiverr/TXB0108)	v0.2 (Claude Code/SN74LVC8T245)
Level Shifter IC	TXB0108PW (TSSOP-20)	SN74LVC8T245PW (TSSOP-24)
Direction Control	Auto-sensing	Explicit DIR pin
Channels	72	72
Shifter ICs	9	9
Decoupling Caps	27	27
Pull-down Resistors	9 (OE)	9 (DIR)
DIR Control Header	None	J11 (1x10)
Board Dimensions	155mm x 90mm	155mm x 90mm
Layers	2	2
Design Tool	KiCad 9.0 (GUI)	Python + pcb-rnd (CLI)
Design Cost	$468.63	$0 (open source tools)
Design Time	~10 days (outsourced)	~2 days (with AI)

The J11 header is the key addition. It's a 1x10 pin header with 9 direction control pins (one per shifter IC) and a ground reference. Each DIR pin has a 10K pull-down resistor that defaults the direction to A-to-B (3.3V to 5V). To reverse a shifter's direction (for example, when the Arduino needs to read from the Z80's data bus) you drive the corresponding J11 pin high. The Arduino firmware manages this dynamically during bus cycles.

The board carries "tinycomputers.io" and "v0.2" on the silkscreen, placed near the bottom edge. Version tracking on the physical board, a lesson learned from the Fiverr experience, where I had to pay $57 for a revision just to add version text to the silkscreen.

Generating Production Files

With the routed board in hand, the final step was generating files suitable for manufacturing. pcb-rnd handles this with command-line exporters:

# Gerber files (9 layers: top/bottom copper, mask, silk, paste, outline, drill, fab)
pcb-rnd -x gerber --gerberfile giga_shield giga_shield_routed.pcb

# Photo-realistic renderings
pcb-rnd -x png --dpi 600 --photo-mode --outfile top.png giga_shield_routed.pcb
pcb-rnd -x png --dpi 600 --photo-mode --photo-flip-x --outfile bottom.png giga_shield_routed.pcb

The Gerber output includes everything a fab house needs: top and bottom copper, solder mask, silkscreen, paste stencil, board outline, and drill locations. The photo-realistic PNG renderings use pcb-rnd's built-in renderer: green solder mask, gold-plated pads, white silkscreen text. They're useful for documentation and for sanity-checking the layout before sending it to fabrication.

The BOM and centroid files were generated separately from the Python script's component data. The centroid file lists every SMD component's X/Y position and rotation, which is essential if you're having the boards assembled by a service rather than hand-soldering.

What's Different About This Approach

The standard way to design a PCB in 2026 is: open KiCad or Altium, draw a schematic, assign footprints, lay out the board, route traces (manually or with the built-in autorouter), and export Gerbers. It's a visual, interactive process that works well for most people and most projects.

What I did is different in a few ways that I think are worth noting, even if they're not universally applicable:

The entire design is a Python script. build_giga_shield.py is the single source of truth. Want to move a component? Change a coordinate in the script. Want to add a net? Add it to the dictionary. Want to change every decoupling cap from 0.1uF to 0.22uF? Change a string. Then re-run the pipeline. There's no "did I save the layout?" ambiguity, no undo history to worry about, no risk of accidentally moving something with a stray mouse click.

Every intermediate file is text. The .pcb file, the .dsn file, the .ses file. All text, all diffable, all version-controllable. When I moved a component and re-routed, I could git diff the PCB file and see exactly what changed. Try that with a binary PCB format.

AI can participate meaningfully. Because the files are text, Claude Code could read them, modify them, and verify them. It could grep for a component reference in the PCB file, find its coordinates, suggest a new position, and make the edit. It could read the Freerouting log and diagnose why routing failed. This level of AI participation simply isn't possible with graphical-only workflows.

The workflow is reproducible. I can hand someone the Python script and the Freerouting JAR and they can regenerate the entire board from scratch, on any machine with Python and Java. No KiCad version compatibility issues, no plugin dependencies, no "works on my machine" problems.

The trade-off is obvious: this approach requires understanding file formats at a level that graphical tools abstract away. If pcb-rnd's parser rejects your file with a misleading error message, you need to debug the file format, not just re-click a button. It's a power-user workflow. But for someone comfortable with text editors and command lines (which describes most of the audience reading a blog called tinycomputers.io), it's a viable alternative.

What's Next

The Gerber files are ready for fabrication. In part two, I'll cover ordering the boards from PCBWay, sourcing the SN74LVC8T245PW and passive components, and the moment of truth: plugging the RetroShield Z80 into the new shield and seeing if the Arduino can finally see the Z80's bus cycles clearly.

I'll also compare the v0.2 board side-by-side with the original Fiverr v0.1 board: the TXB0108 auto-sensing design versus the SN74LVC8T245 driven design. Same board dimensions, same connector layout, fundamentally different level-shifting approach. The comparison should be instructive for anyone choosing between auto-sensing and driven level translators for bus interfaces.

The Python build script, pcb-rnd source files, Gerber outputs, and all helper scripts are open source:

giga-shield: Complete design files, build pipeline, and production outputs

This is part one of a two-part series. Part two will cover fabrication, assembly, and testing.

Previous posts in this series: Fiverr PCB Design ($468) · Dual Z80 RetroShield · CP/M on the Giga R1 · Zork on the Giga