TinyComputers.io (Posts about fuzzing)

The Bullet Doesn't Go Where You Point It

A.C. Jokela — Sat, 01 Aug 2026 17:00:00 GMT

A bullet fired at a target a thousand yards away does not go where you pointed it. It cannot. In the two-and-a-bit seconds it spends in the air, gravity pulls it something like thirty feet below the line you aimed along, so the barrel has to be tilted up to throw it in an arc. A crosswind you can barely feel on your face moves it a couple of feet sideways. Its own spin — the rifling that keeps it nose-forward is turning it some two hundred thousand times a minute — steers it several inches to the right, all on its own, in still air. And the Earth rotates underneath the whole affair, which sounds like a physicist's joke until you work out that at that distance it is worth a few inches too, in a direction that depends on which way you are facing.

So a rifle is not a pointing device. It is a device you solve. Everything past the muzzle is a computation, and the computation is interesting enough that I have been writing one for a while now, in Rust, and giving it away.

The forward problem

The polite name for this is external ballistics: what happens to a projectile after it leaves the barrel and before it arrives somewhere. Strip away the domain vocabulary and it is an initial-value problem of the sort that turns up everywhere. You know the state of a thing at t=0 — position, velocity, spin — you know the forces acting on it as a function of that state, and you want the trajectory. Write down the derivatives, hand them to an integrator, march forward in small steps. The engine uses fourth-order Runge–Kutta, and there is nothing clever about that choice; it is the boring correct default.

All the interest lives in the derivative function, because the forces are where the physics hides — drag that changes character as the bullet slows back through the sound barrier, gyroscopic spin drift, the Magnus force, aerodynamic jump, pitch damping, Coriolis. I have written that deep-dive already and will not repeat it here; if you want the physics term by term, the professional-grade calculator post walks through each effect and the open-sourcing post covers how the Rust core came out of the platform it grew in. The short version is that the effects live behind individual switches, all off by default, because they belong to a strict hierarchy of mattering: get drag and wind wrong and no amount of Magnus force will save you.

That whole forward solver is free and open. cargo add ballistics-engine puts the library in your project; cargo install ballistics-engine gives you a CLI called ballistics that solves trajectories, runs Monte Carlo dispersion, computes the sight-in that zeroes a rifle, backs a true muzzle velocity out of drops you actually measured at the range, and builds reticle and BDC hold tables for a scope. It compiles to WebAssembly, which is what runs the browser calculators at ballistics.rs. There are Python and Ruby bindings. It will speak JSON over stdin and stdout, and it will speak Model Context Protocol, so a language model can ask it questions without anyone writing glue.

The bugs were never in the physics

Here is the thing I did not expect when I started, and which I suspect generalizes far past ballistics: in something like ninety thousand lines of Rust, almost none of the bugs worth remembering have been physics bugs. The equations are in textbooks. They have been in textbooks for decades, they were checked by people better at this than me, and when I have implemented one incorrectly the error was usually so large that it announced itself immediately.

The bugs are in the conventions.

Consider a wind direction. A number of degrees — obviously. Except: is that the direction the wind is blowing toward, or the direction it is coming from? Meteorology says from. Is zero degrees a headwind, a tailwind, or due north? Is it measured relative to the direction you are shooting, or is it a compass bearing that has to be resolved against your shot azimuth before it means anything? Every one of those is a defensible choice, several of them are in active use by real tools, and each pairing produces a number that looks completely reasonable and is wrong by up to a hundred percent of the wind effect. The engine's answer is wind-from, zero degrees is a headwind — and that sentence is now welded into the help text, the API docs, and a shared constant, because the one time the convention drifted between two subcommands, both of them looked right in isolation.

Units are the same disease. Inches or millimeters for the height of the scope above the bore. Feet per second or meters per second. Grains — a unit of mass equal to one seven-thousandth of a pound, used for bullets and gunpowder and essentially nothing else on Earth. Minutes of angle or milliradians for the correction you dial into your scope, where a minute of angle is approximately an inch at a hundred yards, and the approximation has cost people their afternoons. None of this is intellectually difficult. All of it is a place where a plausible number comes out and nobody catches it, which is precisely the definition of the expensive kind of bug.

Testing physics you cannot check by hand

Which raises the real problem: how do you test this? For a normal library you assert that the function returns the right answer. Here, nobody knows the right answer. The whole point of the engine is to compute a number I cannot compute another way — and if I could hand-verify the output, I would not have needed to write it.

The usual answer is a pile of regression tests, and there is one: roughly fourteen hundred of them. But a regression test only tells you that today's answer matches the answer you wrote down on the day you wrote the test, which may have been wrong. What I have found much more valuable is a set of ten fuzzing harnesses that do not look for crashes so much as for lies — properties that must hold for any input at all, checked against millions of inputs nobody thought to write down.

Some of them have an actual oracle. Turn the drag off — set the ballistic coefficient absurdly high so air resistance becomes negligible — and a projectile is just a parabola, with a closed-form solution any physics undergraduate can produce: the drop at distance x is exactly ½g(x/v)². So one harness generates arbitrary velocities, runs the full production solver, and demands that its answer match the exact analytic parabola. If the integrator, the step size, or the coordinate handling is subtly wrong, no amount of physics-modeling opinion can hide it, because for that one degenerate case the truth is known.

Others check symmetry. Switch off spin drift, Magnus, and Coriolis so that the only thing pushing the bullet sideways is wind, then mirror the wind: the drift must mirror too, exactly. Any asymmetry left over is a bug, because there is nothing left in the model that could physically prefer one side. Others check monotonicity — a claim like "at a fixed distance, a bullet with a higher ballistic coefficient must not drop more." No exact answer required; only a direction of change, which is often the strongest statement you can actually make about a system this tangled.

And my favorite links the previous released version of the engine into the same binary as the current one and runs both. Not to demand identical output — the numbers are supposed to change when the physics improves — but to compare their qualitative responses, so that if a refactor quietly inverts how drop responds to a change in ballistic coefficient, the difference surfaces against the version that shipped rather than against my expectations. Which, being the person who wrote the refactor, are the least trustworthy oracle in the building. This trick has one delightful practical wrinkle: linking two versions of the same Rust crate into one executable works fine right up until both of them export identical #[no_mangle] C symbols for the FFI layer and the linker gives up. The fix is a feature flag that turns the C ABI off on one of the two edges — a small ugly accommodation for a technique that has more than paid for it.

Thirteen platforms, because I am like this

One more testing note disguised as a shipping note. Each release goes out as native binaries for thirteen platforms — Linux on x86-64, Arm64, RISC-V and 64-bit MIPS; macOS on Intel and Apple Silicon; Windows; and FreeBSD, OpenBSD and NetBSD each on both x86-64 and Arm64 — built by a small fleet of single-board computers that boots a real BSD guest for the length of one compile and then puts it away again. There is no business case for most of that; there is not, so far as I know, a single OpenBSD/Arm64 user waiting on a ballistics calculator.

But it earns its place in this particular story, because portability is a bug detector. An engine that produces the same numbers on big-endian 64-bit MIPS as it does on Apple Silicon is an engine carrying fewer accidental assumptions about how its floats are laid out and how its integers are ordered. Thirteen platforms is thirteen independent opportunities to be told I assumed something I had not noticed assuming.

Solving it backwards

Everything so far is the forward problem: here is the load, tell me where it hits. That is the well-posed direction, and honestly, it is a solved problem — there are good calculators, mine included, and they mostly agree.

It is also not the question anybody actually has. Nobody at a loading bench wonders where their known load will land. They wonder the other way around: I want it to land here — what gets me there? Which powder, and how many grains of it, produces the velocity I want? What barrel twist rate will stabilize this bullet? How long should I seat this cartridge? What is this bullet's ballistic coefficient really, derived from its measured dimensions, rather than the optimistic number on the box?

Those are inverse problems, and inverse problems are where it stops being a matter of running the textbook forward. They generally have no closed form. You solve them by searching — running the forward model over and over inside a root-find or an optimizer, hunting for the input that produces the output that was asked for — and the search is only as trustworthy as the model inside its loop, which is the entire ballgame. A backward solver built on a mediocre forward model does not fail loudly. It produces a confident, specific, plausible number that is wrong, which is worse than useless in a domain where people act on the answer.

So that is where the real work went: the drag model those searches lean on is calibrated against Doppler radar measurements of how actual bullets actually decelerate — including through the transonic region, where the classical scale-a-reference-curve assumption is at its weakest — rather than assuming every projectile behaves like a reference shape from a century ago.

The split, stated plainly

I would rather just tell you the arrangement than have you discover it.

The engine is free and open: the library, the CLI, the WebAssembly build, the Python and Ruby bindings, the whole forward solver, the fuzzing harnesses, all of it. Compute trajectories, run dispersions, true your velocity against real measured drops, build hold tables, ship it inside something of your own. No account, no key, no telemetry. That is ballistics.rs.

The four inverse solvers are the one piece behind a subscription, at Ballistics Insight. They run as a service — in a browser, or from the same CLI once you have logged in:

ballistics login
ballistics recommend-powder --cartridge "308 Winchester" \
    --bullet-weight 175 --desired-velocity 2600

They are the paid piece because they are the part with real ongoing cost behind them — the measured drag data the searches depend on, and the compute to run them — and because a project like this is more likely to still exist in five years if some part of it pays for itself. If you shoot, they may be worth it to you. If you are here for the numerical methods and the fuzzing and the pile of BSD binaries, that is all free and I am glad you came.

One honest note if you do use the powder solver: what it returns is a starting point, not a recipe. It is where to begin working a load up, carefully, against published load data, over a chronograph, in the slow and boring and safe way. The physics is good. The physics is not your reloading manual and will not pretend to be.

The bullet still does not go where you point it. But it goes somewhere entirely predictable, and I find that a much more interesting fact than it first appears.

Nothing Was Using the Cores: Three Idle Arm64 Boards, One HA k3s Cluster, and a BSD Build Farm

A.C. Jokela — Sun, 05 Jul 2026 01:30:00 GMT

Update (August 2026): This cluster has since grown from three nodes to eight, and it is no longer all Arm — two x86-64 boxes have joined the quorum's company, along with the NanoPC-T6N I reviewed a few weeks later. One of the two projects those BSD builds exist for — the ballistics engine — now has a post of its own: what it actually computes, how you test physics when there is no oracle to check it against, and why it ships to thirteen platforms.

The plan, at first, was to buy my way there. I wanted a three-node cluster, and the tidy way to get one is three identical machines — so I priced out two more Orange Pi 5 Max boards to stand beside the one I already had: same chip, same memory, same clean symmetry. Then I looked at the total.

We are, as I write this, deep in a memory and storage shortage, and the prices show it. DRAM and NAND have both climbed to where two more maxed-out single-board computers — sixteen gigabytes of LPDDR and a fast terabyte of NVMe apiece — added up to real money, for hardware that would spend most of its life, like everything else on that shelf, doing nothing. Buying idle capacity to set it next to idle capacity is a special kind of foolish, and I could feel that before I could quite justify it.

So before spending anything, I did the cheaper thing first — the thing I should have done to begin with. I took stock of what I already owned.

I had three small computers on a shelf, and on any given afternoon they were doing almost nothing.

That is not a complaint about the hardware. The hardware is lovely. There was an Orange Pi 5 Max — an RK3588 board, eight cores split four-and-four between big Cortex-A76s and little A55s, sixteen gigabytes of RAM, a neural processing unit, and a one-terabyte NVMe drive bolted underneath. Next to it, a Raspberry Pi 5, four cores and eight gigabytes and its own terabyte of WD_BLACK NVMe, quietly serving a web page. And a Compute Module 5 on a carrier, four cores, another sixteen gigabytes, two terabytes of NVMe — the fastest storage of the three and the least to do.

Add it up and it's sixteen cores, forty gigabytes of memory, four terabytes of solid-state storage. And the honest utilization graph, most of the time, was a flat line near the floor. The Orange Pi woke up when I asked it to run a local language model on its NPU. The Pi answered HTTP. The CM5 held its breath. Three machines, each doing one modest thing, each idle in the gaps — which is to say, idle almost always.

I have written before on this blog about hauling systems software onto hardware it was never supposed to run on, and there's a particular feeling that precedes those projects: a gap that's infuriating rather than merely inconvenient, a mismatch between what a machine plainly could do and what it was actually doing. This was that feeling, pointed inward at my own shelf. The boards were not underpowered. They were under-asked. And three of anything, on a network together, with fast local disks and nothing to prove — that is not three appliances. That is a cluster that hasn't been told yet.

So I told it.

Three boxes, one quorum

The goal was a real high-availability Kubernetes cluster, not a toy. For three small Arm boards that means k3s — Rancher's single-binary Kubernetes, which folds the API server, scheduler, controller manager, a container runtime, and networking into one process that starts in seconds and sips memory. On a fleet of SBCs it is the difference between "Kubernetes runs here" and "Kubernetes runs here and also leaves you room to run anything else."

High availability, specifically, is the reason there are three boards in this story and not two. Kubernetes keeps its entire state in etcd, and etcd is a consensus system: it stays available only as long as a strict majority of its members can see each other. With two nodes, a majority is two — lose either and the cluster's brain goes read-only. With three, a majority is two out of three, and you can lose any single node — reboot it, unplug it, drop its switch port — and the cluster keeps making decisions. Three is the smallest number where "highly available" stops being marketing. k3s can run etcd embedded across the server nodes, so all three boards became servers, all three became voting members of the quorum, and all three stayed schedulable for real work. No dedicated control plane sitting idle; the control plane is the workers.

Getting there was mostly smooth, with one wrinkle that everyone who has tried this on Raspberry Pi hardware eventually meets. Kubernetes needs the kernel's memory cgroup controller to enforce per-container memory limits, and the Raspberry Pi OS kernel ships with it switched off by default — a small performance optimization for people who aren't running containers, and a hard wall for people who are. The kubelet simply refuses to start. The fix is one line appended to /boot/firmware/cmdline.txt — cgroup_enable=memory cgroup_memory=1 — and a reboot, which I automated across the Pis with Ansible so the change was recorded rather than remembered. The Orange Pi's Armbian kernel already had the controller enabled, so it needed nothing. That asymmetry — same architecture, same job, different kernel defaults — is the texture of a homelab built from whatever excellent boards happened to be on sale.

I pointed k3s's data directory at the NVMe on each node instead of the default location on the boot media. etcd is unusually sensitive to storage latency — it fsyncs every write it commits — and the difference between a consensus log on an SD card and one on an NVMe drive is the difference between a cluster that feels sluggish and one that feels like a server. It also spares the SD and eMMC cards the write amplification that eventually kills them. Fast local disk is the one thing all three boards had in surplus, and the control plane was the workload most grateful to use it.

A short while later, kubectl get nodes returned three lines, all Ready, all control-plane,etcd, and the flat utilization line finally had somewhere to go. But an empty cluster is just a more elaborate way of being idle. The point was never Kubernetes. The point was the thing the Orange Pi had quietly been doing all along, badly.

NAME              STATUS   ROLES                AGE   VERSION
orangepi5-max     Ready    control-plane,etcd   25h   v1.36.2+k3s1
raspberrypi-cm5   Ready    control-plane,etcd   25h   v1.36.2+k3s1
rpi5              Ready    control-plane,etcd   25h   v1.36.2+k3s1

The build servers that never slept

Here is what had actually been running on that Orange Pi, underneath the occasional language model: three virtual machines, on all the time, existing only to compile software.

I maintain two projects that care a great deal about running natively on the BSDs. It is not because there is a huge BSD following for these projects. In fact, there are probably no dedicated BSD users. I do, however, have a soft spot for the BSDs. For nearly two decades, FreeBSD was my server operating system of choice. When I was into vintage hardware, like Sun Microsystems workstations from the 1990s, or the MicroVAX II I once owned, NetBSD was my preferred operating system.

One of the projects is a small programming language and its compiler, written in C, that I like to ship as real binaries for FreeBSD, OpenBSD, and NetBSD rather than telling people to build it themselves. The other is a ballistics engine written in Rust with the same ambition. Cross-compiling to the BSDs from Linux is possible in theory and miserable in practice — you need each target's system libraries, its linker quirks, its idea of where things live — so the honest way to get a native FreeBSD Arm64 binary is to build it on FreeBSD Arm64. The three VMs were exactly that: a FreeBSD guest, an OpenBSD guest, and a NetBSD guest, each an aarch64 machine running under QEMU with KVM acceleration, each holding a full toolchain, each reachable over SSH so a build script could push source in and pull a binary out.

They worked. And they were awake twenty-four hours a day to be useful for the few minutes a week I actually cut a release. Three virtual machines, two gigabytes of RAM apiece, pinned CPU, humming along through the night so that occasionally a compiler could run. It's the same waste as the idle cluster, just wearing a more respectable coat: capacity committed to a workload that is almost never present.

Once there was a cluster underneath them, the reshape was obvious. A build is a job — it starts, it runs, it produces an artifact, it ends. That is precisely the shape Kubernetes was built around. The always-on VMs wanted to become on-demand Jobs: nothing running when there's nothing to build, and when a build is needed, a pod springs up, does the work on real cores, and disappears.

The catch is the part that makes this a story instead of a paragraph. A Kubernetes Job runs a Linux container — a process sharing the host's Linux kernel. You cannot boot a FreeBSD kernel, or an OpenBSD one, or a NetBSD one, inside an ordinary container; there is no BSD kernel there to boot. The real BSD environment lives in those VM disk images, and the only honest way to build on a BSD is to run the BSD. So the design that survived contact with reality was a VM inside the Job: a privileged pod that mounts the host's /dev/kvm, boots the golden BSD image with QEMU at native speed, waits for the guest's SSH to come up, runs the project's own build script over that SSH connection exactly as before, copies the finished binary out to a persistent volume, and powers the guest down. The VM didn't go away. It stopped being a thing that's always on and became a thing that exists for the length of one build.

The pile of small, specific lies

The plan was clean. The path from plan to working was paved with the kind of bugs that only appear when real hardware stops being polite, and I want to walk through them, because each one is a small lesson and together they are the whole texture of the job.

The first was the most alarming and the most educational. I booted the FreeBSD image inside a pod, and QEMU came up spinning at four hundred percent CPU while the guest's console filled with synchronous exception panics — the Arm equivalent of a program tripping over an instruction the processor won't execute. The image booted fine on the host directly. It crashed reliably inside the pod. The difference was the RK3588's defining feature and, here, its trap: it is a big.LITTLE chip, four fast A76 cores and four efficient A55 cores, and they do not implement exactly the same instructions. QEMU was passing the host CPU straight through to the guest, and the Linux scheduler, seeing an unpinned QEMU thread, was free to migrate the virtual CPU from a big core to a little core mid-execution. The guest would start an instruction the A76 supports, get moved to an A55 that doesn't, and fall over. The host's own VM launch scripts had known this all along — they pinned QEMU with taskset -c 4-7, confining it to the four A76 cores — and my pod hadn't. One taskset and the synchronous exceptions vanished. The cluster's cleverest hardware feature was, for this workload, a thing to carefully switch off.

The second bug was a quiet philosophical one. My plan referred throughout to the BSD disk images as "golden images" — static, pristine templates you overlay a scratch copy on top of and never mutate. When I went to make that scratch overlay, QEMU refused: the image was locked, held open read-write by another process. The other process was the always-on VM. The "golden image" was the live disk of the running build server — not a template at all, but the working drive of a machine that was, at that moment, mid-thought. Overlaying a copy-on-write layer on a disk that's being written underneath you is not a snapshot; it's a data-corruption generator. And when I did force a copy, it failed its filesystem check on boot, because the source had been running for thirteen days without a clean shutdown and its dirty bit had never been cleared. The fix was to do the boring, correct thing I'd been skipping: cleanly shut each VM down, take a genuine point-in-time snapshot of the now-consistent disk into a real golden image, and restart the VM. The builds overlay those — static, clean, never open by anything else. It cost a couple of minutes of downtime per guest and bought a foundation that isn't quietly rotting.

Then the small, specific lies of virtual hardware. The first pod boot that got past the CPU pinning died instantly with QEMU complaining it couldn't find efi-virtio.rom — a network-card boot ROM that isn't packaged in a slim container image. The host scripts, again, already knew: they set romfile= empty on the virtual NIC, disabling the ROM the build doesn't need. Add that, and the guest booted further — and then OpenBSD, specifically, hung partway through, wedged on device probing. FreeBSD had sailed through the same configuration. The difference was vectors=0, another flag the host scripts carried and mine didn't, which disables MSI-X interrupts on the virtual devices; OpenBSD's virtio drivers, under emulation, needed the older interrupt style to come up at all. FreeBSD tolerated the modern path. OpenBSD did not. The host's launch scripts, it turned out, were a compressed archive of exactly these lessons, learned once and encoded as flags I had to re-learn by leaving them out.

The BSDs then taught me about their own disks. My build script staged source into the guest's home directory and immediately hit "No space left on device." OpenBSD, by long tradition, slices its disk into many small partitions, and its root filesystem — where a naive home directory lives — was four hundred megabytes and already full. NetBSD does the same, with a root barely over a gigabyte. FreeBSD hands you one big root and doesn't care. So the builder learned to build where the space actually is: on OpenBSD, in /usr/obj, the partition the system reserves for build output; on NetBSD, in /usr/pkg, its roomy second disk. Rust made this worse before it made it better, because Cargo unpacks its entire crate registry into a cache under the home directory, and on NetBSD that cache alone overflowed the tiny root the instant a real dependency graph arrived. Pointing CARGO_HOME at the big partition too, per guest, finally let a Rust project build on a NetBSD machine whose root filesystem couldn't have held its own dependencies.

The last of them wasn't a build bug at all; it was a networking cleanup that turned into an accidental repair. NetBSD's VM, unlike its siblings, had been bridged directly onto my LAN, while FreeBSD and OpenBSD used QEMU's simpler user-mode networking. I wanted them consistent, so I converted NetBSD to user-mode too — and in restarting it discovered that the bridged setup would have failed on its very next boot regardless, because a QEMU package update had silently stripped the CAP_NET_ADMIN capability off the bridge helper, the small privileged binary that lets an unprivileged VM attach to a bridge. The VM had been running for so long it hadn't been restarted since the update landed. The cleanup I wanted to do had been sitting on top of a latent failure I didn't know about, and converting away from bridging both simplified the design and defused a bomb.

None of these were deep. Every one was a mundane mistake or a mundane environmental fact, wearing for an afternoon the costume of something profound. That is the entire pattern of this kind of work, and I've made peace with it: the exotic theory that flatters your understanding almost always sits on top of a boring cause that flatters nothing. taskset. romfile=. A partition that's full. A capability an update took away.

When it was done, the payoff was concrete. A single command — make bsd-build PROJECT=lattice OS=freebsd — now fetches the latest source straight from GitHub, boots the golden FreeBSD image in a pod on the Orange Pi, compiles the project inside it, and drops a genuine aarch64 FreeBSD executable back onto my workstation, stripped and ready. The same command builds for OpenBSD and NetBSD, for both projects, and a release variant does all of it and publishes. Six native BSD binaries, produced by boards that used to keep three virtual machines awake all night to do the same thing on demand for no one.

Naming the tenants

With the builds turned into jobs, the always-on VMs had no reason to exist. I stopped and disabled them — reversibly; the golden images remain, and a systemctl away is the old world back — and the Orange Pi handed back roughly six gigabytes of RAM and its four A76 cores. Which returns us, neatly, to the theme this whole exercise keeps circling: reclaimed idle capacity is only worth reclaiming if you then ask something of it.

So here is the honest census of what actually runs on the cluster now — the jobs, named plainly.

On-demand BSD binary builds are the anchor tenant, the workload the whole thing was justified by: FreeBSD, OpenBSD, and NetBSD Arm64 binaries for the language compiler and the ballistics engine, each build a pod that boots a real BSD, compiles, and exits. Zero footprint between releases; real cores during them.

A full observability stack is the second tenant, and it exists partly because a cluster you can't see is a cluster you can't trust. It's the standard Prometheus-and-Grafana pairing, deployed the k3s-native way, with node exporters on every board reporting not just CPU and memory and disk but the thing that actually matters on fanless Arm boards crammed with NVMe: temperature. SBCs throttle when they cook, and a build farm that silently halves its own clock speed under thermal load is a build farm that lies to you about why it's slow. Grafana lives at a name on my LAN — grafana.localnet, resolved by an Unbound server on a little FreeBSD box that also runs my DNS — behind an ingress controller carefully arranged not to collide with the web page one of the Pis was already serving on port eighty. The stack pins its stateful pieces to the CM5, the node with the two-terabyte drive and the least else to do. The board that was holding its breath now holds the metrics.

Continuous fuzzing of the compiler is the newest tenant, and the one that most makes me glad the cores were sitting there. The language project ships nine libFuzzer harnesses — for the compiler front-end, three different bytecode VMs, the lexer, the formatter, and the JSON, TOML, and YAML parsers — and a fuzzer is the most gluttonous, most embarrassingly-parallel workload I own. It will eat every idle cycle you're willing to feed it and ask for more, forever, and every cycle it eats is a cycle spent trying to crash your code before a user does. On a cluster whose defining problem was too many idle cores, that is a nearly perfect fit. Each target gets its own continuous pod, spread across the three boards, building itself from the latest source and grinding against a corpus that persists and compounds on the NVMe — the first of the nine are coming online as I write this. And the results go somewhere I'll actually see them: when a fuzzer finds a crash, a small watcher reproduces it, fingerprints it so the same bug doesn't file itself twice, and opens a Jira ticket — the real bug tracker the project already uses — with the sanitizer backtrace and a base64 reproducer attached, tagged to the right component. Live throughput and coverage stream into the same Grafana the rest of the cluster reports to; a nightly job renders coverage as HTML behind that same ingress. Idle cores in; deduplicated, actionable bug reports out.

And underneath all of it, still, the Orange Pi's NPU runs a local language model when I ask it to — the original tenant, never evicted, just no longer the only one. Kubernetes shares the boards with it rather than owning them outright, which is exactly the relationship a homelab wants: the cluster is a tenant of the hardware, not its landlord.

The capacity was always there

There's a version of this post that's a shopping list — buy these boards, run these commands, get a cluster. I don't think that's the true version. The commands were the easy part; they always are. The hard part was the pile of small specific lies the hardware tells when you finally push it, and the honest bookkeeping of what's really running versus what's merely installed.

But the part I keep turning over isn't technical at all. It's that none of this required new hardware. Every core, every gigabyte, every terabyte of fast disk in this cluster was already sitting on my shelf, already powered, already mostly doing nothing. The Orange Pi was awake all night to compile software a few minutes a week. The CM5 was holding its breath. The utilization graph was a flat line near the floor, and I'd looked at it a hundred times without seeing it as a to-do item.

What changed wasn't the machines. It was the question I asked them. Three boards that each did one small thing became a single system that builds native binaries for three operating systems, watches its own temperature, and fuzzes a compiler around the clock — and the only thing I added was the willingness to treat idle capacity as a debt worth collecting.

The cluster was already there. It just hadn't been told yet.