TinyComputers.io (Posts about ssh)

A Mac, Two Disk Images, and an SSH Prompt: OpenBSD 7.9 on the Raspberry Pi 4B

A.C. Jokela — Mon, 07 Sep 2026 22:06:31 GMT

A Mac, Two Disk Images, and an SSH Prompt: OpenBSD 7.9 on the Raspberry Pi 4B

I wanted an OpenBSD machine I could plug into the network and immediately start using. Put the card in, connect Ethernet, apply power, find its address, SSH in. An ordinary account with my key, doas when I needed root, and enough room on the card that installing software would not immediately become another storage project. A modest list, which is how these things usually begin.

The machine now exists. A Raspberry Pi 4 Model B Rev 1.5, four Cortex-A72 cores and 4 GB of RAM, running OpenBSD 7.9 from microSD. It answers to rpi4b, negotiated gigabit Ethernet, and has separate /usr and /home filesystems with about 28 GiB each. The installation happened inside a virtual machine on my Apple Silicon Mac. By the time the card reached the Pi, the operating system was installed, patched, configured, and waiting for hardware to inhabit.

The Pi over SSH, running OpenBSD 7.9. Four Cortex-A72 cores, 12 installed packages, and Puffy taking up most of the terminal.

Make your own card: Download the generic OpenBSD 7.9 image (525 MiB), with setup instructions, SHA-256 checksums, and firmware licenses and corresponding source. This unofficial build accepts your SSH public key before boot. Its VM tests passed; a physical Pi boot test of the generic version is still pending.

The handoff from VM to hardware is the interesting part. A VM can tell you quite a lot about an operating system. It has considerably less to say about whether a Raspberry Pi will find the bootloader you put on a card.

OpenBSD already supports the Pi 4 through its arm64 port. The official installation media includes the Raspberry Pi firmware, so the starting point was conveniently unexotic: install79.img from the OpenBSD 7.9 arm64 release, its signed checksum manifest, and the release public key. The arm64 platform page documents the support and the firmware requirements.

I wanted the installation completed before deployment. An installer image gets you to the place where you can install an operating system; the image I wanted would get me to an SSH prompt. That meant running the installer somewhere, making the configuration decisions there, and preserving its finished disk as the thing I would eventually write to microSD.

The Mac made a good workshop. QEMU ran a generic ARM64 virt machine using Apple's Hypervisor Framework acceleration, with four virtual CPUs, 2 GB of RAM, EDK2 UEFI firmware, and virtio disk and network devices. SSH was forwarded to a port on the Mac's loopback interface. I could inspect the console and connect to the guest without giving every intermediate version of the build a presence on the LAN.

Before installation, we verified the release material. The helper checked the Ed25519 signature on the manifest and the SHA-256 digest of the installer. It also compared the release public keys obtained from the official download location and OpenBSD's source repository. The original downloads and verification results stayed in the project directory with the scripts.

There would eventually be several images with very similar names. Keeping a record of which bytes had actually been checked seemed like a better plan than trusting my future ability to remember what “final” meant.

The installer set the hostname to rpi4b, created my alex account, enabled SSH, and selected America/Chicago. We installed the release sets and retained the answer file and customization scripts. OpenBSD then booted as a normal installed guest. At that point, the account, services, and filesystems were all available for testing, and the Pi-specific boot work was still ahead.

QEMU was providing the guest with UEFI firmware. The Pi would need to reach OpenBSD through its own boot chain, using files on the card. Installing under the generic virtual machine did not automatically arrange all of those files for the physical board.

The customization script mounted the target's FAT boot partition and populated it with the Pi firmware, device trees, overlays, and U-Boot supplied by the official installer. It checked for the Pi 4's bcm2711-rpi-4-b.dtb and the EFI loader at efi/boot/bootaa64.efi. The boot configuration fit in four lines:

arm_64bit=1
enable_uart=1
dtoverlay=disable-bt
kernel=u-boot.bin

We compared the resulting configuration, start4.elf, fixup4.dat, Pi 4 device tree, U-Boot, Bluetooth-disabling overlay, and EFI loader against the signed installer. All matched. There was a known source for the firmware on this card, and a check that the files had survived the assembly process intact.

The guest could demonstrate that OpenBSD started and accepted a login. The firmware comparison established that the Pi's boot files matched their source. The eventual hardware boot would connect those observations.

The network configuration had a similar wrinkle. QEMU's network card appeared as vio0. The Pi's onboard Ethernet appeared as bse0, through OpenBSD's Broadcom GENET driver. A perfectly functional DHCP configuration for a virtual network card would be of limited assistance to a physical interface with a different name.

So the image kept its vio0 configuration for future VM testing and gained an /etc/hostname.bse0 containing:

inet autoconf
inet6 autoconf

IPv4 DHCP and IPv6 autoconfiguration. That was the entire wired-network setup. No static address embedded in an image that might eventually land on another network, and no Wi-Fi credentials to get right before the first login. I would find rpi4b in the router's lease table. Automatic .local discovery was not configured, so the lease table was part of the intended procedure.

For SSH, I used the same arrangement I wanted to live with afterward. My public key went into /home/alex/.ssh/authorized_keys; the private key stayed on the Mac. The server started automatically, with public-key authentication enabled and password authentication, keyboard-interactive authentication, and direct root login disabled.

There is consequently no default password for this image. Someone repeating the build needs to supply their own public key. These are the access choices for my installation, rather than a claim about what every OpenBSD installation does by default.

Root access was one line in /etc/doas.conf:

permit nopass alex as root

I like how little explanation that line needs. alex may become root, and another password prompt is not involved. The doas manual describes how to restrict rules to particular commands or arguments, but this account is the machine's administrator. Its rule deliberately permits all commands as root. Anyone holding the corresponding login key has that authority.

The test was equally compact:

doas -n id -u

It printed 0. The -n makes it noninteractive, so a successful result establishes that the permission actually works without a prompt. We also checked the running SSH daemon and its effective configuration. A line in a file is an intention; being able to log in and run the command is the behavior I was trying to buy with it.

Then there was the card, and the apparently small request to use the rest of it.

The portable image was 14 GiB: 15,032,385,536 bytes. Its layout included root, swap, separate /usr and /home filesystems, and a firmware partition. It also retained about 7.1 GiB of unallocated space inside the image. Writing it onto a larger card would reproduce that layout faithfully, including its unused space.

There was no automatic first-boot expansion in this build. The image had exactly the filesystem sizes we gave it. A larger card would not persuade OpenBSD to reconsider them on its own.

I wanted the extra capacity divided between /usr and /home, where installed software and my own work would have room to accumulate. Root, swap, and the boot area could stay as they were. So we made a second image sized to the actual card: 63,864,569,856 bytes.

That number matters more than the “64 GB” on the packaging. Cards sold under the same capacity label can differ in their exact size. An image built to the last usable part of one card needs another card at least that large. The portable version is the smaller starting point; the expanded version is the one prepared for this deployment.

Here is what df -h reported for the filesystems, with swap and the boot partition included for context:

Filesystem or area	Portable image	Expanded image
`/`	1.3 GiB	1.3 GiB
`/usr`	2.9 GiB	28.3 GiB
`/home`	1.9 GiB	27.4 GiB
Swap	256 MiB	256 MiB
Firmware partition	260 MiB	260 MiB

The added space was divided equally. The final totals differ because /usr started larger. Filesystem sizes are also slightly smaller than their containing partitions, which explains why multiplying a displayed number back into sectors will not reproduce the disklabel.

The complication was physical ordering. /home sat after /usr. Giving /usr more contiguous space required relocating /home first. The friendly phrase “expand both partitions” had turned into a sequence where the order of operations mattered quite a bit.

We did this offline, in an OpenBSD installer VM, with the original image attached read-only alongside the larger target image and installer. The scripts checked the disk capacity, partition offsets, filesystem types, and disk identifier against the expected layout before proceeding. They were intentionally specific to these images. I would rather have a resize script decline a disk it does not recognize than improvise with its contents.

First came extending the OpenBSD area in the outer partition table and enlarging the disklabel boundary. Then /home was recreated at its final location, populated from the read-only source, and checked for matching contents and ownership. Only after that copy was verified could /usr grow into the space that had become available.

For /usr, the script checked the existing filesystem, enlarged its partition, ran a growfs dry run, performed the growth, and checked the filesystem again. OpenBSD's growfs documentation calls for an unmounted filesystem and a filesystem check afterward. Booting the installer gave us an environment where the target's /usr was simply another unmounted disk partition.

The expanded filesystems passed their checks. The disk identifier, root, swap, and firmware area were preserved. A normal boot of the expanded image then confirmed that /usr and /home mounted where they belonged, at their new sizes.

Root still has about 1.3 GiB, and /var lives there in this layout. That is a choice with consequences if the machine later becomes a collector of logs or service data. For the current installation it leaves ample room, but I have not allocated 28 GiB to every directory just by putting a larger card underneath them.

Before packaging, we applied the available updates. The September 7, 2026 build had patches 001 through 014 installed, with no further patches reported by the update check. That statement has a date attached for a reason; anyone deploying later should check the OpenBSD 7.9 errata and run the update tools again.

We also waited for kernel relinking to finish. Then the image needed to forget the parts of its identity that belonged to the VM: its generated SSH host keys and temporary DHCP lease. The resolver configuration was reset, and the Pi would generate fresh host keys at its own boot. A first-boot fw_update would detect firmware needs in the physical environment.

My authorized public key stayed. That key says who may administer the machine. The host keys say which machine has answered the connection. Cloning the first is the point of this personalized image; generating the second during deployment gives the running system its own SSH identity.

Both variants booted independently in QEMU. SSH login worked, sshd was running, doas returned root, and the effective SSH settings and filesystem mounts matched the intended configuration. Each guest shut down cleanly before packaging. We compressed the raw images with gzip and verified that decompression reproduced the originals, along with checks of the partition table and boot partition.

The compressed files were about 644 MiB and 738 MiB. Their expanded disk sizes were considerably larger, but empty space is cooperative material for a compressor. A SHA-256 manifest recorded the packaged artifacts.

Writing the expanded image from macOS used the usual whole-device procedure:

diskutil list external physical
diskutil info /dev/diskN
diskutil unmountDisk /dev/diskN
sudo dd if=images/openbsd-7.9-rpi4b-64gb.img of=/dev/rdiskN bs=4m
sync
diskutil eject /dev/diskN

N must be the verified card identifier, and the target must be large enough. That command replaces the whole device's contents. It takes the raw .img; a gzip file needs decompression first. On macOS, Control-T asks dd for a progress report, which is useful when the terminal has spent several minutes giving every appearance of having lost interest.

The entire 63,864,569,856-byte write completed. I stopped the subsequent full-card readback before it finished, and the card was safely ejected. The records therefore say exactly that: complete image-package verification, complete card write, incomplete card readback verification. The physical boot that followed supplied different evidence; it did not retroactively checksum the unread part of the card.

With the card in the Pi and Ethernet connected, the machine booted. The router assigned 10.1.1.32 on my 10.1.1.0/24 network, under the hostname rpi4b. I connected with:

ssh -i ~/.ssh/id_ed25519 alex@10.1.1.32

OpenBSD 7.9, arm64. The boot log identified the Pi 4 Model B Rev 1.5 and all four Cortex-A72 cores. rcctl check sshd returned a running service. doas -n id -u printed 0. /usr mounted at 28.3 GiB and /home at 27.4 GiB.

bse0 reported 1000baseT full-duplex, with the link active. That is a negotiated link speed, not a throughput benchmark. The session established that wired networking and SSH worked on the real board. The expanded image received this physical test; the portable variant had its independent VM validation but was not separately booted on the Pi.

That was the handoff completed: the boot files found the operating system, the operating system found its hardware, and the configuration I had tested on the Mac became the configuration of a machine on the network.

A download for the next card

An image containing my login key had a rather limited audience. For the downloadable version, we created fresh filesystems, copied the system into them, and removed my account, keys, history, and inherited machine identities. Fresh filesystems also keep deleted personal files out of the free space. Each deployment generates its own SSH host keys.

The generic archive is about 525 MiB, expanding to 63,864,569,856 bytes. Check your card's actual capacity; a “64 GB” label is not enough. This version keeps separate /usr and /home filesystems, reporting about 28.4 GiB and 27.4 GiB after fresh formatting. There is no automatic expansion on larger cards.

Download the image and SHA256 from the links above into one directory. On macOS, verify the download before decompressing it:

shasum -a 256 --ignore-missing -c SHA256
gzip -dc openbsd-7.9-rpi4b-generic.img.gz > openbsd-7.9-rpi4b-generic.img

Continue only if verification reports the image as OK. Linux uses sha256sum --ignore-missing -c SHA256. The manifest also covers the companion files; missing companions are skipped. Allow roughly 64 GB of free space for decompression.

Flash the resulting .img using the whole-card procedure above, substituting the generic filename. Reinsert the card afterward and find its small FAT boot volume. Copy your public key into that volume's root directory, replacing BOOT_VOLUME with its actual mounted name:

cp ~/.ssh/id_ed25519.pub /Volumes/BOOT_VOLUME/authorized_keys

The private key stays on your computer. Optionally add a plain-text setup.conf beside authorized_keys:

username=openbsd
hostname=openbsd-rpi4b

Those are the defaults; the timezone is UTC. Safely eject the card, boot the Pi with Ethernet connected, and find its address in the DHCP leases. Then:

ssh -i ~/.ssh/id_ed25519 openbsd@PI_ADDRESS
doas -s

There is no default password. SSH stays disabled until a valid public key is supplied, then permits key login for the owner. Direct root SSH login is disabled; doas grants root without a password. Missing or malformed setup input can be corrected on the FAT partition and retried at the next boot.

Disposable VM copies passed owner setup, SSH, doas, reboot persistence, and installation of a signed package through pkg_add. Unrelated keys and root logins were rejected. The validation record and first-boot helper source are included. The generic image retains the boot firmware from my working card, but it still needs its own physical Pi test. The successful hardware session above belongs to the personalized build.

For now, rpi4b has an address and an SSH prompt. I can install software, copy a project over, or become root without reaching for a monitor. The card arrived with the installation work already behind it, which was the whole ambition when this was still a modest list.

Building Cross-Platform Rust Binaries: A Multi-Architecture Build Orchestration System

A.C. Jokela — Fri, 21 Nov 2025 21:19:35 GMT

When developing ballistics-engine, a high-performance ballistics calculation library written in Rust, I faced a challenge: how do I efficiently build and distribute binaries for multiple operating systems and architectures? The answer led to the creation of an automated build orchestration system that leverages diverse hardware (from single-board computers to powerful x86_64 servers) to build native binaries for macOS, Linux, FreeBSD, NetBSD, and OpenBSD across both ARM64 and x86_64 architectures. Now, you are probably wondering why I am bothering to show love for the BSD Trilogy; the answer is simple: because I want to. Sure they are a bit esoteric, but I ran FreeBSD for years as my mail server. I still like the BSDs.

This article explores the architecture, implementation, and lessons learned from building a production-grade multi-platform build system that powers https://ballistics.zip, where users can download pre-built binaries for their platform with a simple curl command.

curl --proto '=https' --tlsv1.2 -sSf https://ballistics.zip/install.sh | sh

The Problem: Cross-Platform Distribution

Rust's cross-compilation capabilities are impressive, but they have limitations:

Cross-compilation complexity: While Rust supports cross-compilation, getting it working reliably for BSD systems (especially with system dependencies) is challenging
Native testing: You need to test on actual hardware to ensure binaries work correctly
Binary compatibility: Different BSD versions and configurations require native builds
Performance verification: Emulated builds may behave differently than native ones

The solution? Build natively on each target platform using actual hardware or high-performance emulation.

Architecture Overview

The build orchestration system consists of three main components:

1. Build Nodes (Physical and Virtual Machines)

macOS systems (x86_64 and aarch64) - Local builds
Linux x86_64 server - Remote build via SSH
FreeBSD ARM64 - Single-board computer (Raspberry Pi 4)
OpenBSD ARM64 - QEMU VM emulated on x86_64 (rig.localnet)
NetBSD x86_64 and ARM64 - QEMU VMs

2. Orchestrator (Python-based coordinator)

Reads build node configuration from build-nodes.yaml
Executes builds in parallel across all nodes
Collects artifacts via SSH/SCP
Generates SHA256 checksums
Uploads to Google Cloud Storage
Updates version metadata

3. Distribution (ballistics.zip website)

Serves install script at https://ballistics.zip
Hosts binaries in GCS bucket (gs://ballistics-releases/)
Provides version detection and automatic downloads
Supports version fallback for platforms with delayed releases

Hardware Infrastructure

Single-Board Computers

Orange Pi 5 Max (ARM64)

Role: Host for NetBSD ARM64 VM
CPU: Rockchip RK3588 (8-core ARM Cortex-A76/A55)
RAM: 16GB
Why: Native ARM64 hardware for running QEMU VMs
Host IP: 10.1.1.10
VM IPs:
NetBSD ARM64: 10.1.1.15
OpenBSD ARM64 (native, disabled): 10.1.1.11

Raspberry Pi 4 (ARM64)

Role: FreeBSD ARM64 native builds
CPU: Broadcom BCM2711 (quad-core Cortex-A72)
RAM: 8GB
Why: Stable FreeBSD support, reliable ARM64 platform
IP: 10.1.1.7

x86_64 ("rig.localnet")

Role: Linux builds, BSD VM host, emulated ARM64 builds
CPU: Intel i9
RAM: 96GB
IP: 10.1.1.27 (Linux host), 10.1.1.17 (KVM host)
VMs Hosted:
FreeBSD x86_64: 10.1.1.21
OpenBSD x86_64: 10.1.1.20
OpenBSD ARM64 (emulated): 10.1.1.23
NetBSD x86_64: 10.1.1.19

Local macOS Development Machine

Role: macOS binary builds (both architectures)
Build Method: Local cargo builds with target flags
Architectures:
aarch64-apple-darwin (Apple Silicon)
x86_64-apple-darwin (Intel Macs)

A Surprising Discovery: Emulated ARM64 Performance

One of the most interesting findings during development was discovering that emulated ARM64 builds on powerful x86_64 hardware are significantly faster than emulated ARM64 on native ARM64 builds on single-board computers.

Performance Comparison

Emulated ARM64 on ARM64: ~99+ minutes per build
Emulated ARM64 on x86_64: 15m 37s ⚡

The emulated build on rig.localnet (running QEMU with KVM acceleration) completed in about 6x less time than the native ARM64 hardware. This is because:

The x86_64 server has significantly more powerful CPU cores
QEMU with KVM provides near-native performance for many workloads
Rust compilation is primarily CPU-bound and benefits from faster single-core performance
The x86_64 server has faster storage (NVMe vs eMMC/SD card)

As a result, the native OpenBSD ARM64 node on the Orange Pi is now disabled in favor of the emulated version.

Prerequisites

SSH Key-Based Authentication

Critical: The orchestration system requires passwordless SSH access to all remote build nodes. Here's how to set it up:

Generate SSH key (if you don't have one):

ssh-keygen -t ed25519 -C "build-orchestrator"

Copy public key to each build node:

# For each build node
ssh-copy-id user@build-node-ip

# Examples:
ssh-copy-id alex@10.1.1.27      # Linux x86_64
ssh-copy-id freebsd@10.1.1.7     # FreeBSD ARM64
ssh-copy-id root@10.1.1.20       # OpenBSD x86_64
ssh-copy-id root@10.1.1.23       # OpenBSD ARM64 emulated
ssh-copy-id root@10.1.1.19       # NetBSD x86_64
ssh-copy-id root@10.1.1.15       # NetBSD ARM64

Test SSH access:

ssh user@build-node-ip "uname -a"

Software Requirements

On Build Orchestrator Machine:

Python 3.8+
pyyaml (pip install pyyaml)
Google Cloud SDK (gcloud command) for GCS uploads
SSH client

On Each Build Node:

Rust toolchain (cargo, rustc)
Build essentials (compiler, linker)
curl, wget, or ftp (for downloading source)
Sufficient disk space (~2GB for build artifacts)

BSD-Specific Requirements

NetBSD: Install curl via pkgsrc (native ftp doesn't support HTTPS)

# Bootstrap pkgsrc
cd /usr && ftp -o pkgsrc.tar.gz http://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz
tar -xzf pkgsrc.tar.gz
cd /usr/pkgsrc/bootstrap && ./bootstrap --prefix=/usr/pkg

# Install curl
/usr/pkg/bin/pkgin -y update
/usr/pkg/bin/pkgin -y install curl

OpenBSD: Native ftp supports HTTPS

pkg_add rust git

FreeBSD: Use pkg for everything

pkg install -y rust git curl

The ballistics.zip Website and Install Script

How It Works

https://ballistics.zip serves as the primary distribution point for pre-built ballistics-engine binaries. The system uses:

GCS Bucket:
gs://ballistics-releases/ - Binary artifacts
CDN: Google Cloud CDN provides global distribution
Install Script: Universal installer that:
Detects OS and architecture
Downloads appropriate binary
Verifies SHA256 checksum
Installs to /usr/local/bin

Usage

Basic installation:

curl -sSL https://ballistics.zip/install.sh | bash

Specific version:

curl -sSL https://ballistics.zip/install.sh | bash -s -- --version 0.13.3

Different install location:

curl -sSL https://ballistics.zip/install.sh | bash -s -- --prefix ~/.local

Install Script Architecture

The install.sh script intelligently handles:

Platform Detection:

OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)

case "$ARCH" in
  x86_64|amd64) ARCH="x86_64" ;;
  aarch64|arm64) ARCH="aarch64" ;;
  *) echo "Unsupported architecture: $ARCH"; exit 1 ;;
esac

PLATFORM="${OS}-${ARCH}"  # e.g., "openbsd-aarch64"

Version Fallback: If a requested version isn't available for a platform, the script automatically finds the latest available version:

# If openbsd-aarch64 0.13.3 doesn't exist, fall back to 0.13.2
AVAILABLE_VERSION=$(curl -sL $BASE_URL/versions.txt | grep "^$PLATFORM:" | cut -d: -f2)

Checksum Verification:

EXPECTED_SHA=$(cat "$BINARY.sha256")
ACTUAL_SHA=$(sha256sum "$BINARY" | awk '{print $1}')

if [ "$EXPECTED_SHA" != "$ACTUAL_SHA" ]; then
  echo "Checksum verification failed!"
  exit 1
fi

Build Orchestration System Deep Dive

Configuration: build-nodes.yaml

The heart of the system is build-nodes.yaml, which defines all build targets:

nodes:
  # macOS builds (local machine)
  - name: macos-aarch64
    host: local
    target: aarch64-apple-darwin
    build_command: |
      cd /tmp && rm -rf ballistics-engine-{version}
      curl -L -o v{version}.tar.gz https://github.com/ajokela/ballistics-engine/archive/refs/tags/v{version}.tar.gz
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      cargo build --release --target {target}
    binary_path: /tmp/ballistics-engine-{version}/target/{target}/release/ballistics
    enabled: true

  # Linux x86_64 (remote via SSH)
  - name: linux-x86_64
    host: alex@10.1.1.27
    target: x86_64-unknown-linux-gnu
    build_command: |
      cd /tmp && rm -rf ballistics-engine-{version}
      wget -q https://github.com/ajokela/ballistics-engine/archive/refs/tags/v{version}.tar.gz
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      ~/.cargo/bin/cargo build --release --target {target}
    binary_path: /tmp/ballistics-engine-{version}/target/{target}/release/ballistics
    enabled: true

  # OpenBSD ARM64 emulated (FASTEST ARM64 BUILD!)
  - name: openbsd-aarch64-emulated
    host: root@10.1.1.23
    target: aarch64-unknown-openbsd
    build_command: |
      cd /tmp && rm -rf ballistics-engine-{version}
      ftp -o v{version}.tar.gz https://github.com/ajokela/ballistics-engine/archive/refs/tags/v{version}.tar.gz
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      cargo build --release
    binary_path: /tmp/ballistics-engine-{version}/target/release/ballistics
    enabled: true

  # NetBSD x86_64 (HTTPS support via pkgsrc curl)
  - name: netbsd-x86_64
    host: root@10.1.1.19
    target: x86_64-unknown-netbsd
    build_command: |
      cd /tmp && rm -rf ballistics-engine-{version}
      /usr/pkg/bin/curl -L -o v{version}.tar.gz https://github.com/ajokela/ballistics-engine/archive/refs/tags/v{version}.tar.gz
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      /usr/pkg/bin/cargo build --release
    binary_path: /tmp/ballistics-engine-{version}/target/release/ballistics
    enabled: true

Orchestrator Workflow

The orchestrator.py script coordinates the entire build process:

Step 1: Parallel Build Execution

def build_on_node(node, version):
    if node['host'] == 'local':
        # Local build
        subprocess.run(build_command, shell=True, check=True)
    else:
        # Remote build via SSH
        ssh_command = f"ssh {node['host']} '{build_command}'"
        subprocess.run(ssh_command, shell=True, check=True)

Step 2: Artifact Collection

def collect_artifacts(node, version):
    binary_name = f"ballistics-{version}-{node['name']}"

    if node['host'] == 'local':
        shutil.copy(node['binary_path'], f"./{binary_name}")
    else:
        # Download via SCP
        scp_command = f"scp {node['host']}:{node['binary_path']} ./{binary_name}"
        subprocess.run(scp_command, shell=True, check=True)

Step 3: Checksum Generation

def generate_checksum(binary_path):
    with open(binary_path, 'rb') as f:
        sha256 = hashlib.sha256(f.read()).hexdigest()

    with open(f"{binary_path}.sha256", 'w') as f:
        f.write(sha256)

Step 4: Upload to GCS

def upload_to_gcs(version):
    bucket_path = f"gs://ballistics-releases/{version}/"

    # Upload binaries and checksums
    subprocess.run(f"gsutil -m cp ballistics-* {bucket_path}", shell=True)

    # Set public read permissions
    subprocess.run(f"gsutil -m acl ch -u AllUsers:R {bucket_path}*", shell=True)

    # Update latest-version.txt
    with open('latest-version.txt', 'w') as f:
        f.write(version)
    subprocess.run("gsutil cp latest-version.txt gs://ballistics-releases/", shell=True)

Running a Build

Dry-run (test without uploading):

cd build-orchestrator
./build.sh --version 0.13.4 --dry-run

Production build:

./build.sh --version 0.13.4

Output:

Building ballistics-engine v0.13.4
===========================================

Enabled build nodes: 7
- macos-aarch64 (local)
- macos-x86_64 (local)
- linux-x86_64 (alex@10.1.1.27)
- freebsd-aarch64 (freebsd@10.1.1.7)
- openbsd-aarch64-emulated (root@10.1.1.23)
- netbsd-x86_64 (root@10.1.1.19)
- netbsd-aarch64 (root@10.1.1.15)

Starting parallel builds...
[macos-aarch64] Building... (PID: 12345)
[linux-x86_64] Building... (PID: 12346)
...

Build results:
✓ macos-aarch64 (45s)
✓ linux-x86_64 (28s)
✓ freebsd-aarch64 (6m 32s)
✓ openbsd-aarch64-emulated (15m 37s)  ⚡ FASTEST ARM64!
...

Uploading to gs://ballistics-releases/0.13.4/
✓ Uploaded 7 binaries
✓ Uploaded 7 checksums
✓ Updated latest-version.txt

Build complete! 🎉
Total time: 16m 12s

Adding New Build Nodes

Interactive Script

The easiest way to add a new node is using the interactive script:

cd build-orchestrator
./add-node.sh

This will prompt you for: - Node name (e.g., openbsd-aarch64-emulated) - SSH host (e.g., root@10.1.1.23 or local) - Rust target triple (e.g., aarch64-unknown-openbsd) - Build commands (how to download and build) - Binary location (where the compiled binary is located)

Manual Configuration

Alternatively, edit build-nodes.yaml directly:

  - name: your-new-platform
    host: user@ip-address  # or 'local' for local builds
    target: rust-target-triple
    build_command: |
      # Commands to download source and build
      cd /tmp && rm -rf ballistics-engine-{version}
      curl -L -o v{version}.tar.gz https://github.com/...
      tar xzf v{version}.tar.gz
      cd ballistics-engine-{version}
      cargo build --release
    binary_path: /path/to/compiled/binary
    enabled: true

Variables: - {version}: Replaced with target version (e.g., 0.13.4) - {target}: Replaced with Rust target triple

Setting Up a New VM

Example: OpenBSD ARM64 Emulated

Create VM on host:

ssh alex@rig.localnet
cd /opt/bsd-vms/openbsd-arm64-emulated

Create boot script:

cat > boot.sh << 'EOF'
#!/bin/bash
exec qemu-system-aarch64 \
  -M virt,highmem=off \
  -cpu cortex-a57 \
  -smp 4 \
  -m 2G \
  -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd \
  -drive file=openbsd.qcow2,if=virtio,format=qcow2 \
  -netdev bridge,id=net0,br=br0 \
  -device virtio-net-pci,netdev=net0,romfile=,mac=52:54:00:12:34:99 \
  -nographic
EOF
chmod +x boot.sh

Create systemd service:

sudo cat > /etc/systemd/system/openbsd-arm64-emulated-vm.service << 'EOF'
[Unit]
Description=OpenBSD ARM64 VM (Emulated on x86_64)
After=network.target

[Service]
Type=simple
User=alex
WorkingDirectory=/opt/bsd-vms/openbsd-arm64-emulated
ExecStart=/opt/bsd-vms/openbsd-arm64-emulated/boot.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable openbsd-arm64-emulated-vm.service
sudo systemctl start openbsd-arm64-emulated-vm.service

Configure networking (assign static IP 10.1.1.23)
Install build tools inside VM:

ssh root@10.1.1.23
pkg_add rust git

Test SSH access:

ssh root@10.1.1.23 "cargo --version"

Add to build-nodes.yaml and test:

./build.sh --version 0.13.3 --dry-run

GitHub Webhook Integration (Optional)

For fully automated builds triggered by GitHub releases:

1. Deploy Webhook Receiver to Cloud Run

cd build-orchestrator
gcloud run deploy ballistics-build-webhook \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --set-env-vars GITHUB_WEBHOOK_SECRET=your-secret-here

2. Configure GitHub Webhook

Go to: https://github.com/yourusername/your-repo/settings/hooks
Add webhook:
Payload URL: https://ballistics-build-webhook-xxx.run.app/webhook
Content type: application/json
Secret: Your webhook secret
Events: Select "Releases" only

3. Test

Create a new release on GitHub, and the webhook will automatically trigger builds for all platforms!

Performance Metrics and Insights

From real-world builds of ballistics-engine v0.13.3:

Platform	Hardware	Build Time	Notes
macOS aarch64	Apple M1/M2	45s	Native Apple Silicon
macOS x86_64	Intel i7/i9	30s	Cross-compile on Apple Silicon
Linux x86_64	Xeon/EPYC	25s	Fastest overall ⚡
FreeBSD aarch64	Raspberry Pi 4	6m 32s	Native ARM64 hardware
OpenBSD aarch64 (emulated)	x86_64 QEMU	15m 37s	⚡ FASTEST ARM64
OpenBSD aarch64 (native)	Orange Pi 5 Max	99+ min	Disabled due to slower speed
NetBSD x86_64	x86_64 VM	3m 45s	KVM acceleration
NetBSD aarch64	Orange Pi VM	8m 12s	QEMU on ARM64 host

Key Insights:

x86_64 is fastest: Modern x86_64 CPUs dominate for single-threaded compilation
Emulation wins for ARM64: x86_64 emulating ARM64 beats native ARM64 SBCs
SBCs are viable: Raspberry Pi and Orange Pi work well for native builds, but slower
Parallel execution: Running all 7 builds in parallel takes only ~16 minutes (longest pole is FreeBSD ARM64)

Conclusion

Building a custom multi-platform build orchestration system may seem daunting, but the benefits are substantial:

→ Full control: Own your build infrastructure

→ Native builds: Real hardware ensures compatibility

→ Cost-effective: Low operational costs after initial hardware investment

→ Fast iteration: Parallel builds complete in ~16 minutes

→ Flexibility: Easy to add new platforms

→ Learning: Deep understanding of cross-platform development

The surprising discovery that emulated ARM64 on powerful x86_64 hardware outperforms native ARM64 single-board computers has practical implications: you don't always need native hardware for every architecture. Strategic use of emulation can provide better performance while maintaining compatibility.

For projects requiring broad platform support (especially BSD systems not well-served by traditional CI/CD), this approach offers a reliable, maintainable, and cost-effective solution.

Architecture Diagram

graph TB subgraph "Trigger Sources" GH[GitHub Release
v0.13.x] MANUAL[Manual Execution
./build.sh] end subgraph "Build Orchestrator" ORCH[Python Orchestrator
orchestrator.py] CONFIG[Build Configuration
build-nodes.yaml] end subgraph "Build Nodes - Local" MAC_ARM[macOS ARM64
Apple Silicon
~45s] MAC_X86[macOS x86_64
Rosetta 2
~30s] end subgraph "Build Nodes - Remote x86_64" LINUX_X86[Linux x86_64
alex@10.1.1.27
~25s] FREEBSD_X86[FreeBSD x86_64
root@10.1.1.21
~4m] OPENBSD_X86[OpenBSD x86_64
root@10.1.1.20
~12m] NETBSD_X86[NetBSD x86_64
root@10.1.1.19
~3m 45s] end subgraph "Build Nodes - Remote ARM64" FREEBSD_ARM[FreeBSD ARM64
freebsd@10.1.1.7
~6m 32s] OPENBSD_ARM_EMU[OpenBSD ARM64
root@10.1.1.23
Emulated on x86_64
~15m 37s ⚡] NETBSD_ARM[NetBSD ARM64
root@10.1.1.15
~8m 12s] end subgraph "Artifact Collection" COLLECT[SCP Collection
Pull binaries from nodes] CHECKSUM[Generate SHA256
checksums] end subgraph "Distribution" GCS[Google Cloud Storage
gs://ballistics-releases/] WEBSITE[ballistics.zip
Install Script] end GH -->|webhook| ORCH MANUAL -->|CLI| ORCH CONFIG -->|reads| ORCH ORCH -->|SSH parallel builds| MAC_ARM ORCH -->|SSH parallel builds| MAC_X86 ORCH -->|SSH parallel builds| LINUX_X86 ORCH -->|SSH parallel builds| FREEBSD_X86 ORCH -->|SSH parallel builds| OPENBSD_X86 ORCH -->|SSH parallel builds| NETBSD_X86 ORCH -->|SSH parallel builds| FREEBSD_ARM ORCH -->|SSH parallel builds| OPENBSD_ARM_EMU ORCH -->|SSH parallel builds| NETBSD_ARM MAC_ARM -->|binary| COLLECT MAC_X86 -->|binary| COLLECT LINUX_X86 -->|binary| COLLECT FREEBSD_X86 -->|binary| COLLECT OPENBSD_X86 -->|binary| COLLECT NETBSD_X86 -->|binary| COLLECT FREEBSD_ARM -->|binary| COLLECT OPENBSD_ARM_EMU -->|binary| COLLECT NETBSD_ARM -->|binary| COLLECT COLLECT --> CHECKSUM CHECKSUM --> GCS GCS --> WEBSITE style OPENBSD_ARM_EMU fill:#90EE90 style LINUX_X86 fill:#87CEEB style GCS fill:#FFD700 style WEBSITE fill:#FFD700

Diagram Legend

Green: Fastest ARM64 build (emulated on powerful x86_64)
Blue: Fastest overall build (native Linux x86_64)
Yellow: Distribution endpoints

Build Flow

Trigger: GitHub release webhook or manual execution
Parallel Execution: All enabled build nodes start simultaneously
Collection: Orchestrator collects binaries via SCP
Verification: SHA256 checksums generated for integrity
Upload: Binaries and checksums uploaded to GCS
Availability: Install script immediately serves new version