Two Addresses, One Page: Finding the Firmware Bug Behind NetBSD Corruption on the Milk-V Mars

A.C. Jokela — Sat, 05 Sep 2026 14:00:00 GMT

Two Addresses, One Page: Finding the Firmware Bug Behind NetBSD Corruption on the Milk-V Mars

The page was supposed to contain a fixed pattern. Instead it began with this:

pkgsrc/print/tex-convbkmk-doc

That string had just come out of a compressed pkgsrc archive. It belonged in a tmpfs file. It had no business appearing in a permanent page of NetBSD kernel memory, much less in one of 256 pages I had allocated for the sole purpose of catching corruption. The kernel still believed it owned the page. Its allocation record was unchanged. Every audit I had added to the page allocator, pmap, direct-copy path, and SD controller reported zero events.

The page had a physical address of 0x23e6d6000. A live tmpfs page containing the same pathname sat at 0x13e6d6000. The difference was exactly 0x100000000: four gibibytes.

In July I wrote Four Partitions and a Borrowed Bootloader, the story of getting NetBSD-current to boot on my Milk-V Mars. That article ended with a working SSH prompt and one unresolved caveat: sustained writes could panic the kernel. I thought the board was running NetBSD with a RISC-V virtual-memory bug waiting to be fixed upstream.

I was wrong. At least one pair of physical addresses was reaching the same CPU-visible memory. NetBSD could track both pages perfectly and still lose, because the distinction disappeared below the caches. Finding that took a reliable reproducer, several instrumented kernels, a small custom QEMU machine, a serial console, a correction to my own public diagnosis, and two one-line firmware fixes involving two different device trees.

Booting was only the first half

The first article covers the bring-up in detail, but the shape of it matters here. A Mars SD card needs four partitions. The JH7110 BootROM finds an SPL by a special GPT type GUID. SPL initializes DDR and loads an OpenSBI/U-Boot FIT from a second GUID. U-Boot starts NetBSD's EFI loader from a FAT partition, and that loader reads /netbsd from an FFS root filesystem.

NetBSD supplied the last two pieces. I borrowed mainline U-Boot 2025.01 from Debian's VisionFive 2 package for the first two, then scripted the whole image in milkv-netbsd. That got the machine through all four stages, but it initially handed NetBSD a VisionFive 2 device tree. The board's EEPROM is blank, so U-Boot never constructed fdtfile; its empty default actually deleted the variable. Compiling this into U-Boot selected the right payload device tree:

CONFIG_DEFAULT_FDT_FILE="starfive/jh7110-milkv-mars.dtb"

The Mars device tree fixed the Ethernet PHY configuration. Packet loss went from roughly 80 percent to zero, and the board became a stable SSH target. I called that the one-line ending to the device-tree saga.

Then I extracted pkgsrc.

The first failure was an assertion in pmap_segtab.c from the pagedaemon. Nick Hudson fixed a separate deferred pmap activation bug in NetBSD's RISC-V code, and that assertion disappeared. The machine still corrupted memory. Another run died in the UBC list code. A Rust bootstrap extraction reached the same failure. Compiling Rust entirely in tmpfs eventually produced a fatal load fault with 0x808080... smeared through a pointer.

Those variants removed several convenient suspects. The failure happened with one CPU online, so it was not an ordinary SMP race. It happened with Ethernet down, so it was not the network controller writing through a bad DMA address. It happened on tmpfs as well as FFS, with plenty of free memory, and it did not happen under qemu -M virt. Four workers repeatedly extracting and deleting pkgsrc in a large tmpfs turned the mystery into a reproducer that usually failed in about a minute.

That was enough to debug, though not yet enough to understand.

A very convincing wrong answer

UBC is NetBSD's unified buffer cache, the layer that connects file data to virtual memory. Early in boot, ubc_init() allocates arrays used to track its mappings and hash buckets. Those pages remain kernel-owned for the life of the system. In one crash, exactly one 4 KiB page of a UBC array had been overwritten while the pages on either side remained intact. In another, the damaged page was full of data from the archive being extracted.

The obvious story was that UVM had lost track of a physical page. Perhaps a boot-time reservation overlapped a range later passed to uvm_page_physload(). Perhaps one physical page had acquired two struct vm_page descriptors. UBC would still think it owned its permanent allocation while tmpfs received the same page and filled it with file data. Page-aligned damage, two logical owners, one physical page: every observation seemed to fit.

I sent that diagnosis to NetBSD's port-riscv list. The page-aligned evidence was real. The conclusion was wrong.

The first instrument, UBC_PAGE_GUARD, recorded the physical pages behind those permanent arrays and watched for the allocator to return one again. Its fatal version could call panic() while allocator locks were held. When the board began wedging silently during autoconfiguration, I blamed the guard for finding the bug too early to print it. A nonfatal version wedged too. So did an untouched kernel. The instrument had become another plausible explanation for a failure it did not cause.

This was expensive because the original firmware could not reset the Mars reliably. A reboot ended at a PMIC error and every new test needed a physical power cycle. Before spending more of those, I built a faster place to make mistakes.

Build a witness before another theory

mars-virt is a small QEMU hybrid: Mars RAM placement, its four hart IDs, the U74 architectural subset, and its 4 MHz timer, combined with virtual I/O. It is deliberately not a JH7110 emulator. It cannot model the board's DDR controller or physical cache behavior, and it never reproduced the corruption. Its job was to boot instrumented NetBSD kernels in minutes and tell me when the instruments lied.

That happened immediately. The first full PGAUDIT run reported a double free. Following the event through UVM showed that my allocation hook sat after a legitimate failure path: UVM freed a page and returned NULL before the hook had recorded the allocation. Moving the hook to the point where a page had definitely left the free lists made the event disappear under the same load. The emulator saved a hardware boot and, more usefully, kept a false positive out of the final evidence.

PGAUDIT grew into a set of independent witnesses:

a boot-time audit compared every physical segment for overlap;
a PA-indexed bitmap watched normal and page-list allocations for double allocation, double free, and pages outside the tracked range;
owner records kept the previous and current allocating program counters;
UBC checks re-resolved every permanent KVA and compared it with the direct-map view, looking for remapping or a disagreement consistent with a stale translation or cache-coherency problem;
hooks watched direct copies, page zeroes, page copies, and foreign mappings;
the SD/MMC path counted final DMA descriptors at or above 4 GiB;
256 permanently owned pages near the top of RAM carried patterns derived from their physical addresses.

The canaries were intentionally boring. Nothing should write them after initialization. They also widened the search beyond UBC, because a second panic had already destroyed a pool page-header tree rather than a UBC array. Registers at that crash contained fragments of pkgsrc pathnames: eneral/m and text/doc. UBC was not uniquely cursed. It was simply a frequent victim near the top of memory.

On mars-virt, PGAUDIT booted with four CPUs, recorded 104 UBC pages, and completed repeated extraction cycles with every counter at zero. That proved the diagnostic could survive the workload. It said nothing about the board, which was exactly the boundary I needed it to keep.

Every counter stayed zero

The hardware result first looked like a disappointment. Across successive runs, the ownership and UBC-mapping checks remained clean. Later kernels added the direct-copy, foreign-mapping, canary, and final MMC-descriptor checks. At the decisive panic, every counter other than the canary check was still zero:

physseg_overlaps = 0
double_alloc     = 0
double_free      = 0
untracked        = 0
direct_hits      = 0
map_hits         = 0
dma_high         = 0
104 KVAs, 0 remapped, 0 stale

One UBC page began, exactly on its boundary, with /DESCR/1.1/Sun May 3 05:30:35, a CVS Entries line present in the pkgsrc archive. The pool corruption showed that arbitrary kernel pages could be hit. Because the text appeared only after decompression, a disk read was not spraying compressed input into RAM. An ordinary CPU copy into a legitimate tmpfs page was somehow also changing kernel-owned memory.

The decisive kernel was PGAUDIT build number ten. Its four-worker run reached 1,722,848 KiB in tmpfs at 60 seconds and faulted in ubc_alloc(). A hash-bucket head had become the invalid pointer 0x100000000. From DDB, the canary check reported 64 damaged pages out of 256. Canary 25 held the pkgsrc/print/tex-convbkmk-doc pathname. Its audit state still said allocated, and its original owner still resolved to ubc_init().

PGAUDIT still recorded the high page as owned by ubc_init(), while the lower page had a separate descriptor and an allocation record from uao_get(), exactly where a live tmpfs page should come from. The evidence now pointed to two separately tracked physical addresses rather than a recorded ownership transition. The lower address was four GiB below the damaged canary.

That changed the question. I no longer needed to find a missing ownership transition. I needed to learn whether those two addresses were actually different once a memory access left the CPU caches.

Two addresses, one page

NetBSD's RISC-V direct map gives the kernel a simple virtual address for each physical address. In DDB, pmap_extract() walked the real page tables and confirmed the mappings:

Direct-map virtual address	Physical address
`0xffffffe13e6d6000`	`0x13e6d6000`
`0xffffffe23e6d6000`	`0x23e6d6000`

The virtual-memory machinery saw two legitimate mappings to two physical addresses. Both views initially contained the same pkgsrc pathname. That alone was suggestive, but caches made it ambiguous. Two independent physical cache lines can temporarily contain the same bytes even if the memory behind them differs. I needed to change one and force both back to backing memory.

I wrote 0x11223344 through the lower direct-map address. The lower view changed; the upper view still showed the old pkgs bytes:

low:  11223344 702f6372
high: 73676b70 702f6372

That mismatch was useful. It showed that the two addresses had distinct cache-resident lines, so DDB was not truncating an address or displaying one location twice. I then called NetBSD's fu540_ccache_cache_wbinv_range() for each 64-byte line. The documented U74 Flush64 operation back-probes L1 and writes back and invalidates the relevant L1/L2 line. Assuming the JH7110 cache block implements those semantics, reading both addresses again forced them to refill:

low:  11223344 702f6372
high: 11223344 702f6372

The marker had crossed a boundary that NetBSD, the page tables, and the cache tags all treated as real. The common point lay downstream of L2 under that Flush64 assumption. Under this firmware's 8 GiB configuration, the tested physical addresses four GiB apart reached the same CPU-visible DDR backing.

This aliasing mechanism explains the whole strange history. UVM was free to allocate the lower and upper pages to unrelated owners because their physical addresses differed. Stores to the tmpfs page could then change the kernel page at the matching offset in the other four-GiB half. Damage began on page boundaries because file writes target pages. It became common only when enough memory was in use to reach the upper addresses. QEMU remained clean because its RAM backend really had separate storage for every advertised address.

It does not prove that the board physically contains only four GiB. The unit may be a four-GiB SKU advertised as eight, an eight-GiB topology may have been programmed incorrectly, the controller may have a decode fault, or the hardware may be defective. What the experiment establishes is narrower and sufficient: this firmware exposed an eight-GiB range in which at least one tested pair, four GiB apart, shared backing memory. NetBSD could not safely use the upper half.

Put up a four-GiB fence

Before changing firmware, I tested that conclusion in the kernel. A temporary MEMLIMIT stopped physical memory at the absolute exclusive address 0x140000000, preserving [0x40000000, 0x140000000). The word absolute matters. Firmware reservations made NetBSD's runtime start 0x40060000; adding four GiB to that value would have admitted 384 KiB of the aliased upper aperture.

The capped kernel reported 4,075 MB total and ran the same four-worker, 2,500 MiB tmpfs workload for 600 seconds. The uncapped control had failed after 60 seconds under less pressure. During the capped run, tmpfs repeatedly grew and emptied, memory pressure killed and restarted tar children, and all four worker loops were still alive when the script stopped them. Every PGAUDIT counter, all 256 canaries, and all 104 UBC mappings remained clean.

Ten failure windows were enough to make the address limit a strong containment result. It was still a kernel workaround for a platform description that claimed unsafe memory. The next job was finding who had made that claim.

There were two device trees

The Mars device tree on the EFI partition describes four GiB and contains no reserved-memory entries. Yet the affected boot advertised an eight-GiB range from 0x40000000 to 0x240000000. NetBSD's EFI loader rebuilt /memory from the EFI memory map supplied by U-Boot, so the static node on disk did not determine the ranges NetBSD ultimately managed. The disk file was telling the truth and still losing the argument.

The trap was that this boot chain uses device trees at two different times. The payload tree describes the finished board to NetBSD. That is the tree selected by CONFIG_DEFAULT_FDT_FILE, the first one-line fix, and it repaired Ethernet. Much earlier, SPL needs its own built-in control device tree to initialize DDR before full U-Boot or the operating-system payload exists.

Mainline U-Boot uses the VisionFive 2 configuration for the Mars. In U-Boot 2024.04, jh7110-starfive-visionfive-2.dtsi declares a bank at 0x40000000 with size cells <0x2 0x0>: eight GiB. SPL loads that value, then replaces it only if the board EEPROM reports a valid capacity of 2, 4, 8, or 16 GiB. This Mars has a blank EEPROM. No override occurs. The DDR driver selects DDR_SIZE_8G, and the same size eventually reaches NetBSD.

The blank EEPROM had now caused two independent failures. In full U-Boot it prevented selection of the Mars payload tree, which broke Ethernet. In SPL it left the VisionFive 2 control tree's eight-GiB default in place, which exposed the alias. Fixing fdtfile could never change DDR geometry because DDR had already been initialized before that variable mattered.

Milk-V's own Mars SDK supplied the missing clue. Its no-EEPROM path defaults to four GiB and keeps that value unless it reads a validated capacity. The 4G and 8G controller tables otherwise agreed with mainline. Rather than write a guessed capacity into the blank EEPROM, I changed the control tree's size by one cell:

- reg = <0x0 0x40000000 0x2 0x0>;
+ reg = <0x0 0x40000000 0x1 0x0>;

That line does two things. It selects the four-GiB DDR geometry and tells later firmware to advertise one four-GiB bank. Because both change together, the test deliberately leaves the upper addresses unprobed.

Firmware had a second confession

There was another result hiding in the boot failures I had blamed on diagnostic kernels. U-Boot 2025.01 with OpenSBI 1.6 failed three of four measured hardware boots. Two kernels wedged silently during early autoconfiguration. One boot never reached NetBSD and instead printed repeated OpenSBI load-access faults from a secondary hart. The same failure points appeared with instrumented and untouched kernels.

A U-Boot 2024.04/OpenSBI 1.4 pair had been staged for comparison since July. Once installed, every subsequent boot succeeded. Normal /sbin/reboot began working too, turning a test loop that had required a person at the power cable into an autonomous 32-second reboot. The older firmware did not fix the memory problem by itself: its unmodified control tree still selected eight GiB, and the corruption remained. It separated boot stability from memory geometry, then gave me a stable base for the one-line four-GiB change.

I rebuilt that pair from pinned source archives, preserving the known configuration byte for byte and changing only the control-tree line. The build verified that the SPL and U-Boot copies of the four-GiB control tree were embedded in their payloads.

Flashing the card from the card

Installing the corrected firmware did not require moving the microSD card to my laptop. NetBSD was already running from it, so I copied the two payloads over SSH, identified the live spl and uboot wedges by GPT label and type GUID, retained the baseline payloads for rollback, and wrote only those wedges. The EFI and FFS partitions were not written.

Even that produced one last small trap. The SPL was 141,027 bytes. NetBSD's raw wedge accepted the first 275 complete sectors and rejected the final 227-byte fragment. The board was still running and had not rebooted into the incomplete write, so I fixed the flash script to pad only the final sector with bs=512 conv=sync, rewrote both payloads, and verified hashes over their original lengths. The script now refuses to write unless the live disk mapping, labels, GUIDs, and expected payload hashes all agree.

The final hardware A/B used the same uncapped PGAUDIT kernel that had failed at 60 seconds under the eight-GiB firmware. With the corrected firmware, U-Boot printed DRAM: 4 GiB, bdinfo showed one bank ending at 0x13fffffff, and NetBSD received an end address of 0x140000000. It reported 4,075 MB without any kernel memory limit.

Then the same stress test ran for 600 seconds. Zero double allocations. Zero double frees. Zero suspicious direct writes or foreign mappings. Zero high DMA descriptors. Zero damaged canaries. All 104 UBC mappings still agreed.

The image builder has the same attitude. It defaults to the exact 2024.04/OpenSBI 1.4 firmware tested on this board, verifies both four-GiB control trees and their embedded copies, and ignores the old top-level eight-GiB files rather than falling back to them. build-uboot.sh rebuilds the patched firmware from pinned archives and performs the same checks. A clean end-to-end image build passed without an SD card attached.

What “running” means now

The corrected firmware succeeded in both validation boots, while the underlying 2024.04/OpenSBI 1.4 pair had already made repeated boots and software resets reliable. NetBSD-current starts all four U74 cores, selects the Mars payload device tree, and receives a safe 4,075 MB physical-memory map. The exact uncapped PGAUDIT kernel that failed under the eight-GiB firmware completed the 600-second extraction test under the corrected firmware. Ethernet is stable, although NetBSD's eqos transmit path still tops out around 2.45 Mbit/s on this board; receive performance is far better. There is more driver work available, but the reproducer no longer turns pathname text into kernel pointers.

The live Mars after the firmware correction, running the exact uncapped PGAUDIT #10 kernel used in the A/B.

The old netbsd-mars-20260719.img.xz linked from my July article contains the affected U-Boot 2025.01/OpenSBI 1.6 eight-GiB firmware. Do not use that image for sustained writes or data you care about. Current source builds from the repository use the corrected, hardware-tested firmware. The older artifact documents the original work; it is not a safe installation image.

I still do not know how much DRAM is fitted to this particular board, or whether different topology programming could make all eight advertised GiB usable. The tested pair aliased under the eight-GiB configuration. Restricting the platform to the first four GiB contained the failure in both a NetBSD A/B and a firmware A/B. That is enough to run the machine safely and too little to advertise an eight-GiB fix.

My first article ended with one line selecting the right device tree for the operating system. This one ends with one line correcting the private device tree SPL had already used before the operating system existed. The first made the Mars reachable. The second made its memory trustworthy. The lines were tiny. Proving which one told the hardware the wrong story was the project.

TinyComputers.io (Posts about firmware)

Two Addresses, One Page: Finding the Firmware Bug Behind NetBSD Corruption on the Milk-V Mars

Two Addresses, One Page: Finding the Firmware Bug Behind NetBSD Corruption on the Milk-V Mars

Booting was only the first half

A very convincing wrong answer

Build a witness before another theory

Every counter stayed zero

Two addresses, one page

Put up a four-GiB fence

There were two device trees

Firmware had a second confession

Flashing the card from the card

What “running” means now