TinyComputers.io (Posts about strix halo)

Upgrading ROCm 7.0 to 7.2 on AMD Strix Halo (gfx1151)

A.C. Jokela — Wed, 18 Feb 2026 16:00:00 GMT

Introduction

If you're running AMD's Strix Halo hardware -- specifically the Ryzen AI MAX+ 395 with its integrated Radeon 8060S GPU -- you already know the software ecosystem is a moving target. The gfx1151 architecture sits in an awkward spot: powerful hardware that isn't officially listed on AMD's ROCm support matrix, yet functional enough to run real workloads with the right driver stack. When ROCm 7.2 landed in early 2026, upgrading from 7.0.2 was a priority. The newer stack brings an updated HSA runtime, a refreshed amdgpu kernel module, and broader compatibility improvements that matter on bleeding-edge silicon.

This post documents the complete upgrade procedure from ROCm 7.0.2 to 7.2 on a production Ubuntu 24.04 system. It's not a theoretical exercise -- this was performed on a live server running QEMU virtual machines and network services, with the expectation that everything would come back online after a single reboot.

AMD's official documentation states that in-place ROCm upgrades are not supported. The recommended path is a full uninstall followed by a clean reinstall. That's exactly what we did, and the entire process took about 20 minutes of wall-clock time (excluding the reboot).

System Overview

The target system is a Bosgame mini PC running the Ryzen AI MAX+ 395 APU. If you've read the earlier review of this hardware, you'll be familiar with the specs. For context on this upgrade, here's what matters:

Hardware

CPU: AMD Ryzen AI MAX+ 395, 16 cores / 32 threads, Zen 5
GPU: Integrated Radeon 8060S, 40 Compute Units, RDNA 3.5 (gfx1151)
Memory: 32 GB DDR5, unified architecture with 96 GB allocatable to GPU
Peak GPU Clock: 2,900 MHz

Software (Pre-Upgrade)

OS: Ubuntu 24.04.3 LTS (Noble Numbat)
Kernel: 6.14.0-37-generic (HWE, pinned)
ROCm: 7.0.2
amdgpu-dkms: 6.14.14 (from repo.radeon.com/amdgpu/30.10.2)
ROCk Module: 6.14.14

Running Services

The system was actively serving several roles during the upgrade:

Five QEMU virtual machines (three x86, two aarch64)
A PXE boot server (dnsmasq) for the local network
Docker daemon with various containers

None of these services are tied to the GPU driver stack, so the plan was to perform the upgrade and reboot without shutting them down first. The VMs and network services would come back automatically after the reboot.

Why Upgrade

ROCm 7.0.2 worked on this hardware. Models loaded, inference ran, rocminfo detected the GPU. So why bother upgrading?

Three reasons:

Driver maturity for gfx1151: The amdgpu kernel module jumped from 6.14.14 to 6.16.13 between the two releases. That's not a minor revision -- it represents months of kernel driver development. On hardware that isn't officially supported, newer drivers tend to bring meaningful stability improvements as AMD's internal teams encounter and fix issues on adjacent architectures.
HSA Runtime improvements: ROCm 7.2 ships HSA Runtime Extension version 1.15, up from 1.11 in ROCm 7.0.2. The HSA (Heterogeneous System Architecture) runtime is the lowest layer of the ROCm software stack -- it handles device discovery, memory management, and kernel dispatch. Improvements here affect everything built on top of it.
Ecosystem alignment: PyTorch wheels, Ollama builds, and other ROCm-dependent tools increasingly target 7.2 as the baseline. Running 7.0.2 was becoming an exercise in version pinning and compatibility workarounds.

The Kernel Hold: Why It Matters

Before diving into the procedure, a note on kernel management. This system runs the Ubuntu HWE (Hardware Enablement) kernel, which provides newer kernel versions on LTS releases. At the time of this upgrade, the HWE kernel was 6.14.0-37-generic. The upstream kernel had already moved to 6.17, but we didn't want the ROCm upgrade to pull in a kernel that AMD's DKMS module might not build against.

The solution is apt-mark hold:

sudo apt-mark hold linux-generic-hwe-24.04 linux-headers-generic-hwe-24.04 linux-image-generic-hwe-24.04

This prevents apt from upgrading the kernel meta-packages, effectively pinning the system to 6.14.0-37-generic. The hold was already in place before the upgrade and remained untouched throughout. After the upgrade, we confirmed it was still active:

apt-mark showhold

linux-generic-hwe-24.04
linux-headers-generic-hwe-24.04
linux-image-generic-hwe-24.04

If you're running Strix Halo or any other hardware where kernel compatibility with amdgpu-dkms is uncertain, kernel holds are essential. A kernel upgrade that breaks the DKMS build means no GPU driver after reboot.

Upgrade Procedure

Step 1: Uninstall the Current ROCm Stack

AMD provides the amdgpu-uninstall script for exactly this purpose. It removes all ROCm userspace packages and the amdgpu-dkms kernel module in a single operation:

sudo amdgpu-uninstall -y

This command removed approximately 120 packages, including the full HIP runtime, rocBLAS, MIOpen, MIGraphX, ROCm SMI, the LLVM-based compiler toolchain, and the Mesa graphics drivers that ship with ROCm. The DKMS module was purged, which means the amdgpu kernel module was removed from the 6.14.0-37-generic kernel's module tree.

After the ROCm stack was removed, we purged the amdgpu-install meta-package itself:

sudo apt purge -y amdgpu-install

This also cleaned up the APT repository entries that amdgpu-install had configured in /etc/apt/sources.list.d/. The old repos -- repo.radeon.com/amdgpu/30.10.2, repo.radeon.com/rocm/apt/7.0.2, and repo.radeon.com/graphics/7.0.2 -- were all removed automatically.

Step 2: Clean Up Leftover Files

The package removal was thorough but not perfect. A few leftover directories remained in /opt/:

ls /opt/ | grep rocm

rocm-7.0.0
rocm-7.0.2
rocm-7.9.0

The rocm-7.0.0 directory was from a previous installation attempt. The rocm-7.9.0 was from an earlier experiment with a release candidate build. The rocm-7.0.2 directory contained a single orphaned shared library (libamdhip64.so.6) that dpkg couldn't remove because the directory wasn't empty. All three were cleaned up manually:

sudo rm -rf /opt/rocm-7.0.0 /opt/rocm-7.0.2 /opt/rocm-7.9.0

It's worth checking for stale ROCm directories after any uninstall. They consume negligible disk space but can confuse build systems and scripts that scan /opt/rocm* for active installations.

Step 3: Install the ROCm 7.2 Installer

AMD distributes ROCm through a meta-package called amdgpu-install. Each ROCm release has its own version of this package, which configures the appropriate APT repositories. The 7.2 installer was downloaded directly from AMD's repository:

cd /tmp
wget https://repo.radeon.com/amdgpu-install/7.2/ubuntu/noble/amdgpu-install_7.2.70200-1_all.deb
sudo apt install -y ./amdgpu-install_7.2.70200-1_all.deb
sudo apt update

After installation and apt update, three new repositories were active:

https://repo.radeon.com/amdgpu/30.30/ubuntu noble -- the kernel driver and Mesa components
https://repo.radeon.com/rocm/apt/7.2 noble -- the ROCm userspace stack
https://repo.radeon.com/graphics/7.2/ubuntu noble -- graphics libraries

The version numbering can be confusing. The amdgpu-install package version is 30.30.0.0.30300000-2278356.24.04, which maps to the amdgpu driver release 30.30. The ROCm version is 7.2.0. These are different version tracks that AMD maintains in parallel.

Step 4: Install ROCm 7.2

With the repositories configured, the actual installation was a single command:

sudo amdgpu-install -y --usecase=graphics,rocm

The --usecase=graphics,rocm flag tells the installer to include both the Mesa graphics drivers and the full ROCm compute stack. This is the right choice for a system that needs both display output and GPU compute capabilities.

The installation took approximately 10 minutes and included:

amdgpu-dkms 6.16.13: The kernel module, compiled via DKMS against the running kernel
Full ROCm 7.2 stack: HIP runtime, hipcc compiler, rocBLAS, rocFFT, MIOpen, MIGraphX, RCCL, ROCm SMI, ROCProfiler, and dozens of other libraries
Mesa graphics: Updated EGL, OpenGL, and Vulkan drivers from the amdgpu Mesa fork
ROCm LLVM toolchain: The LLVM-based compiler infrastructure that HIP uses for kernel compilation

The DKMS build is the critical step. During installation, DKMS compiled the amdgpu module against the kernel headers for 6.14.0-37-generic. The output confirmed a successful build:

depmod...
update-initramfs: Generating /boot/initrd.img-6.14.0-37-generic

The initramfs was regenerated to include the new module, ensuring it would be loaded at boot.

Step 5: Verify DKMS

Before rebooting, we confirmed the DKMS status:

dkms status

amdgpu/6.16.13-2278356.24.04, 6.14.0-37-generic, x86_64: installed
virtualbox/7.0.16, 6.14.0-36-generic, x86_64: installed
virtualbox/7.0.16, 6.14.0-37-generic, x86_64: installed
virtualbox/7.0.16, 6.8.0-100-generic, x86_64: installed

The new amdgpu module (6.16.13) was built and installed for 6.14.0-37-generic. Note that it only built for the currently running kernel, unlike VirtualBox which had modules built for older kernels as well. This is expected -- DKMS builds against available kernel headers, and the old kernel headers for 6.14.0-36 and 6.8.0-100 were still present from earlier installations.

Step 6: Reboot

sudo reboot

The server came back online in approximately 50 seconds.

Post-Reboot Verification

rocminfo

The first check after reboot was rocminfo, which queries the HSA runtime for available agents:

rocminfo

ROCk module version 6.16.13 is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.18
Runtime Ext Version:     1.15
...
==========
HSA Agents
==========
Agent 1: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (CPU)
Agent 2: gfx1151 (GPU)
  Marketing Name:          AMD Radeon Graphics
  Compute Unit:            40
  Max Clock Freq. (MHz):   2900
  Memory Properties:       APU
  ISA 1: amdgcn-amd-amdhsa--gfx1151
  ISA 2: amdgcn-amd-amdhsa--gfx11-generic

Key observations:

ROCk module 6.16.13: The new kernel module loaded successfully.
Runtime Ext Version 1.15: Upgraded from 1.11 in ROCm 7.0.2.
gfx1151 detected: The GPU was recognized with its correct ISA identifier.
gfx11-generic ISA: ROCm 7.2 also exposes a generic gfx11 ISA, which allows software compiled for the broader RDNA 3 family to run on this device without gfx1151-specific builds.
APU memory: The memory properties correctly identify this as an APU with unified memory.

ROCm SMI

rocm-smi

Device  Node  Temp    Power     SCLK  MCLK     Fan  Perf  VRAM%  GPU%
0       1     33.0C   9.087W    N/A   1000Mhz  0%   auto  0%     0%

The GPU was visible and reporting telemetry. The 0% VRAM reading is expected on an APU -- rocm-smi reports dedicated VRAM usage, but on a unified memory architecture, GPU memory allocations come from system RAM and aren't reflected in this counter.

ROCm Version

cat /opt/rocm/.info/version

7.2.0

DKMS

dkms status

Confirmed amdgpu/6.16.13 remained installed for 6.14.0-37-generic after reboot.

PyTorch Validation

With the driver stack verified, the next step was confirming that PyTorch could see and use the GPU. ROCm 7.2 ships with prebuilt PyTorch wheels on AMD's repository.

Installing PyTorch for ROCm 7.2

We set up a Python virtual environment and installed the ROCm-specific wheels:

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

The PyTorch wheel for ROCm 7.2 requires a matching ROCm-specific build of Triton. Both are available from AMD's manylinux repository. The order matters -- Triton must be installed first, since the PyTorch wheel declares it as a dependency with a specific version that doesn't exist on PyPI:

pip install https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/triton-3.5.1%2Brocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl
pip install https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torch-2.9.1%2Brocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl
pip install https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchvision-0.24.0%2Brocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl

These are the ROCm 7.2 builds for Python 3.12. AMD also provides wheels for Python 3.10, 3.11, and 3.13.

Smoke Test

import torch
print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("Device:", torch.cuda.get_device_name(0))
print("VRAM:", round(torch.cuda.get_device_properties(0).total_memory / 1e9, 1), "GB")

PyTorch: 2.9.1+rocm7.2.0.git7e1940d4
CUDA available: True
Device: AMD Radeon Graphics
VRAM: 103.1 GB

PyTorch detected the GPU through ROCm's HIP-to-CUDA translation layer. The 103.1 GB figure represents the total addressable memory on this unified-memory APU, which includes both the 96 GB GPU allocation and additional system memory accessible through the HSA runtime.

Note the use of torch.cuda despite this being an AMD GPU. ROCm's HIP runtime presents itself through PyTorch's CUDA interface, so all CUDA API calls in PyTorch (device selection, memory management, kernel launches) work transparently with AMD hardware.

Before and After Summary

Component	ROCm 7.0.2	ROCm 7.2.0
ROCm Version	7.0.2	7.2.0
amdgpu-dkms	6.14.14	6.16.13
ROCk Module	6.14.14	6.16.13
HSA Runtime Ext	1.11	1.15
amdgpu Repo	30.10.2	30.30
PyTorch	N/A	2.9.1+rocm7.2.0
Triton	N/A	3.5.1+rocm7.2.0
Kernel	6.14.0-37-generic	6.14.0-37-generic
Kernel Holds	In place	In place

Notes on gfx1151 Support

It's worth being explicit about the support situation. As of February 2026, gfx1151 (Strix Halo) is not listed on AMD's official ROCm support matrix. The supported RDNA 3 targets are gfx1100 (Navi 31, RX 7900 XTX) and gfx1101 (Navi 32). Strix Halo's gfx1151 is an RDNA 3.5 derivative that shares much of the ISA with gfx1100 but has architectural differences in the memory subsystem and compute unit layout.

In practice, ROCm 7.2 works on gfx1151. The kernel driver loads, rocminfo detects the GPU, and PyTorch can allocate tensors and dispatch compute kernels. The gfx11-generic ISA target in ROCm 7.2 is particularly helpful -- it provides a compatibility path for software that hasn't been explicitly compiled for gfx1151.

However, "works" and "fully supported" are different things. There are known quirks:

rocm-smi VRAM reporting: Always shows 0% on the APU since it only tracks discrete VRAM
No official PyTorch gfx1151 builds: The ROCm PyTorch wheels target gfx1100. They run on gfx1151 through ISA compatibility, but performance may not be optimal
Large model loading latency: Moving large models to the GPU device can be slow on the unified memory architecture, as the HSA runtime handles page migration differently than discrete GPU DMA transfers

If you're considering this hardware for production AI workloads, treat ROCm support as "functional but experimental." It works well enough for development, testing, and moderate inference workloads. For production training or latency-sensitive deployment, stick with hardware on AMD's official support list.

Rollback Plan

If the upgrade fails -- the DKMS module doesn't build, the GPU isn't detected after reboot, or something else goes wrong -- the rollback path is straightforward:

Uninstall ROCm 7.2:

sudo amdgpu-uninstall -y
sudo apt purge -y amdgpu-install

Reinstall ROCm 7.0.2:

wget https://repo.radeon.com/amdgpu-install/30.10.2/ubuntu/noble/amdgpu-install_30.10.2.0.30100200-2226257.24.04_all.deb
sudo apt install -y ./amdgpu-install_30.10.2.0.30100200-2226257.24.04_all.deb
sudo apt update
sudo amdgpu-install -y --usecase=graphics,rocm
sudo reboot

The entire rollback takes about 15 minutes. Keep the old amdgpu-install deb URL handy -- it's not linked from AMD's current download pages once a newer version is published.

Conclusion

Upgrading ROCm on hardware that isn't officially supported always carries some risk, but this upgrade from 7.0.2 to 7.2 on gfx1151 was uneventful. The procedure follows AMD's documented uninstall-reinstall approach with no deviations. The kernel hold strategy kept the kernel stable, the DKMS module built cleanly against 6.14.0-37-generic, and all post-reboot checks passed.

The improvements in ROCm 7.2 -- particularly the HSA runtime bump to 1.15 and the introduction of the gfx11-generic ISA target -- represent meaningful progress for Strix Halo users. The ecosystem is slowly catching up to the hardware. It's not there yet, but each release closes the gap.

For anyone running a Ryzen AI MAX+ 395 or similar Strix Halo hardware on Ubuntu 24.04, this upgrade is worth doing. The procedure is well-defined, the rollback path is clear, and the newer driver stack brings tangible benefits. Just remember to hold your kernel first.

Recommended Resources

Hardware

Bosgame M5 AI Mini PC (Ryzen AI MAX+ 395) - The system used in this post
GMKtec EVO X2 (Ryzen AI MAX+ 395) - Another Strix Halo mini PC option on Amazon

Books

Deep Learning with PyTorch by Stevens, Antiga, Huang, Viehmann - Comprehensive guide to building, training, and tuning neural networks with PyTorch
Programming PyTorch for Deep Learning by Ian Pointer - Practical guide to creating and deploying deep learning applications
Understanding Deep Learning by Simon Prince - Modern treatment of deep learning fundamentals

Partial LLM Loading: Running Models Too Big for VRAM

A.C. Jokela — Thu, 05 Feb 2026 16:00:00 GMT

What happens when you want to run a 70B parameter model but only have 24GB of VRAM? Traditionally, you either quantize aggressively, rent cloud GPUs, or accept that the model is simply out of reach. But there's a third option that's becoming increasingly viable: partial loading, where you keep some layers on the CPU or disk and stream them to the GPU on demand.

I spent a couple days testing partial loading strategies on an AMD Strix Halo APU with 128GB of unified memory, configured with 96GB allocated to VRAM, trying to answer a simple question: can you actually run models that don't fit in VRAM, and if so, how much performance do you sacrifice?

The answer turns out to be: yes, you can, and the performance penalty is more nuanced than I expected.

The Memory Problem

Large language models are memory hogs. A 7B parameter model in bfloat16 needs roughly 14GB just for the weights. A 70B model needs 140GB. An 80B MoE model might need 160GB or more. Most consumer GPUs max out at 24GB, with only a handful of prosumer cards reaching 48GB.

The traditional solutions each have trade-offs:

Quantization reduces memory requirements by storing weights in lower precision formats. INT8 cuts memory in half. INT4 cuts it to a quarter. But quantization also reduces quality, sometimes significantly for complex reasoning tasks.

Model sharding across multiple GPUs works if you have multiple GPUs. Most people don't. Early on (e.g. two years ago), this is how I experimented with models, a handful of Pascal chipset NVIDIA GPUs in a former crypto mining server.

Cloud inference works but adds latency, costs money per token, and means your data leaves your machine.

Partial loading offers a fourth path: keep the model weights somewhere other than VRAM (CPU RAM, disk, NVMe) and load them into the GPU only when needed. You take a latency hit on every layer that needs to be fetched, but you can run models that would otherwise be impossible.

Understanding Transformer Layer Architecture

To understand why partial loading works, you need to understand how transformers process information. A typical LLM consists of:

Embedding layer: Converts input tokens to vectors. Relatively small.
Decoder layers: The bulk of the model. A 70B parameter model might have 80+ decoder layers, each containing attention heads and a feed-forward network (FFN).
Final layer norm and output projection: Converts the final hidden states back to token probabilities. Relatively small.

The key insight is that inference is sequential through the layers. When processing a token, you go through layer 0, then layer 1, then layer 2, and so on. You never need layer 5 while you're processing layer 3. This means you can theoretically keep only one layer's weights in VRAM at a time, loading the next layer while processing the current one.

In practice, keeping all layers streaming from disk adds too much latency. The sweet spot is typically keeping some layers resident in VRAM (usually the first few and last few, which see the most traffic) while streaming the middle layers on demand.

The Hardware Setup

My test machine is an AMD Strix Halo APU:

Hardware Configuration:
- AMD Radeon 8060S (gfx1151)
- 128GB unified memory (96GB VRAM / 32GB system)
- ROCm 7.0 with HSA_OVERRIDE_GFX_VERSION=11.0.0
- PyTorch 2.9.1+rocm6.3
- NVMe SSD: Samsung 990 Pro 2TB (PCIe 4.0, 7450 MB/s read)

The unified memory architecture is interesting for this experiment. On a discrete GPU, moving weights from CPU RAM to VRAM requires crossing the PCIe bus, which tops out at around 64GB/s for PCIe 4.0 x16. On the Strix Halo APU, both "GPU memory" and "CPU memory" share the same physical RAM—it's just a question of which pages are mapped for GPU access.

This should give partial loading an advantage on APUs, since there's no physical data movement—just page table updates. The actual numbers bear this out, as we'll see.

Three Approaches to Partial Loading

I tested three different strategies for loading models that exceed VRAM:

1. llama.cpp with Partial GPU Offloading

The simplest approach uses llama.cpp's -ngl (number of GPU layers) flag. This lets you specify exactly how many transformer layers go on the GPU, with the rest staying on CPU.

./main -m models/llama-70b-chat.gguf -ngl 35 \
    -p "The capital of France is" -n 50

With a 70B model that has 80 layers, setting -ngl 35 puts roughly 44% of the model on the GPU and 56% on CPU. The GPU handles the compute-intensive matrix multiplications, while the CPU layers run on AMD's Zen cores.

Advantages:

Simple to configure
Automatic handling of which layers go where
Works with GGUF quantized models
CPU layers use optimized AVX-512 implementations

Disadvantages:

Static partitioning—layers stay where they're assigned
CPU inference is much slower than GPU
Limited to llama.cpp's supported architectures

2. HuggingFace Accelerate Disk Offloading

HuggingFace's Accelerate library provides device_map="auto" with disk offloading:

from transformers import AutoModelForCausalLM, AutoTokenizer
from accelerate import infer_auto_device_map

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-70B-Instruct",
    device_map="auto",
    offload_folder="./offload",
    torch_dtype=torch.bfloat16
)

When VRAM is insufficient, Accelerate automatically spills layers to disk (or CPU RAM if you use offload_buffers=True). During inference, layers are loaded as needed.

Advantages:

Works with any HuggingFace model
Automatic layer placement decisions
Can use disk for infinite capacity
Integrates with the broader HuggingFace ecosystem

Disadvantages:

Disk I/O is slow (even NVMe)
Layer loading happens synchronously
Each token generation can require full model traversal
Memory peaks during layer swaps

3. oLLM Layer Streaming

The most sophisticated approach I tested was oLLM, a library designed specifically for layer-by-layer streaming from SSD to GPU. Unlike HuggingFace's approach, oLLM implements asynchronous layer prefetching—while one layer is processing on the GPU, the next layer is being loaded.

from ollm import Inference

o = Inference("llama3-1B-chat", device="cuda:0")
o.ini_model(models_dir="./models/")

# Offload half the layers to CPU
o.offload_layers_to_cpu(layers_num=8)

# Generate
response = o.generate("The capital of France is", max_tokens=50)

The library instruments each layer load, giving you visibility into the streaming behavior:

layer_load: [0.002, 0.004, 0.004, 0.004, 0.004] t:0.242

This tells you each layer took about 4ms to load, and the total token generation time was 242ms.

Advantages:

Asynchronous prefetching reduces latency
Per-layer timing instrumentation
Designed specifically for memory-constrained scenarios
Can leverage GPU Direct Storage (GDS) for faster NVMe-to-GPU transfers

Disadvantages:

Limited model architecture support
Requires transformers 4.x (incompatible with 5.0)
Less mature than llama.cpp or HuggingFace

Benchmarking Partial Loading

I ran a series of tests with Llama 3.2 1B (16 layers, 2.8GB model size) to measure the impact of partial loading:

Test Configuration

model_id = "llama3-1B-chat"
prompt = "The capital of France is"
max_tokens = 50
configurations = [
    {"gpu_layers": 16, "cpu_layers": 0},   # Full GPU
    {"gpu_layers": 12, "cpu_layers": 4},   # 75% GPU
    {"gpu_layers": 8, "cpu_layers": 8},    # 50% GPU
    {"gpu_layers": 4, "cpu_layers": 12},   # 25% GPU
]

Results: oLLM Layer Streaming

With the oLLM library and 8 of 16 layers offloaded to CPU:

Configuration: 8 GPU layers, 8 CPU layers
Model loading: 2.3 seconds
First token latency: 242ms
Per-layer load time: ~4ms average
Output quality: Correct ("The capital of France is Paris.")

The layer load times are interesting. At 4ms per layer, you might expect significant overhead when 8 layers need to be fetched from CPU RAM. But because oLLM prefetches the next layer while the current one is executing, the effective latency impact is much smaller.

On a discrete GPU with PCIe transfers, these numbers would be different. Loading a 200MB layer across PCIe 4.0 x16 takes about 3ms at full bandwidth. But PCIe rarely achieves full bandwidth due to protocol overhead, so real-world numbers are typically 4-6ms per layer—similar to what I measured on the APU.

The Quality Question

A critical question with partial loading: does offloading layers affect output quality?

The answer is no, with an important caveat. Partial loading doesn't change the weights—it just changes where they're stored. The same matrices participate in the same computations. The outputs are bit-identical to full GPU inference.

The caveat is that some partial loading implementations use reduced precision for CPU layers (FP32 instead of bfloat16, or even FP16) to speed up CPU computation. This can introduce small numerical differences. In my testing with oLLM, both GPU and CPU layers used the same bfloat16 precision, so outputs matched exactly.

Practical Performance Analysis

Let's break down what partial loading actually costs in terms of latency.

Layer Loading Overhead

For a model with N layers, where K layers are on CPU: - Each token generation requires K layer loads - If each load takes T_load milliseconds - The total added latency per token is approximately K * T_load

With oLLM's prefetching, the effective latency is lower because loads overlap with computation. In my tests: - K = 8 layers on CPU - T_load = 4ms per layer - Naive overhead = 32ms per token - Actual overhead (with prefetching) = ~10-15ms per token

Memory Bandwidth Bottleneck

The real constraint isn't CPU speed—it's memory bandwidth. A single transformer layer in a 70B model might be 800MB-1.2GB. Loading this from:

NVMe SSD: 7.4GB/s = 108-162ms per layer
DDR5 RAM: 80GB/s = 10-15ms per layer
PCIe 4.0 x16: 32GB/s = 25-37ms per layer (in practice)

This is why oLLM's authors recommend fast NVMe SSDs (Samsung 990 Pro, WD SN850X) and ideally GPU Direct Storage, which bypasses the CPU entirely for disk-to-GPU transfers.

Token Generation Speed Comparison

For the Llama 3.2 1B model (16 layers total), I ran benchmarks across multiple prompts and averaging the results:

Configuration	Avg Tokens/sec	Avg Inference Time	Load Time
Full GPU (0 offloaded)	1.92 tok/s	13.90s	0.46s
4 layers on CPU	2.23 tok/s	11.09s	0.55s
8 layers on CPU	2.26 tok/s	10.87s	0.65s
12 layers on CPU	3.36 tok/s	7.30s	0.75s

Wait—that's backwards from what you'd expect. More layers on CPU resulted in faster inference?

This counterintuitive result makes sense when you consider the Strix Halo's unified memory architecture. Unlike a discrete GPU where CPU-to-GPU transfers cross the PCIe bus, the APU's "CPU memory" and "GPU memory" are the same physical RAM. Moving layers between them is essentially just a page table operation, not a data copy.

The performance improvement with more offloading likely comes from reduced memory bandwidth contention. When all layers are "on GPU," they're competing for the same memory channels. With layer streaming, only the active layer's weights occupy high-bandwidth GPU memory paths, while inactive layers sit in lower-priority memory regions.

This finding suggests that on unified memory systems (AMD APUs, Apple Silicon), partial loading might actually be preferable to full GPU loading for memory-bandwidth-bound workloads. The conventional wisdom—that GPU is always faster—doesn't hold when there's no physical separation between GPU and CPU memory.

Transformer Version Compatibility Issues

One challenge I encountered was library compatibility. oLLM was designed for transformers 4.x, and when I initially ran it with transformers 5.0, I hit several errors:

TypeError: 'Qwen3NextExperts' object is not iterable

This error occurred because transformers 5.0 changed how Mixture of Experts (MoE) layers expose their expert modules. The oLLM library's layer streaming code assumed it could iterate over self.mlp.experts, but the new implementation uses a different structure.

There were also weight shape mismatches:

model.layers.0.input_layernorm.weight: found shape torch.Size([2048])
in the checkpoint and torch.Size([0]) in the model instantiated

This happened because oLLM creates placeholder layers with zero-size tensors to save memory, then loads the actual weights on demand. The new transformers version changed how these placeholder shapes were inferred.

The solution was straightforward: pin transformers to version 4.57.6:

pip install 'transformers<5.0'

This is a common pattern with cutting-edge ML libraries. The ecosystem moves fast, and specialized tools often lag behind major version updates.

Storage and System Requirements

Before diving into partial loading, it's worth understanding the storage requirements. Unlike full GPU loading where you only need enough VRAM, partial loading requires sufficient storage capacity and bandwidth.

Disk Space Calculations

Model files on disk are typically stored in safetensors or GGUF format. A rough calculation: - 7B model (bfloat16): ~14GB - 13B model (bfloat16): ~26GB - 70B model (bfloat16): ~140GB - 70B model (GGUF Q4_K_M): ~40GB

For oLLM's layer streaming, you also need the model to be split into per-layer shards, which the library handles automatically during the first load. This adds temporary storage overhead during the conversion process.

RAM Requirements

CPU offloading means the offloaded layers live in system RAM. If you're offloading 40 of 80 layers from a 70B model, you need roughly 70GB of system RAM available—in addition to whatever the operating system and other applications need.

On my Strix Halo system with 128GB unified memory (96GB allocated to VRAM, 32GB to system), this gets interesting. The "CPU" portion of memory and the "GPU" portion share the same physical DIMMs. Allocating layers to "CPU" really just means they're in a different memory region that the GPU can still access, but through a different (slower) path.

SSD Endurance Considerations

If you're streaming weights from disk rather than CPU RAM, consider your SSD's endurance. A 70B model with 80 layers means moving roughly 1.75GB per token generated (all layers traversed once). Generate 1000 tokens and you've read 1.75TB from the SSD.

For occasional use, this is fine. For continuous operation (like a chatbot running 24/7), you might wear out a consumer SSD within months. Enterprise SSDs with higher TBW (Terabytes Written) ratings are worth considering for heavy use cases, or preferring CPU RAM offloading over disk offloading.

Memory Mapping and Page Tables

Under the hood, partial loading relies on the operating system's memory management. When a layer is "loaded" to the GPU, this typically involves:

Reading the layer from storage (if not already in RAM)
Pinning the memory pages so they can't be swapped
Mapping the pages into GPU-accessible memory space
Synchronizing to ensure the GPU sees the updated data

On Linux, this uses mmap() and mlock() syscalls. The vm.max_map_count sysctl may need to be increased for very large models:

# Check current value
cat /proc/sys/vm/max_map_count

# Increase if needed
sudo sysctl -w vm.max_map_count=1048576

I hit this limit when testing 70B+ models and saw cryptic "cannot allocate memory" errors until increasing the map count.

When Partial Loading Makes Sense

Based on my testing, here's when partial loading is a good fit:

Good Use Cases:

Batch processing where latency isn't critical (overnight analysis, embedding generation)
Interactive use with smaller models where the overhead is manageable
Running larger models occasionally without investing in more VRAM
Testing different model sizes before committing to hardware
APU systems where CPU-GPU transfer costs are minimal

Poor Use Cases:

Real-time applications (chatbots, live transcription)
High-throughput production systems
When quantization gives acceptable quality with lower overhead
Systems with slow storage (spinning disks, older SSDs)

The break-even point depends heavily on your specific hardware. On my APU system with unified memory, offloading 50% of layers costs about 33% of throughput. On a discrete GPU with PCIe 3.0, the same configuration might cost 60-70%.

Future Directions

Several developments could make partial loading more practical:

GPU Direct Storage (GDS): NVIDIA's GDS and AMD's equivalent allow direct SSD-to-GPU transfers, bypassing the CPU and PCIe. Early implementations show 3-4x improvements in layer load times.

Better Prefetching Algorithms: Current implementations use simple next-layer prefetching. More sophisticated approaches could predict multiple layers ahead, or prioritize layers that are accessed most frequently (relevant for some architectures with skip connections).

Hardware Evolution: Unified memory architectures like Apple Silicon and AMD APUs eliminate the CPU-GPU transfer bottleneck entirely. As these architectures gain more memory capacity, partial loading becomes increasingly attractive.

Compression: Applying neural compression to stored weights (not quantization, but actual neural codecs) could reduce the bandwidth requirements by 2-4x without quality loss.

Building a Benchmark Framework

For those who want to measure partial loading on their own hardware, here's the framework I developed:

from dataclasses import dataclass
from typing import List, Optional
import time
import torch

@dataclass
class BenchmarkResult:
    model_name: str
    total_layers: int
    gpu_layers: int
    cpu_layers: int
    loading_mode: str  # 'partial' or 'full'
    load_time_seconds: float
    tokens_generated: int
    total_inference_time: float
    tokens_per_second: float
    per_layer_load_times: Optional[List[float]] = None
    peak_vram_gb: Optional[float] = None

def benchmark_ollm_inference(
    model_id: str,
    offload_layers: int,
    prompt: str,
    max_tokens: int = 50
) -> BenchmarkResult:
    from ollm import Inference

    # Measure loading time
    load_start = time.perf_counter()
    o = Inference(model_id, device="cuda:0", logging=False)
    o.ini_model(models_dir="./models/", force_download=False)

    total_layers = len(o.model.model.layers)

    if offload_layers > 0:
        o.offload_layers_to_cpu(layers_num=offload_layers)

    load_time = time.perf_counter() - load_start

    # Measure inference time
    torch.cuda.synchronize()
    infer_start = time.perf_counter()

    output = o.generate(prompt, max_tokens=max_tokens)

    torch.cuda.synchronize()
    infer_time = time.perf_counter() - infer_start

    # Count tokens
    tokens = len(o.tokenizer.encode(output))

    return BenchmarkResult(
        model_name=model_id,
        total_layers=total_layers,
        gpu_layers=total_layers - offload_layers,
        cpu_layers=offload_layers,
        loading_mode='partial' if offload_layers > 0 else 'full',
        load_time_seconds=load_time,
        tokens_generated=tokens,
        total_inference_time=infer_time,
        tokens_per_second=tokens / infer_time if infer_time > 0 else 0,
        peak_vram_gb=torch.cuda.max_memory_allocated() / 1e9
    )

This framework measures the key metrics: loading time, inference time, tokens per second, and VRAM usage. Run it with different offload_layers values to map out the performance curve for your specific hardware.

Conclusion

Partial LLM loading isn't a silver bullet, but it's a valuable technique for expanding what's possible on memory-constrained hardware. On my 128GB APU system, I found something unexpected: partial loading with 12 of 16 layers on CPU actually outperformed full GPU loading by 75% (3.36 tok/s vs 1.92 tok/s).

The key takeaways:

Unified memory changes everything. On APUs and Apple Silicon, the conventional wisdom that "GPU is always faster" doesn't hold. Reduced memory bandwidth contention can make partial loading preferable even when you have enough VRAM.
Prefetching is essential. Naive layer loading is too slow. Libraries like oLLM that prefetch the next layer during current layer computation can reduce overhead by 50% or more.
Memory bandwidth matters more than CPU speed. The bottleneck is getting bytes from storage/RAM to the GPU, not processing them once they're there.
Library maturity varies. Expect compatibility issues with newer transformers versions. Pin your dependencies—oLLM requires transformers 4.x.
Quality is preserved. Partial loading changes where weights live, not what they are. Outputs match full GPU inference exactly (assuming matching precision).
Benchmark your specific hardware. My results on a Strix Halo APU won't match discrete GPU performance. The only way to know what works best is to measure it.

For batch processing and experimentation, partial loading lets you access models that would otherwise require more expensive hardware. For unified memory systems specifically, partial loading might be the optimal configuration, not just a fallback.

The era of "if it doesn't fit in VRAM, you can't run it" is ending. With the right techniques, nearly any model becomes accessible—and on the right hardware, you might even get a performance bonus for your trouble.

Running Qwen TTS on AMD Strix Halo: A Complete Guide to Local Text-to-Speech

A.C. Jokela — Sat, 24 Jan 2026 18:00:00 GMT

The rise of high-quality text-to-speech models has opened new possibilities for content creators, accessibility advocates, and developers alike. Qwen3-TTS, developed by Alibaba's Qwen team, represents a significant leap forward in neural TTS technology, offering natural-sounding speech synthesis with multiple speaker voices. In this guide, we'll walk through setting up Qwen3-TTS on AMD's Strix Halo platform—specifically the AI Max+ 395 with its integrated Radeon 8060S graphics—and demonstrate how we use it to generate audio narrations for blog posts right here on TinyComputers.

Why Qwen3-TTS?

The text-to-speech landscape has evolved dramatically over the past few years. While cloud-based services like Amazon Polly, Google Cloud TTS, and ElevenLabs offer impressive quality, they come with ongoing costs, privacy considerations, and internet dependency. Local TTS solutions have historically lagged behind in quality, often producing robotic or unnatural speech.

Qwen3-TTS changes this equation. The model produces remarkably natural speech with proper intonation, pacing, and emphasis. It supports multiple pre-trained speaker voices—including options like Eric, Aiden, Dylan, Serena, and others—each with distinct characteristics suitable for different content types. For technical content like our blog posts, the Eric voice provides clear, professional narration that listeners find easy to follow.

The model we're using, Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice, weighs in at 1.7 billion parameters. While not small, this is manageable on modern hardware and runs efficiently on GPU. The 12Hz designation refers to the audio frame rate used during generation, balancing quality with computational requirements.

The Hardware: AMD AI Max+ 395

AMD's Strix Halo architecture represents their latest push into the high-performance APU market, combining powerful CPU cores with substantial integrated graphics. Our test system features:

CPU: AMD Ryzen AI Max+ 395 with 16 Zen 5 cores (32 threads)
GPU: Integrated Radeon 8060S (RDNA 3.5 architecture)
Memory: 128GB unified DDR5, configured with 96GB VRAM and 32GB system RAM
Compute Units: 40 CUs dedicated to graphics/compute workloads

Our test system is the Bosgame M5 AI Mini Desktop, one of the first mini PCs to ship with AMD's Strix Halo silicon. The GMKtec EVO-X2 is an extremely similar system if you're looking to replicate this setup. The unified memory architecture is particularly relevant for machine learning workloads. Unlike discrete GPUs with their own VRAM, the Radeon 8060S shares system memory with the CPU. This means no PCIe bottleneck for data transfers, and with 96GB allocated as VRAM, even large models fit comfortably.

For our TTS workload, the 8060S provides adequate performance. The 1.7B parameter model fits comfortably in memory, and inference runs entirely on GPU once loaded. We see 100% GPU utilization during speech synthesis, indicating the hardware is being fully leveraged.

Setting Up the Environment

The first challenge with AMD GPUs is getting PyTorch working correctly with ROCm, AMD's open-source GPU compute stack. The Strix Halo uses a newer GPU architecture (gfx1151) that requires ROCm 6.x and some environment variable overrides.

Step 1: Create a Python Virtual Environment

We'll use a dedicated virtual environment to isolate our TTS dependencies:

mkdir -p ~/qwen-tts
cd ~/qwen-tts
python3 -m venv venv
source venv/bin/activate

Step 2: Install PyTorch with ROCm Support

The standard PyTorch installation won't work—we need the ROCm-enabled build. As of this writing, ROCm 6.4 is the latest stable release:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.4

This downloads PyTorch builds compiled specifically for AMD GPUs. The installation is larger than the standard CUDA builds due to the different compute libraries involved.

Step 3: Install Qwen-TTS

With PyTorch in place, install the Qwen TTS package:

pip install qwen-tts soundfile numpy

The soundfile library handles WAV file I/O, while numpy is needed for audio array manipulation.

Step 4: Install xformers for ROCm (Optional but Recommended)

The xformers library provides optimized attention implementations that can improve performance:

pip install xformers --index-url https://download.pytorch.org/whl/rocm6.4

While Qwen-TTS will work without xformers, having it available enables more efficient memory-attention patterns during inference.

Step 5: Configure Environment Variables

The Strix Halo's gfx1151 architecture isn't explicitly recognized by all ROCm components yet. We need to tell the system to treat it as a compatible architecture:

export HSA_OVERRIDE_GFX_VERSION=11.0.0
export GPU_MAX_ALLOC_PERCENT=100
export GPU_MAX_HEAP_SIZE=100
export TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1

Let's break down what these do:

HSA_OVERRIDE_GFX_VERSION=11.0.0: Tells the HSA runtime to report the GPU as gfx1100, which has broader library support
GPU_MAX_ALLOC_PERCENT=100: Allows the GPU to use up to 100% of available memory for allocations
GPU_MAX_HEAP_SIZE=100: Similar memory allocation setting for heap operations
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1: Enables experimental efficient attention implementations for AMD GPUs

Add these to your .bashrc or create an activation script for convenience.

Step 6: Verify GPU Detection

Before proceeding, confirm PyTorch can see your GPU:

import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"Device count: {torch.cuda.device_count()}")
print(f"Device name: {torch.cuda.get_device_name(0)}")

You should see output like:

CUDA available: True
Device count: 1
Device name: AMD Radeon 8060S

Note that PyTorch uses "CUDA" terminology even for AMD GPUs when using ROCm—this is for API compatibility.

Basic TTS Usage

With the environment configured, let's test basic speech synthesis:

from qwen_tts import Qwen3TTSModel
import soundfile as sf
import torch

# Load model on GPU with bfloat16 precision
model = Qwen3TTSModel.from_pretrained(
    'Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice',
    attn_implementation='sdpa',
    device_map='cuda:0',
    dtype=torch.bfloat16
)

# Check available speakers
print(f"Available speakers: {model.get_supported_speakers()}")

# Generate speech
text = "Hello, and welcome to TinyComputers. Today we're exploring text-to-speech on AMD hardware."
audios, sample_rate = model.generate_custom_voice(
    text,
    speaker='eric',
    language='english'
)

# Save to file
sf.write('output.wav', audios[0], sample_rate)
print(f"Saved audio at {sample_rate}Hz")

A few important notes:

We use attn_implementation='sdpa' for scaled dot-product attention, which works on ROCm
The device_map='cuda:0' explicitly places the model on the GPU
Using dtype=torch.bfloat16 reduces memory usage while maintaining quality
The language parameter must be the full word 'english', not the abbreviation 'en'

Building a Blog-to-Speech Pipeline

For our use case—generating audio versions of blog posts—we need more than basic TTS. Blog posts contain markdown formatting, code blocks, images, and other elements that shouldn't be read aloud. We built a complete pipeline that handles these challenges.

The Blog Cleaner

Our cleaning process strips out non-spoken content while preserving the narrative flow:

import re

def clean_markdown(content):
    # Remove YAML frontmatter
    if content.startswith('---'):
        end = content.find('---', 3)
        if end != -1:
            content = content[end + 3:]

    # Strip HTML tags (audio, video, images)
    content = re.sub(r'<audio[^>]*>[\s\S]*?</audio>', '', content, flags=re.IGNORECASE)
    content = re.sub(r'<video[^>]*>[\s\S]*?</video>', '', content, flags=re.IGNORECASE)
    content = re.sub(r'<img[^>]*/?>', '', content, flags=re.IGNORECASE)
    content = re.sub(r'<[^>]+>', '', content)

    # Remove markdown images and convert links to just text
    content = re.sub(r'!\[[^\]]*\]\([^)]+\)', '', content)
    content = re.sub(r'\[([^\]]+)\]\([^)]+\)', r'\1', content)

    # Remove code blocks
    content = re.sub(r'```[\s\S]*?```', '', content)
    content = re.sub(r'`[^`]+`', '', content)

    # Convert headers to sentences
    content = re.sub(r'^(#{1,6})\s+(.+)$', r'\2.', content, flags=re.MULTILINE)

    # Remove emphasis markers
    content = re.sub(r'\*\*([^*]+)\*\*', r'\1', content)
    content = re.sub(r'\*([^*]+)\*', r'\1', content)

    return content

Unit Conversion for Speech

Technical content often includes abbreviations that sound awkward when read literally. We convert common units to their spoken forms:

def convert_units_for_speech(text):
    text = re.sub(r'(\d+)\s*GB\b', r'\1 gigabytes', text)
    text = re.sub(r'(\d+)\s*MB\b', r'\1 megabytes', text)
    text = re.sub(r'(\d+)\s*GHz\b', r'\1 gigahertz', text)
    text = re.sub(r'(\d+)\s*MHz\b', r'\1 megahertz', text)
    text = re.sub(r'(\d+)\s*KB\b', r'\1 kilobytes', text)
    return text

Chunking Long Content

TTS models work best with moderate-length inputs. Very long passages can cause quality degradation or memory issues. We split content into chunks at sentence boundaries:

def chunk_text(text, max_chars=500):
    chunks = []
    sentences = re.split(r'(?<=[.!?])\s+', text)
    current = ""

    for sentence in sentences:
        if len(current) + len(sentence) < max_chars:
            current += sentence + " "
        else:
            if current:
                chunks.append(current.strip())
            current = sentence + " "

    if current:
        chunks.append(current.strip())

    return chunks

The Complete Script

Putting it all together, here's our blog_to_speech.py script:

#!/usr/bin/env python3
import argparse
import re
import torch
from pathlib import Path
from qwen_tts import Qwen3TTSModel
import soundfile as sf
import numpy as np

def clean_blog_post(source):
    content = Path(source).read_text()
    # Apply cleaning functions...
    return cleaned_text

def synthesize_speech(text, output_file, speaker="eric"):
    model = Qwen3TTSModel.from_pretrained(
        'Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice',
        attn_implementation='sdpa',
        device_map='cuda:0',
        dtype=torch.bfloat16
    )

    chunks = chunk_text(text)
    all_audio = []

    for i, chunk in enumerate(chunks):
        print(f"Processing chunk {i+1}/{len(chunks)}")
        audios, sample_rate = model.generate_custom_voice(
            chunk,
            speaker=speaker,
            language='english'
        )
        all_audio.append(audios[0])

    combined = np.concatenate(all_audio)
    sf.write(output_file, combined, sample_rate)
    duration = len(combined) / sample_rate
    print(f"Saved {duration:.1f}s audio to: {output_file}")

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('source', help='Blog post markdown file')
    parser.add_argument('-o', '--output', default='output.wav')
    parser.add_argument('--speaker', default='eric')
    args = parser.parse_args()

    text = clean_blog_post(args.source)
    synthesize_speech(text, args.output, args.speaker)

Choosing the Right Speaker Voice

Qwen3-TTS ships with nine pre-trained speaker voices, each with distinct characteristics:

Speaker	Characteristics	Best For
Eric	Clear, professional male voice with measured pacing	Technical content, tutorials, documentation
Aiden	Younger male voice, slightly more casual	Blog posts, conversational content
Dylan	Deeper male voice with authoritative tone	Formal presentations, announcements
Ryan	Energetic male voice	Marketing content, product demos
Serena	Clear female voice, professional	Corporate content, tutorials
Vivian	Warm female voice	Storytelling, narrative content
Ono Anna	Female voice with distinct character	Creative content
Sohee	Female voice, versatile	General purpose
Uncle Fu	Character voice	Specialized applications

For our technical blog content, we primarily use Eric. His clear enunciation and measured pacing work well for complex technical explanations. The voice handles acronyms, numbers, and technical terminology naturally, making it ideal for content about hardware, programming, and system administration.

You can easily switch voices by changing the speaker parameter:

audios, sample_rate = model.generate_custom_voice(
    text,
    speaker='serena',  # Try different voices
    language='english'
)

Consider matching voice characteristics to content type. A hardware review might work better with Eric's authoritative tone, while a personal essay might benefit from Aiden's more conversational style.

Comparing TTS Options

Before settling on Qwen3-TTS, we evaluated several alternatives. Here's how they compare for our use case:

Cloud Services

Amazon Polly and Google Cloud TTS offer excellent quality with minimal setup. However, costs accumulate quickly for long-form content. At roughly \$4-16 per million characters (depending on voice quality), a 3000-word blog post costs \$0.10-0.40 per generation. For a site with dozens of posts requiring periodic regeneration, this adds up.

ElevenLabs produces arguably the most natural voices available, with impressive emotional range. But their pricing model—based on character quotas—makes it expensive for regular content generation. The quality is exceptional, but overkill for straightforward narration.

Local Alternatives

Coqui TTS (now deprecated) was a popular open-source option but development has stalled. Bark from Suno produces impressive results but runs slowly and lacks fine-grained control. XTTS offers voice cloning but requires more setup and compute resources.

Piper deserves special mention as a lightweight option. It runs quickly even on CPU and produces acceptable quality for many applications. However, the voices sound noticeably synthetic compared to Qwen3-TTS—fine for notifications or short snippets, but fatiguing for 30-minute narrations.

Qwen3-TTS hits a sweet spot: quality approaching cloud services, reasonable compute requirements, and fully local operation. The 1.7B parameter model is large enough for natural prosody but small enough to run on consumer hardware.

Batch Processing for Multiple Posts

When generating audio for multiple blog posts, efficiency matters. Loading the model takes 15-30 seconds, so we keep it loaded while processing multiple files:

#!/usr/bin/env python3
"""Batch TTS processing for multiple blog posts"""

import os
from pathlib import Path
from qwen_tts import Qwen3TTSModel
import torch
import soundfile as sf
import numpy as np

# Load model once
print("Loading model...")
model = Qwen3TTSModel.from_pretrained(
    'Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice',
    attn_implementation='sdpa',
    device_map='cuda:0',
    dtype=torch.bfloat16
)

posts = [
    'post1.md',
    'post2.md',
    'post3.md',
]

for post in posts:
    print(f"\n{'='*50}")
    print(f"Processing: {post}")
    print(f"{'='*50}")

    text = clean_blog_post(post)
    output = f"/tmp/{Path(post).stem}_tts.wav"

    # Process chunks
    chunks = chunk_text(text)
    all_audio = []

    for i, chunk in enumerate(chunks):
        print(f"  Chunk {i+1}/{len(chunks)}")
        audios, sr = model.generate_custom_voice(
            chunk, speaker='eric', language='english'
        )
        all_audio.append(audios[0])

    combined = np.concatenate(all_audio)
    sf.write(output, combined, sr)
    print(f"Saved: {output}")

This approach processes our five-post backlog overnight, with results ready for review in the morning.

Performance Characteristics

On the AI Max+ 395, speech synthesis runs at roughly real-time to 0.5x real-time speed—meaning a 30-minute audio file takes 30-60 minutes to generate. This is slower than high-end discrete GPUs but perfectly acceptable for batch processing.

For reference, here's how different content lengths performed in our testing:

Content	Characters	Chunks	Audio Duration	Generation Time
Short post	5,000	12	~5 min	~15 min
Medium post	15,000	35	~15 min	~45 min
Long post	25,000	55	~27 min	~90 min
Very long	40,000	85	~45 min	~150 min

The relationship between content length and generation time is roughly linear after the initial model warmup.

Some observations from our testing:

First chunk latency: The first chunk takes longer due to GPU kernel compilation and caching
Memory usage: Peak usage around 8-10GB during inference
GPU utilization: Consistent 100% during active synthesis
Quality: Indistinguishable from cloud TTS services for most content

The MIOpen library sometimes logs workspace warnings during execution. These don't affect output quality and can be safely ignored:

MIOpen(HIP): Warning [IsEnoughWorkspace] Solver <GemmFwdRest>, workspace required: 103133184

Integrating Audio into Blog Posts

Once we have the WAV file, we convert to MP3 for web delivery and embed an HTML5 audio player:

ffmpeg -i blog_post.wav -codec:a libmp3lame -qscale:a 2 blog_post.mp3

For reviewing TTS output quality, we recommend using studio monitor headphones that reveal any artifacts or unnatural tones in the generated speech.

The player HTML is straightforward:

<div style="background: #f8f9fa; border: 1px solid #e9ecef;
            border-radius: 8px; padding: 16px 20px; margin: 20px 0;">
  <div class="audio-widget-header">
    <span class="audio-widget-icon">🎧</span>
    <span style="color: #495057; font-weight: 600;">Listen to this article</span>
  </div>
  <audio controls preload="metadata" style="width: 100%;">
    <source src="/audio/blog_post.mp3" type="audio/mpeg">
  </audio>
  <div class="audio-widget-footer">
    27 min · AI-generated narration
  </div>
</div>

Why We're Doing This

Adding audio narration to blog posts serves multiple purposes:

Accessibility: Readers with visual impairments or reading difficulties can consume content aurally
Convenience: Listeners can enjoy posts during commutes, workouts, or other activities
Engagement: Audio content creates a more personal connection with the audience
Reach: Some audiences prefer audio format, expanding our potential readership

Running TTS locally rather than using cloud services gives us:

Cost control: No per-character or per-minute fees
Privacy: Content never leaves our infrastructure
Consistency: Same voice and quality across all posts
Flexibility: Full control over processing pipeline

Troubleshooting Common Issues

"CUDA not available" despite GPU present

Ensure you've installed the ROCm version of PyTorch, not the standard build:

pip uninstall torch torchvision torchaudio
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.4

Model runs on CPU instead of GPU

Check that device_map='cuda:0' is specified when loading the model. Also verify the environment variables are set before starting Python.

"Unsupported language 'en'"

Use the full language name: language='english' not language='en'.

Out of memory errors

Try reducing chunk size or using a smaller batch. The model should fit in 16GB, but very long chunks can spike memory usage.

Slow first chunk

This is normal—ROCm compiles GPU kernels on first use. Subsequent chunks process faster.

Future Improvements

Our current pipeline works well but has room for enhancement. Some improvements we're considering:

Voice cloning: Qwen3-TTS supports custom voice training. With sufficient audio samples, we could create a unique voice for TinyComputers rather than using the stock speakers. This would provide brand consistency and differentiation.

Automatic post detection: Currently we manually select posts for TTS generation. A CI/CD integration could automatically generate audio for new posts when they're published, keeping the audio library current without manual intervention.

Chapter markers: For longer posts, embedding chapter markers in the audio file would allow listeners to skip to specific sections. This requires parsing the markdown headers and mapping them to audio timestamps.

Multiple format export: Beyond MP3, offering Opus or AAC formats could reduce file sizes while maintaining quality, benefiting listeners on metered connections.

Speed adjustment: Some listeners prefer 1.25x or 1.5x playback speed. Pre-generating speed-adjusted versions could provide better quality than real-time speed adjustment in the browser.

Conclusion

Running Qwen3-TTS on AMD's Strix Halo platform demonstrates that high-quality local TTS is now accessible beyond NVIDIA hardware. While setup requires some ROCm-specific configuration, the results are impressive—natural-sounding narration suitable for professional content.

The democratization of AI capabilities continues apace. What once required expensive cloud subscriptions or high-end NVIDIA GPUs now runs on integrated graphics. The AI Max+ 395's Radeon 8060S, primarily designed for gaming and general compute tasks, handles a 1.7-billion parameter language model without breaking a sweat.

We're actively using this pipeline to generate audio versions of posts across TinyComputers, making our technical content more accessible and convenient for our readers. As of this writing, we've processed our retrocomputing series, hardware reviews, and technical tutorials—dozens of hours of content generated entirely on local hardware.

The combination of AMD's capable integrated graphics and Qwen's excellent TTS model proves that you don't need expensive discrete GPUs or cloud subscriptions to achieve broadcast-quality speech synthesis. For content creators, educators, and accessibility advocates, this opens new possibilities for enriching written content with audio without ongoing service costs.

If you're running AMD hardware and want to add audio narration to your own content, this guide should get you started. The initial setup investment pays dividends in ongoing cost savings and the satisfaction of running capable AI models entirely on your own infrastructure. And if you encounter issues along the way, the troubleshooting section above addresses the most common pitfalls we discovered during our own setup process.

The audio player at the top of many TinyComputers posts now represents a small but meaningful step toward making technical content more accessible. Every post you can listen to while commuting, exercising, or doing dishes is content that might otherwise go unread. That's the real value of local TTS—not just cost savings, but expanded reach for the ideas we share.

A Bespoke LLM Code Scanner

A.C. Jokela — Wed, 26 Nov 2025 16:49:15 GMT

Building a Nightly AI Code Scanner with vLLM, ROCm, and JIRA Integration

I've been running a ballistics calculation engine — a Rust physics library with several components, like a Flask app wrapper with machine learning capabilities, bindings for a python library as well as a Ruby gem library. There are also Android and iOS apps, too. The codebase has grown to about 15,000 lines of Rust and another 10,000 lines of Python. At this scale, bugs hide in edge cases: division by zero, floating-point precision issues in transonic drag calculations, unwrap() panics on unexpected input.

What if I could run an AI code reviewer every night while I sleep? Not a cloud API with per-token billing that could run up a $500 bill scanning 50 files, but a local model running on my own hardware, grinding through the codebase and filing JIRA tickets for anything suspicious.

This is the story of building that system.

The Hardware: AMD Strix Halo on ROCm 7.0

I'm running this on a server with an AMD Radeon 8060S (Strix Halo APU) — specifically the gfx1151 architecture. This isn't a data center GPU. It's essentially an integrated GPU with 128GB of shared memory; configured to give 96GB to VRAM and the rest to system RAM. Not the 80GB of HBM3 you'd get on an H100, but enough to run a 32B parameter model comfortably.

The key insight: for batch processing where latency doesn't matter, you don't need bleeding-edge hardware. A nightly scan can take hours. I'm not serving production traffic; I'm analyzing code files one at a time with a 30-second cooldown between requests. The APU handles this fine.

Hardware Configuration:
- AMD Radeon 8060S (gfx1151 Strix Halo APU)
- 96GB shared memory
- ROCm 7.0 with HSA_OVERRIDE_GFX_VERSION=11.5.1

The HSA_OVERRIDE_GFX_VERSION environment variable is critical. Without it, ROCm doesn't recognize the Strix Halo architecture. This is the kind of sharp edge you hit running ML on AMD consumer hardware.

Model Selection: Qwen2.5-Coder-7B-Instruct

I tested several models:

Model	Parameters	Context	Quality	Notes
DeepSeek-Coder-V2-Lite	16B	32k	Good	Requires flash_attn (ROCm issues)
Qwen3-Coder-30B	30B	32k	Excellent	Too slow on APU
Qwen2.5-Coder-7B-Instruct	7B	16k	Good	Sweet spot
TinyLlama-1.1B	1.1B	4k	Poor	Too small for code review

Qwen2.5-Coder-7B-Instruct hits the sweet spot. It understands Rust and Python well enough to spot real issues, runs fast enough to process 50 files per night, and doesn't require flash attention (which has ROCm compatibility issues on consumer hardware).

vLLM Setup

vLLM provides an OpenAI-compatible API server that makes integration trivial. Here's the startup command:

source ~/vllm-rocm7-venv/bin/activate
export HSA_OVERRIDE_GFX_VERSION=11.5.1
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-Coder-7B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --trust-remote-code \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.85

The --max-model-len 16384 limits context to 16k tokens. My code files rarely exceed 500 lines (truncated), so this is plenty. The --gpu-memory-utilization 0.85 leaves headroom for the system.

I run this in a Python venv rather than Docker because ROCm device passthrough with Docker on Strix Halo is finicky. Sometimes you have to choose pragmatism over elegance.

Docker Configuration (When It Works)

For reference, here's the Docker Compose configuration I initially built. It works on dedicated AMD GPUs but has issues on integrated APUs:

services:
  vllm:
    image: rocm/vllm-dev:latest
    container_name: vllm-code-scanner
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    group_add:
      - video
      - render
    security_opt:
      - seccomp:unconfined
    cap_add:
      - SYS_PTRACE
    ipc: host
    environment:
      - HSA_OVERRIDE_GFX_VERSION=11.5.1
      - PYTORCH_ROCM_ARCH=gfx1151
      - HIP_VISIBLE_DEVICES=0
    volumes:
      - /home/alex/models:/models
      - /home/alex/.cache/huggingface:/root/.cache/huggingface
    ports:
      - "8000:8000"
    command: >
      python -m vllm.entrypoints.openai.api_server
      --model Qwen/Qwen2.5-Coder-7B-Instruct
      --host 0.0.0.0
      --port 8000
      --trust-remote-code
      --max-model-len 16384
      --gpu-memory-utilization 0.85
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 120s

  scanner:
    build: .
    container_name: code-scanner-agent
    depends_on:
      vllm:
        condition: service_healthy
    environment:
      - VLLM_HOST=vllm
      - VLLM_PORT=8000
      - JIRA_EMAIL=${JIRA_EMAIL}
      - JIRA_API_KEY=${JIRA_API_KEY}
    volumes:
      - /home/alex/projects:/projects:ro
      - ./config:/app/config:ro
      - /home/alex/projects/code-scanner-results:/app/results

The ipc: host and seccomp:unconfined are necessary for ROCm to function properly. The depends_on with service_healthy ensures the scanner waits for vLLM to be fully loaded before starting — important since model loading can take 2-3 minutes.

The scanner Dockerfile is minimal:

FROM python:3.11-slim

WORKDIR /app

RUN apt-get update && apt-get install -y \
    git curl ripgrep \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY agent/ /app/agent/
COPY prompts/ /app/prompts/
COPY config/ /app/config/

CMD ["python", "-m", "agent.scanner"]

Including ripgrep in the container enables fast pattern matching when the scanner needs to search for related code.

The Scanner Architecture

The system has three main components:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Systemd       │     │    vLLM         │     │     JIRA        │
│   Timer         │────▶│    Server       │────▶│     API         │
│   (11pm daily)  │     │  (Qwen 7B)      │     │   (tickets)     │
└─────────────────┘     └─────────────────┘     └─────────────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │   Scanner Agent     │
                    │ - File discovery    │
                    │ - Code analysis     │
                    │ - Finding validation│
                    │ - JIRA integration  │
                    └─────────────────────┘

Configuration

Everything is driven by a YAML configuration file:

vllm:
  host: "10.1.1.27"
  port: 8000
  model: "Qwen/Qwen2.5-Coder-7B-Instruct"

schedule:
  start_hour: 23  # 11pm
  end_hour: 6     # 6am
  max_iterations: 50
  cooldown_seconds: 30

repositories:
  - name: "ballistics-engine"
    path: "/home/alex/projects/ballistics-engine"
    languages: ["rust"]
    scan_patterns:
      - "src//*.rs"
    exclude_patterns:
      - "target/"
      - "*.lock"

  - name: "ballistics-api"
    path: "/home/alex/projects/ballistics-api"
    languages: ["python", "rust"]
    scan_patterns:
      - "ballistics//*.py"
      - "ballistics_rust/src//*.rs"
    exclude_patterns:
      - "__pycache__/"
      - "target/"
      - ".venv/"

jira:
  enabled: true
  project_key: "MBA"
  confidence_threshold: 0.75
  labels: ["ai-detected", "code-scanner"]
  max_tickets_per_run: 10
  review_threshold: 5

The confidence_threshold: 0.75 is crucial. Without it, the model reports every minor style issue. At 75%, it focuses on things it's genuinely concerned about.

The review_threshold: 5 triggers a different behavior: if the model finds more than 5 issues, it creates a single summary ticket for manual review rather than flooding JIRA with individual tickets. This is a safety valve for when the model goes haywire.

Structured Outputs with Pydantic

LLMs are great at finding issues but terrible at formatting output consistently. Left to their own devices, they'll return findings as markdown, prose, JSON with missing fields, or creative combinations thereof.

The solution is structured outputs. I define Pydantic models for exactly what I expect:

class Severity(str, Enum):
    CRITICAL = "critical"
    HIGH = "high"
    MEDIUM = "medium"
    LOW = "low"
    INFO = "info"

class FindingType(str, Enum):
    BUG = "bug"
    PERFORMANCE = "performance"
    SECURITY = "security"
    CODE_QUALITY = "code_quality"
    POTENTIAL_ISSUE = "potential_issue"

class CodeFinding(BaseModel):
    file_path: str = Field(description="Path to the file")
    line_start: int = Field(description="Starting line number")
    line_end: Optional[int] = Field(default=None)
    finding_type: FindingType
    severity: Severity
    title: str = Field(max_length=100)
    description: str
    suggestion: Optional[str] = None
    confidence: float = Field(ge=0.0, le=1.0)
    code_snippet: Optional[str] = None

The confidence field is a float between 0 and 1. The model learns to be honest about uncertainty — "I think this might be a bug (0.6)" versus "This is definitely division by zero (0.95)."

In a perfect world, I'd use vLLM's Outlines integration for guided JSON generation. In practice, I found that prompting Qwen for JSON and parsing the response works reliably:

def _analyze_code(self, file_path: str, content: str) -> List[CodeFinding]:
    messages = [
        {"role": "system", "content": self.system_prompt},
        {"role": "user", "content": f"""Analyze this code for bugs and issues.

File: {file_path}

{content}

Return a JSON array of findings. Each finding must have:
- file_path: string
- line_start: number
- finding_type: "bug" | "performance" | "security" | "code_quality"
- severity: "critical" | "high" | "medium" | "low" | "info"
- title: string (max 100 chars)
- description: string
- suggestion: string or null
- confidence: number 0-1

If no issues found, return an empty array: []"""}
    ]

    response = self._call_llm(messages)

    # Parse JSON from response (handles markdown code blocks too)
    if response.strip().startswith('['):
        findings_data = json.loads(response)
    elif '```json' in response:
        json_str = response.split('```json')[1].split('```')[0]
        findings_data = json.loads(json_str)
    elif '[' in response:
        start = response.index('[')
        end = response.rindex(']') + 1
        findings_data = json.loads(response[start:end])
    else:
        return []

    # Validate each finding with Pydantic
    findings = []
    for item in findings_data:
        try:
            finding = CodeFinding(item)
            findings.append(finding)
        except ValidationError:
            pass  # Skip malformed findings

    return findings

The System Prompt

The system prompt is where you teach the model what you care about. Here's mine:

You are an expert code reviewer specializing in Rust and Python.
Your job is to find bugs, performance issues, security vulnerabilities,
and code quality problems.

You are analyzing code from a ballistics calculation project that includes:
- A Rust physics engine for trajectory calculations
- Python Flask API with ML models
- PyO3 bindings between Rust and Python

Key areas to focus on:
1. Numerical precision issues (floating point errors, rounding)
2. Edge cases in physics calculations (division by zero, negative values)
3. Memory safety in Rust code
4. Error handling (silent failures, unwrap panics)
5. Performance bottlenecks (unnecessary allocations, redundant calculations)
6. Security issues (input validation, injection vulnerabilities)

Be conservative with findings - only report issues you are confident about.
Avoid false positives.

The phrase "Be conservative with findings" is doing heavy lifting. Without it, the model reports everything that looks slightly unusual. With it, it focuses on actual problems.

Timeout Handling

Large files (500+ lines) can take a while to analyze. My initial 120-second timeout caused failures on complex files. I bumped it to 600 seconds (10 minutes):

response = requests.post(
    f"{self.base_url}/chat/completions",
    json=payload,
    headers={"Content-Type": "application/json"},
    timeout=600
)

I also truncate files to 300 lines. For longer files, the model only sees the first 300 lines. This is a trade-off — I might miss bugs in the back half of long files — but it keeps scans predictable and prevents timeout cascades. I plan to revisit this in future iterations.

lines = content.split('\n')
if len(lines) > 300:
    content = '\n'.join(lines[:300])
    logger.info("Truncated to 300 lines for analysis")

JIRA Integration

When the scanner finds issues, it creates JIRA tickets automatically. The API is straightforward:

def create_jira_tickets(self, findings: List[CodeFinding]):
    jira_base_url = f"https://{jira_domain}/rest/api/3"

    for finding in findings:
        # Map severity to JIRA priority
        priority_map = {
            Severity.CRITICAL: "Highest",
            Severity.HIGH: "High",
            Severity.MEDIUM: "Medium",
            Severity.LOW: "Low",
            Severity.INFO: "Lowest"
        }

        payload = {
            "fields": {
                "project": {"key": "MBA"},
                "summary": f"[AI] {finding.title}",
                "description": {
                    "type": "doc",
                    "version": 1,
                    "content": [{"type": "paragraph", "content": [
                        {"type": "text", "text": build_description(finding)}
                    ]}]
                },
                "issuetype": {"name": "Bug" if finding.finding_type == FindingType.BUG else "Task"},
                "priority": {"name": priority_map[finding.severity]},
                "labels": ["ai-detected", "code-scanner"]
            }
        }

        response = requests.post(
            f"{jira_base_url}/issue",
            json=payload,
            auth=(jira_email, jira_api_key),
            headers={"Content-Type": "application/json"}
        )

The [AI] prefix in the summary makes it obvious these tickets came from the scanner. The ai-detected label allows filtering.

I add a 2-second delay between ticket creation to avoid rate limiting:

time.sleep(2)  # Rate limit protection

Systemd Scheduling

The scanner runs nightly via systemd timer:

# /etc/systemd/system/code-scanner.timer
[Unit]
Description=Run Code Scanner nightly at 11pm

[Timer]
OnCalendar=*-*-* 23:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

The RandomizedDelaySec=300 adds up to 5 minutes of random delay. This prevents the scanner from always starting at exactly 11:00:00, which helps if multiple services share the same schedule.

The service unit is a oneshot that runs the scanner script:

# /etc/systemd/system/code-scanner.service
[Unit]
Description=Code Scanner Agent
After=docker.service

[Service]
Type=oneshot
User=alex
WorkingDirectory=/home/alex/projects/ballistics/code-scanner
ExecStart=/home/alex/projects/ballistics/code-scanner/scripts/start_scanner.sh
TimeoutStartSec=25200

The TimeoutStartSec=25200 (7 hours) gives the scanner enough time to complete even if it scans every file.

Sample Findings

Here's what the scanner actually finds. From a recent run:

{
  "file_path": "/home/alex/projects/ballistics-engine/src/fast_trajectory.rs",
  "line_start": 115,
  "finding_type": "bug",
  "severity": "high",
  "title": "Division by zero in fast_integrate when velocity approaches zero",
  "description": "The division dt / velocity_magnitude could result in division by zero if the projectile stalls (velocity_magnitude = 0). This can happen at the apex of a high-angle shot.",
  "suggestion": "Add a check for velocity_magnitude < epsilon before division, or clamp to a minimum value.",
  "confidence": 0.85
}

This is a real issue. In ballistics calculations, a projectile fired at a high angle momentarily has zero horizontal velocity at the apex. Without a guard, this causes a panic.

Not every finding is valid. The model occasionally flags intentional design decisions as "issues." But at a 75% confidence threshold, the false positive rate is manageable — maybe 1 in 10 findings needs to be closed as "not a bug."

Trade-offs and Lessons

What works well: - Finding numerical edge cases (division by zero, overflow) - Spotting unwrap() calls on Options that might be None - Identifying missing error handling - Flagging dead code and unreachable branches

What doesn't work as well: - Understanding business logic (the model doesn't know physics) - Spotting subtle race conditions in concurrent code - False positives on intentional patterns

Operational lessons: - Start with a low iteration limit (10-20 files) to test the pipeline - Monitor the first few runs manually before trusting it - Keep credentials in .env files excluded from rsync - The 300-line truncation is aggressive; consider chunking for long files

Handling JSON Parse Failures

Despite asking for JSON, LLMs sometimes produce malformed output. I see two failure modes:

Truncated JSON: The model runs out of tokens mid-response, leaving an unterminated string or missing closing brackets.
Wrapped JSON: The model adds explanatory text around the JSON, like "Here are the findings:" before the array.

My parser handles both:

def parse_findings_response(response: str) -> list:
    """Extract JSON from potentially messy LLM output."""
    response = response.strip()

    # Best case: raw JSON array
    if response.startswith('['):
        try:
            return json.loads(response)
        except json.JSONDecodeError:
            pass  # Fall through to extraction

    # Common case: JSON in markdown code block
    if '```json' in response:
        try:
            json_str = response.split('```json')[1].split('```')[0]
            return json.loads(json_str)
        except (IndexError, json.JSONDecodeError):
            pass

    # Fallback: extract JSON array from surrounding text
    if '[' in response and ']' in response:
        try:
            start = response.index('[')
            end = response.rindex(']') + 1
            return json.loads(response[start:end])
        except json.JSONDecodeError:
            pass

    # Give up
    logger.warning("Could not extract JSON from response")
    return []

When parsing fails, I log the error and skip that file rather than crashing the entire scan. In a typical 50-file run, I see 2-3 parse failures — annoying but acceptable.

Testing the Pipeline

Before trusting the scanner with JIRA ticket creation, I ran it in "dry run" mode:

# Set max iterations low and disable JIRA
export MAX_ITERATIONS=5
# In config: jira.enabled: false

python run_scanner_direct.py

This scans just 5 files and prints findings without creating tickets. I manually reviewed each finding:

True positive: Division by zero in trajectory calculation — good catch
False positive: Flagged intentional unwrap() on a guaranteed-Some Option — needs better context
True positive: Dead code path never executed — valid cleanup suggestion
Marginal: Style suggestion about variable naming — below my quality threshold

After tuning the confidence threshold and system prompt, the true positive rate improved to roughly 90%.

Monitoring and Observability

The scanner writes detailed logs to stdout and a JSON results file. Sample log output:

2025-11-26 15:48:25 - CODE SCANNER AGENT STARTING
2025-11-26 15:48:25 - Max iterations: 50
2025-11-26 15:48:25 - Model: Qwen/Qwen2.5-Coder-7B-Instruct
2025-11-26 15:48:25 - Starting scan of ballistics-engine
2025-11-26 15:48:25 - Found 35 files to scan
2025-11-26 15:48:25 - Scanning: src/trajectory_sampling.rs
2025-11-26 15:48:25 -   Truncated to 300 lines for analysis
2025-11-26 15:49:24 -   Found 5 findings (>= 75% confidence)
2025-11-26 15:49:24 -     [LOW] Redundant check for step_m value
2025-11-26 15:49:24 -     [LOW] Potential off-by-one error

The JSON results include full finding details:

{
  "timestamp": "20251126_151136",
  "total_findings": 12,
  "repositories": [
    {
      "repository": "ballistics-engine",
      "files_scanned": 35,
      "findings": [...],
      "duration_seconds": 1842.5,
      "iterations_used": 35
    }
  ]
}

I keep the last 30 result files (configurable) for historical comparison. Eventually I'll build a dashboard showing finding trends over time.

What's Next

The current system is batch-oriented: run once per night, file tickets, done. Future improvements I'm considering:

Pre-commit integration: Run on changed files only, fast enough for CI
Retrieval-augmented context: Include related files when analyzing (e.g., when scanning a function, include its callers)
Learning from feedback: Track which tickets get closed as "not a bug" and use that to tune prompts
Multi-model ensemble: Run the same code through two models, only file tickets when both agree

For now, though, the simple approach works. Every morning I check JIRA, triage the overnight findings, and fix the real bugs. The model isn't perfect, but it finds things I miss. And unlike a human reviewer, it never gets tired, never skips files, and never has a bad day.

Get the Code

I've open-sourced the complete scanner implementation on GitHub: llm-code-scanner

The project includes:

Dual scanning modes: Fast nightly scans via vLLM and comprehensive weekly analyses through Ollama
Smart deduplication: SQLite database prevents redundant issue tracking across runs
JIRA integration: Automatically creates tickets for findings above your confidence threshold
Email reports: SendGrid integration for daily/weekly summaries
Multi-language support: Python, Rust, TypeScript, Kotlin, Swift, Go, and more

To get started, clone the repo, configure your scanner_config.yaml with your vLLM/Ollama server details, and run python -m agent.scanner. The README has full setup instructions including environment variables for JIRA and SendGrid integration.

AMD AI Max+ 395 System Review: A Comprehensive Analysis

A.C. Jokela — Sun, 21 Sep 2025 20:25:28 GMT

Executive Summary

The AMD AI Max+ 395 system represents AMD's latest entry into the high-performance computing and AI acceleration market, featuring the company's cutting-edge Strix Halo architecture. This comprehensive review examines the system's performance characteristics, software compatibility, and overall viability for AI workloads and general computing tasks. While the hardware shows impressive potential with its 16-core CPU and integrated Radeon 8060S graphics, significant software ecosystem challenges, particularly with PyTorch/ROCm compatibility for the gfx1151 architecture, present substantial barriers to immediate adoption for AI development workflows.

Note: An Orange Pi 5 Max was photobombing this photograph

System Specifications and Architecture Overview

CPU Specifications

Processor: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
Architecture: x86_64 with Zen 5 cores
Cores/Threads: 16 cores / 32 threads
Base Clock: 599 MHz (minimum)
Boost Clock: 5,185 MHz (maximum)
Cache Configuration:
L1d Cache: 768 KiB (16 instances, 48 KiB per core)
L1i Cache: 512 KiB (16 instances, 32 KiB per core)
L2 Cache: 16 MiB (16 instances, 1 MiB per core)
L3 Cache: 64 MiB (2 instances, 32 MiB per CCX)
Instruction Set Extensions: Full AVX-512, AVX-VNNI, BF16 support

Memory Subsystem

Total System Memory: 32 GB DDR5
Memory Configuration: Unified memory architecture with shared GPU/CPU access
Memory Bandwidth: Achieved ~13.5 GB/s in multi-threaded tests

Graphics Processing Unit

GPU Architecture: Strix Halo (RDNA 3.5 based)
GPU Designation: gfx1151
Compute Units: 40 CUs (80 reported in ROCm, likely accounting for dual SIMD per CU)
Peak GPU Clock: 2,900 MHz
VRAM: 96 GB shared system memory (103 GB total addressable) - Note: This allocation was intentionally configured to maximize GPU memory for large language model inference
Memory Bandwidth: Shared with system memory
OpenCL Compute Units: 20 (as reported by clinfo)

Platform Details

Operating System: Ubuntu 24.04.3 LTS (Noble)
Kernel Version: 6.8.0-83-generic
Architecture: x86_64
Virtualization: AMD-V enabled

Performance Benchmarks

Figure 1: Comprehensive performance analysis and compatibility overview of the AMD AI Max+ 395 system

CPU Performance Analysis

Single-Threaded Performance

The sysbench CPU benchmark with prime number calculation revealed strong single-threaded performance:

Events per second: 6,368.92
Average latency: 0.16 ms
95th percentile latency: 0.16 ms

This performance places the AMD AI Max+ 395 in the upper tier of modern processors for single-threaded workloads, demonstrating the effectiveness of the Zen 5 architecture's IPC improvements and high boost clocks.

Multi-Threaded Performance

Multi-threaded testing across all 32 threads showed excellent scaling:

Events per second: 103,690.35
Scaling efficiency: 16.3x improvement over single-threaded (theoretical maximum 32x)
Thread fairness: Excellent distribution with minimal standard deviation

The scaling efficiency of approximately 51% indicates good multi-threading performance, though there's room for optimization in workloads that can fully utilize all available threads.

Memory Performance

Memory Bandwidth Testing

Memory performance testing using sysbench revealed:

Single-threaded bandwidth: 9.3 GB/s
Multi-threaded bandwidth: 13.5 GB/s (16 threads)
Latency characteristics: Sub-millisecond access times

The memory bandwidth results suggest the system is well-balanced for most workloads, though AI applications requiring extremely high memory bandwidth may find this a limiting factor compared to discrete GPU solutions with dedicated VRAM.

GPU Performance and Capabilities

Hardware Specifications

The integrated Radeon 8060S GPU presents impressive specifications on paper:

Architecture: RDNA 3.5 (Strix Halo)
Compute Units: 40 CUs with 2 SIMDs each
Memory Access: Full 96 GB of shared system memory
Clock Speed: Up to 2.9 GHz

OpenCL Capabilities

OpenCL enumeration reveals solid compute capabilities:

Device Type: GPU with full OpenCL 2.1 support
Max Compute Units: 20 (OpenCL reporting)
Max Work Group Size: 256
Image Support: Full 2D/3D image processing capabilities
Memory Allocation: Up to 87 GB maximum allocation

Network Performance Testing

Network infrastructure testing using iperf3 demonstrated excellent localhost performance:

Loopback Bandwidth: 122 Gbits/sec sustained
Latency: Minimal retransmissions (0 retries)
Consistency: Stable performance across 10-second test duration

This indicates robust internal networking capabilities suitable for distributed computing scenarios and high-bandwidth data transfer requirements.

PyTorch/ROCm Compatibility Analysis

Current State of ROCm Support

We installed ROCm 7.0 and related components: - ROCm Version: 7.0.0 - HIP Version: 7.0.51831 - PyTorch Version: 2.5.1+rocm6.2

gfx1151 Compatibility Issues

The most significant finding of this review centers on the gfx1151 architecture compatibility with current AI software stacks. Testing revealed critical limitations:

PyTorch Compatibility Problems

rocBLAS error: Cannot read TensileLibrary.dat: Illegal seek for GPU arch : gfx1151
List of available TensileLibrary Files:
- TensileLibrary_lazy_gfx1030.dat
- TensileLibrary_lazy_gfx906.dat
- TensileLibrary_lazy_gfx908.dat
- TensileLibrary_lazy_gfx942.dat
- TensileLibrary_lazy_gfx900.dat
- TensileLibrary_lazy_gfx90a.dat
- TensileLibrary_lazy_gfx1100.dat

This error indicates that PyTorch's ROCm backend lacks pre-compiled optimized kernels for the gfx1151 architecture. The absence of gfx1151 in the TensileLibrary files means:

No Optimized BLAS Operations: Matrix multiplication, convolutions, and other fundamental AI operations cannot leverage GPU acceleration
Training Workflows Broken: Most deep learning training pipelines will fail or fall back to CPU execution
Inference Limitations: Even basic neural network inference is compromised

Root Cause Analysis

The gfx1151 architecture represents a newer GPU design that hasn't been fully integrated into the ROCm software stack. While the hardware is detected and basic OpenCL operations function, the optimized compute libraries essential for AI workloads are missing.

Workaround Attempts

Testing various workarounds yielded limited success:

HSA_OVERRIDE_GFX_VERSION=11.0.0: Failed to resolve compatibility issues
CPU Fallback: PyTorch operates normally on CPU, but defeats the purpose of GPU acceleration
Basic GPU Operations: Simple tensor allocation succeeds, but compute operations fail

Software Ecosystem Gaps

Beyond PyTorch, the gfx1151 compatibility issues extend to:

TensorFlow: Likely similar rocBLAS dependency issues
JAX: ROCm backend compatibility uncertain
Scientific Computing: NumPy/SciPy GPU acceleration unavailable
Machine Learning Frameworks: Most frameworks dependent on rocBLAS will encounter issues

AMD GPU Software Support Ecosystem Analysis

Current State Assessment

AMD's GPU software ecosystem has made significant strides but remains fragmented compared to NVIDIA's CUDA platform:

Strengths

Open Source Foundation: ROCm's open-source nature enables community contributions
Standard API Support: OpenCL 2.1 and HIP provide industry-standard interfaces
Linux Integration: Strong kernel-level support through AMDGPU drivers
Professional Tools: rocm-smi and related utilities provide comprehensive monitoring

Weaknesses

Fragmented Architecture Support: New architectures like gfx1151 lag behind in software support
Limited Documentation: Less comprehensive than CUDA documentation
Smaller Developer Community: Fewer third-party tools and optimizations
Compatibility Matrix Complexity: Different software versions support different GPU architectures

Long-term Viability Concerns

The gfx1151 compatibility issues highlight broader ecosystem challenges:

Release Coordination Problems

Hardware releases outpace software ecosystem updates
Critical libraries (rocBLAS, Tensile) require architecture-specific optimization
Coordination between AMD hardware and software teams appears insufficient

Market Adoption Barriers

Developers hesitant to adopt platform with uncertain software support
Enterprise customers require guaranteed compatibility
Academic researchers need stable, well-documented platforms

Recommendations for AMD

Accelerated Software Development: Prioritize gfx1151 support in rocBLAS and related libraries
Pre-release Testing: Ensure software ecosystem readiness before hardware launches
Better Documentation: Comprehensive compatibility matrices and migration guides
Community Engagement: More responsive developer relations and support channels

Network Infrastructure and Connectivity

The system demonstrates excellent network performance characteristics suitable for modern computing workloads:

Internal Performance

Memory-to-Network Efficiency: 122 Gbps loopback performance indicates minimal bottlenecks
System Integration: Unified memory architecture benefits network-intensive applications
Scalability: Architecture suitable for distributed computing scenarios

External Connectivity Assessment

While specific external network testing wasn't performed, the system's infrastructure suggests:

Support for high-speed Ethernet (2.5GbE+)
Low-latency interconnects suitable for cluster computing
Adequate bandwidth for data center deployment scenarios

Power Efficiency and Thermal Characteristics

Limited thermal data was available during testing:

Idle Temperature: 29°C (GPU sensor)
Idle Power: 8.059W (GPU subsystem)
Thermal Management: Appears well-controlled under light loads

The unified architecture's power efficiency represents a significant advantage over discrete GPU solutions, particularly for mobile and edge computing applications.

Competitive Analysis

Comparison with Intel Arc

Intel's Arc GPUs face similar software ecosystem challenges, though Intel has made more aggressive investments in AI software stack development. The Arc series benefits from Intel's deeper software engineering resources but still lags behind NVIDIA in AI framework support.

Comparison with NVIDIA

NVIDIA maintains a substantial advantage in:

Software Maturity: CUDA ecosystem is mature and well-supported
AI Framework Integration: Native support across all major frameworks
Developer Tools: Comprehensive profiling and debugging tools
Documentation: Extensive, well-maintained documentation

AMD's advantages include:

Open Source Approach: More flexible licensing and community development
Unified Memory: Simplified programming model for certain applications
Cost: Potentially more cost-effective solutions

Market Positioning

The AMD AI Max+ 395 occupies a unique position as a high-performance integrated solution, but software limitations significantly impact its competitiveness in AI-focused markets.

Use Case Suitability Analysis

Recommended Use Cases

General Computing: Excellent performance for traditional computational workloads
Development Platforms: Strong for general software development (non-AI)
Edge Computing: Unified architecture benefits power-constrained deployments
Future AI Workloads: When software ecosystem matures

Not Recommended For

Current AI Development: gfx1151 compatibility issues are blocking
Production AI Inference: Unreliable software support
Machine Learning Research: Limited framework compatibility
Time-Critical Projects: Uncertain timeline for software fixes

Large Language Model Performance and Stability

Ollama LLM Inference Testing

Testing with Ollama reveals a mixed picture for LLM inference on the AMD AI Max+ 395 system. The platform successfully runs various models through CPU-based inference, though GPU acceleration faces significant challenges.

Performance Metrics

Testing with various model sizes revealed the following performance characteristics:

GPT-OSS 20B Model Performance:

Prompt evaluation rate: 61.29 tokens/second
Text generation rate: 8.99 tokens/second
Total inference time: ~13 seconds for 117 tokens
Memory utilization: ~54 GB VRAM usage

Llama 4 (67B) Model:

Successfully loads and runs
Generation coherent and accurate

The system demonstrates adequate performance for smaller models (20B parameters and below) when running through Ollama, though performance significantly lags behind NVIDIA GPUs with proper CUDA acceleration. The large unified memory configuration (96 GB VRAM, deliberately maximized for this testing) allows loading of substantial models that would typically require multiple GPUs or extensive system RAM on other platforms. This conscious decision to allocate maximum memory to the GPU was specifically made to evaluate the system's potential for large language model workloads.

Critical Stability Issues with Large Models

Driver Crashes with Advanced AI Workloads

Testing revealed severe stability issues when attempting to run larger models or when using AI-accelerated development tools:

Affected Scenarios:

Large Model Loading: GPT-OSS 120B model causes immediate amdgpu driver crashes
AI Development Tools: Continue.dev with certain LLMs triggers GPU reset
OpenAI Codex Integration: Consistent driver failures with models exceeding 70B parameters

GPU Reset Events

System logs reveal frequent GPU reset events during AI workload attempts:

[ 1030.960155] amdgpu 0000:c5:00.0: amdgpu: GPU reset begin!
[ 1033.972213] amdgpu 0000:c5:00.0: amdgpu: MODE2 reset
[ 1034.002615] amdgpu 0000:c5:00.0: amdgpu: GPU reset succeeded, trying to resume
[ 1034.003141] [drm] VRAM is lost due to GPU reset!
[ 1034.037824] amdgpu 0000:c5:00.0: amdgpu: GPU reset(1) succeeded!

These crashes result in:

Complete loss of VRAM contents
Application termination
Potential system instability requiring reboot
Interrupted workflows and data loss

Root Cause Analysis

The driver instability appears to stem from the same underlying issue as the PyTorch/ROCm incompatibility: immature driver support for the gfx1151 architecture. The drivers struggle with:

Memory Management: Large model allocations exceed driver's tested parameters
Compute Dispatch: Complex kernel launches trigger unhandled edge cases
Power State Transitions: Rapid load changes cause driver state machine failures
Synchronization Issues: Multi-threaded inference workloads expose race conditions

Implications for AI Development

The combination of LLM testing results and driver stability issues reinforces that the AMD AI Max+ 395 system, despite impressive hardware specifications, remains unsuitable for production AI workloads. The platform shows promise for future AI applications once driver maturity improves, but current limitations include:

Unreliable Large Model Support: Models over 70B parameters risk system crashes
Limited Tool Compatibility: Popular AI development tools cause instability
Workflow Interruptions: Frequent crashes disrupt development productivity
Data Loss Risk: VRAM resets can lose unsaved work or model states

Future Outlook and Development Roadmap

Short-term Expectations (3-6 months)

ROCm updates likely to address gfx1151 compatibility
PyTorch/TensorFlow support should improve
Community-driven workarounds may emerge

Medium-term Prospects (6-18 months)

Full AI framework support expected
Optimization improvements for Strix Halo architecture
Better documentation and developer resources

Long-term Considerations (18+ months)

AMD's commitment to open-source ecosystem should pay dividends
Potential for superior price/performance ratios
Growing developer community around ROCm platform

Conclusions and Recommendations

The AMD AI Max+ 395 system represents impressive hardware engineering with its unified memory architecture, strong CPU performance, and substantial GPU compute capabilities. However, critical software ecosystem gaps, particularly the gfx1151 compatibility issues with PyTorch and ROCm, severely limit its immediate utility for AI and machine learning workloads.

Key Findings Summary

Hardware Strengths:

Excellent CPU performance with 16 Zen 5 cores
Innovative unified memory architecture with 96 GB addressable
Strong integrated GPU with 40 compute units
Efficient power management and thermal characteristics

Software Limitations:

Critical gfx1151 architecture support gaps in ROCm ecosystem
PyTorch integration completely broken for GPU acceleration
Limited AI framework compatibility across the board
Insufficient documentation for troubleshooting

Market Position:

Competitive hardware specifications
Unique integrated architecture advantages
Significant software ecosystem disadvantages versus NVIDIA
Uncertain timeline for compatibility improvements

Purchasing Recommendations

Buy If: - Primary use case is general computing or traditional HPC workloads - Willing to wait 6-12 months for AI software ecosystem maturity - Value open-source software development approach - Need power-efficient integrated solution

Avoid If:

Immediate AI/ML development requirements
Production AI inference deployments planned
Time-critical project timelines
Require guaranteed software support

Final Verdict

The AMD AI Max+ 395 system shows tremendous promise as a unified computing platform, but premature software ecosystem development makes it unsuitable for current AI workloads. Organizations should monitor ROCm development progress closely, as this hardware could become highly competitive once software support matures. For general computing applications, the system offers excellent performance and value, representing AMD's continued progress in processor design and integration.

The AMD AI Max+ 395 represents a glimpse into the future of integrated computing platforms, but early adopters should be prepared for software ecosystem growing pains. As AMD continues investing in ROCm development and the open-source community contributes solutions, this platform has the potential to become a compelling alternative to NVIDIA's ecosystem dominance.