Why FP16 Is Slow on a Tesla P40, and Why FP32 Was Slow Too

A.C. Jokela — Sat, 31 Oct 2026 21:30:00 GMT

In February I ran FireRed-Image-Edit on four Tesla P40s and found that FP32 was faster than FP16, about 9 minutes per denoising step in FP16 against 1.5 to 2.4 minutes in FP32. I explained it with one sentence: FP16 on Pascal "is handled through slower pathways." That sentence names no pathway. It was a guess dressed up as an explanation.

Then Sevan emailed me a second data point. Sevan runs the AI behind a free wardrobe app on a single P40 and had been loading a virtual try-on model, Leffa, in FP16 since launch to save memory. Switching to FP32 took a 20-step render from 1,241 seconds to 85: 63.8 seconds per step in FP16 against 4.7 in FP32, with images that differ by 0.77/255 per pixel on average. The measurements and the dtype rule Sevan landed on are in Sevan's write-up. Thank you, Sevan, for taking the time to write to me; it is the reason this post exists.

Two independent P40s, two different diffusion models, the same 4x to 14x penalty for FP16. I wanted to know which operation causes it, so I measured each one.

The setup

Everything ran on the same machine as the February post: four Tesla P40s in a Penguin Computing 2U chassis, NVIDIA driver 580.178.04, and the same Python environment, PyTorch 2.6.0 with CUDA 12.4 and cuDNN 9.1. Single-card tests ran on one P40, which held its 1,531 MHz boost clock throughout (the shop is cold enough that the cards never passed 33°C). I used FireRed-Image-Edit-1.0's real weights, sized every test to the February edit job (a roughly 1024x1024 target plus a 1024x1024 reference image, which comes to 8,192 image tokens plus a short prompt), and ran each precision in its own process.

There are three layers of test: raw operations (matrix multiply, attention, convolution, elementwise math), the first six of FireRed's 60 transformer blocks with real weights, and the full pipeline in FP32 across all four cards.

FP16 matrix math is not the problem

The P40's GP102 chip does FP16 arithmetic at 1/64 of its FP32 rate, so the natural assumption is that every FP16 operation crawls. Matrix multiplies do not:

Operation (one P40)	FP32	FP16	BF16
Matrix multiply, 4096³	9.4 TFLOPS	9.8 TFLOPS	5.0 TFLOPS
Matrix multiply, MLP-shaped (8448×3072×12288)	9.7 TFLOPS	10.2 TFLOPS	4.5 TFLOPS
Attention, 24 heads × 8448 tokens × 128	5.94 TFLOPS	0.24 TFLOPS	2.67 TFLOPS
3×3 convolution, 128 channels at 512²	16.8 TFLOPS	13.3 TFLOPS	3.4 TFLOPS
Elementwise multiply, 2²⁸ values	24.9 G/s	22.8 G/s	45.9 G/s

(The FP32 and FP16 convolution figures exceed the card's 11.8 TFLOPS peak because cuDNN picks a fast algorithm such as Winograd, which does fewer multiplies than the nominal count I divided by.)

The profiler shows why FP16 matrix multiplies run at full speed: cuBLAS on Pascal uses kernels named maxwell_fp16_sgemm_fp16, which read FP16, compute in FP32, and write FP16. The data is half precision; the arithmetic is not. Speed varies by shape (one FP16 down-projection shape ran at 2.8 TFLOPS through F.linear), but nothing like 1/64.

One row in that table is 25 times slower than FP32: attention.

One attention kernel

PyTorch's scaled_dot_product_attention picks a backend at runtime. On a P40 in FP16 it picks the memory-efficient kernel, fmha_cutlassF_f16_aligned_32x128_rf_sm50, and that kernel takes 3.69 seconds for an attention call that takes 0.148 seconds in FP32. It evidently does at least part of its arithmetic in FP16, which this chip runs at 1/64 speed. Forcing the plain "math" backend, which upcasts to FP32 and runs ordinary matrix multiplies, brings the same call down to 0.328 seconds.

On real FireRed blocks, that one kernel is 89% of the GPU time in FP16: 7.40 of 8.32 seconds across two blocks. Everything else in an FP16 block is faster than FP32.

Six real FireRed blocks, one P40	Per block	Output vs FP32
FP32, stock	1.08 s	reference
FP16, stock	4.17 s	every value inf or NaN
FP16, math attention forced	0.92 s	every value inf or NaN
BF16 (math attention is automatic)	0.86 s	0.8% relative error

Scaled to a full denoising step (60 blocks, two passes for classifier-free guidance), stock FP16 comes to about 500 seconds, 8.3 minutes. February's measured "approximately 9 minutes per step" sits right next to it, which is the best evidence I have that this kernel was February's problem too.

Fixing the speed does not rescue FP16 for this model. After six blocks, all 524,288 output values had overflowed. These blocks were fed random inputs rather than real text embeddings, so I would not read too much into exactly where it breaks; the February pipeline, with real inputs, produced a solid black image for the same reason. FP16's maximum value is 65,504, and FireRed's activations exceed it.

BF16 runs on a P40

Several of my earlier P40 posts say the card cannot compute in BF16. That is half right. Pascal has no BF16 arithmetic, but PyTorch and cuBLAS treat BF16 as a storage format: the matrix multiplies run through magma_sgemmEx kernels that read BF16 and compute in FP32. BF16 has FP32's exponent range, so nothing overflowed, and its error against FP32 after six blocks was 0.8%.

The costs are real but specific. BF16 matrix multiplies run at about half the FP32 rate. BF16 convolutions run at a fifth of it, so the VAE decode of a 1024x1024 image took 3.3 seconds in BF16 against 1.8 in FP32. (FP16 took 3.5 seconds there and, unlike the transformer, decoded cleanly: 0.03/255 average difference from FP32.) And there is no memory-efficient attention kernel for BF16 on Pascal, so PyTorch falls back to the math backend, which materializes the full attention matrix: six BF16 blocks peaked at 20.5 GB against 9.9 GB for six FP32 blocks. The March home-lab post's figure of "roughly 21% slower" does not match any of these measurements; the right answer depends entirely on which operation you are running.

FP32 was running at half speed

BF16 beating FP32 on the blocks made no sense if BF16 matrix multiplies are half as fast, so I profiled FP32. Inside the model, its matrix multiplies averaged 2.3 TFLOPS, about a quarter of what the same shapes reach in isolation. The difference is the bias.

nn.Linear calls F.linear(x, weight, bias), and PyTorch hands the bias to cuBLAS to fuse into the matrix multiply. For FP32 on Pascal, that fused path selects a different kernel, maxwell_sgemm_32x128_tn, which runs at 3.9 TFLOPS. The same multiply without the bias selects sgemm_128x128x8_NT_vec and runs at 8.5 TFLOPS:

FP32 linear layer, 8192 tokens, 3072→12288	Time	Throughput
`F.linear(x, w, b)` (bias fused)	157 ms	3.9 TFLOPS
`F.linear(x, w) + b`	73–77 ms	8.0–8.5 TFLOPS

In that test the two outputs were bit-identical. PyTorch 2.10 with CUDA 12.6 behaves the same as 2.6. FP16 is not affected: with or without the bias, the same shape takes 79 ms.

The fix is a replacement forward for nn.Linear. Apply it before loading the model, because accelerate's device hooks capture forward on some modules when a model is spread across GPUs:

import torch
import torch.nn.functional as F

def linear_forward(self, x):
    y = F.linear(x, self.weight)
    return y if self.bias is None else y + self.bias

torch.nn.Linear.forward = linear_forward

On six FireRed blocks it took FP32 from 1.08 to 0.53 seconds per block, 2.06 times faster, with a relative difference from the stock output of 6×10⁻⁷ and the same 9.9 GB peak memory. On these blocks, split-bias FP32 beats BF16 by 1.6x and stock FP16 by 7.9x, and it is the only configuration that is both the fastest and numerically clean.

The full pipeline

Block numbers are extrapolations, so I ran the real thing: FireRed-Image-Edit end to end with February's inputs (the same photo, "Add a red hat on the cat," seed 43, true CFG at 4.0). The FP32 transformer is 82 GB, so it is split across all four P40s, 12 blocks on the first card and 16 on each of the others, and every card peaked between 21.0 and 21.8 GiB. As in February, the text encoder ran once in FP16 and was freed before the transformer loaded. The final VAE decode ran on the CPU, so a 40-step run could not run out of GPU memory at the very end.

FireRed-Image-Edit, four P40s	Per step	40 steps
FP32, stock (5-step run)	144.2 s	about 96 min
FP32, bias split out (40-step run)	62.8 s	41.9 min
February: INT8 transformer, FP32 everything else	87.5 s	58 min

The steps barely vary: 144.2 to 144.3 seconds stock, 62.6 to 63.0 seconds with the fix. After five steps, the split-bias image differed from the stock one by 0.005/255 per pixel on average and 6/255 at most; the two matrix-multiply kernels add in a different order, so the results are not bit-identical, but they are the same picture. The 40-step result is the same red hat on the same cat that February's INT8 run produced.

Split-bias FP32 takes 28% less time per step than February's INT8 configuration on the same cards, with no quantization at all.

Stock FP32 came in slower than my block-level estimate, 144 seconds against 130. The block test leaves out the input and output layers, three hand-offs between cards on every pass, and the scheduler; that is where the other 14 seconds go.

And the Strix Halo

February's headline was that a single AMD Strix Halo APU ran this model 6% faster than all four P40s: 82.6 seconds per step against 87.5. If the P40s now take 62.8, that headline flips, but only if the Strix is held to the same standard. The P40s just got tuned. So did the Strix: same photo, prompt, seed and guidance, the whole transformer in BF16 as in February, five steps per configuration, timed the same way.

FireRed-Image-Edit, one Strix Halo (BF16)	Per step
February's software (PyTorch 2.7.1, ROCm 7.9 RC1), stock	106.9 s
same, bias split out	106.9 s
same, AOTriton attention enabled	106.9 s
Newer software (PyTorch 2.9.1, ROCm 7.11 RC2), stock	107.7 s
newer, AOTriton attention enabled	21.6 s
newer, AOTriton plus bias split out	20.0 s
newer, AOTriton plus TunableOp	20.6 s
newer, AOTriton, TunableOp and bias split out	19.4 s

One setting does almost all of it. TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 lets PyTorch use AOTriton's flash-attention kernels on this GPU instead of the math fallback, and on the newer software it takes a step from 107.7 seconds to 21.6 and peak memory from 52.9 GiB to 38.9. February's software accepts the variable and ignores it. TunableOp, which times the available matrix-multiply kernels for each shape and keeps the fastest, takes another 5%; the first step pays for the search (one to three minutes) and the rest reuse it. Splitting out the bias, the P40's big win, is worth nothing on February's ROCm stack and 7% on the newer one: it is a cuBLAS kernel-selection problem, not a PyTorch one.

Fully tuned, the Strix Halo is 3.2 times faster than the four tuned P40s, 19.4 seconds per step against 62.8. February's 6% was a comparison between two untuned machines, and the untuned Strix was leaving more on the table than the untuned P40s were. The P40 fix is real and worth having. It does not change who wins.

Two things here do not reconcile, and I am not going to paper over them. First, February's software, run today, takes 106.9 seconds per step, not February's 82.6. The one difference I can see is the GPU memory configuration: in February the pipeline used about 75 GB of GPU memory, and today PyTorch sees 59.6 GiB, of which only 4 GiB is a dedicated carve-out, with the rest borrowed from system memory. I have not changed the BIOS to test February's setting, so the 29% gap is unexplained. Second, that memory limit is why the text encoder and the VAE run on the CPU in these tests; the whole BF16 pipeline (57.7 GB) no longer fits. Neither runs inside the timed denoising loop, so the per-step numbers compare like with like, but it is not February's exact setup. The images agree: every configuration produced the same red hat on the same cat, with BF16 and different kernels drifting by 1.5 to 2.7/255 per pixel on average after five steps.

Caveats

The block tests use FireRed's real weights but random inputs, and they time 6 of 60 blocks; the per-step figures for anything other than the full pipeline runs are extrapolations that leave out the text encoder, the VAE, and transfers between cards. The P40 runs used PyTorch 2.6.0 with CUDA 12.4 on driver 580.178.04 (the bias result also holds on PyTorch 2.10 with CUDA 12.6); the Strix runs used the two ROCm stacks in the table, with the same diffusers, transformers and accelerate versions. The Strix numbers are five-step runs, and the P40 runs are FP32 across four cards while the Strix runs are BF16 on one APU: each is that machine's fastest correct configuration, not the same arithmetic. I did not test training, other models, or newer GPUs, which have FP16 and BF16 hardware and will rank the three precisions differently. The bias result is a cuBLAS kernel-selection behavior, and a future cuBLAS could change it.

What I would do on a P40 now

Run FP32 and patch nn.Linear to split out the bias. If a model does not fit in FP32, store it in BF16 rather than FP16, because BF16 will not overflow, and budget for the math-attention memory. If you must use FP16, force the math attention backend with torch.nn.attention.sdpa_kernel(SDPBackend.MATH) and check the output for inf and NaN, because the speed fix does nothing for the range problem.

The March LTX-Video run on these same four cards patched F.linear to upcast each weight and bias to FP32 and then call the original F.linear with the bias attached, which is exactly the slow path. It probably paid this penalty on every linear layer. I have not re-run it.

TinyComputers.io (Posts about cublas)