A dtype conversion can remain visible after the final FP8 quantization. It belongs in a fusion's numerical contract, even when its intermediate allocation is removed.
Here is a small CPU example:
x = torch . tensor ([ 1.0626 ], dtype = torch . float32 ) x . to ( torch . bfloat16 ). to ( torch . float8_e4m3fn ). float () # 1.0 x . to ( torch . float8_e4m3fn ). float () # 1.125
Enter fullscreen mode
Exit fullscreen mode
The exact FP32 input is 1.062600016593933 . BF16 rounds it to 1.0625 : the midpoint between the two FP8 values 1.0 and 1.125 . Round-to-nearest-even sends that midpoint to 1.0 ; the original FP32 value sits just above it.
A complete control band
I used all 65,536 exactly representable FP32 values 1 + i/65536 , for i = 0β¦65535 . The two eager conversion paths differ at 2,048 values, or 3.125% . A standard-library checker reproduces the geometry.
For CPU compilation I tested these two functions:
def cast_only ( x ): return x . to ( torch . bfloat16 ). to ( torch . float8_e4m3fn ).β¦