🖼️00Read later0Read MoreWhy Long bf16 Training Runs Stall: The Residual Stream Has No Room for Small UpdatesDEV Community: pytorch·AI OpenFree·5 days ago#zleSNZin#devtopytorch#devto#dev#residual#fp32+5 more🧰Tag tools✨Add tagA bf16 residual near 600 has a resolution of 4, so small updates vanish. Measured gaps, a reproducible simulation and fixes for long training runs.15s
00Read later0Read MoreTechnical question about Mamba Selective Scan kernel and FP16/FP32 precisionReddit r/learnmachinelearning·u/Dry-Trouble4373·6 months ago#kGw7XvYx#kernel#fp16#mamba#fp32#precision+4 more🧰Tag tools✨Add tagI'm trying to evaluate the model's accuracy when all internal operations are strictly limited to **FP16**. However, I noticed that the `selective_scan` CUDA kernel seems to use **FP32 accumulators** by default.… Read more15s
