Menu

#Mamba

6 posts

Feed·
6 of 6 posts
Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch Sizes
🖼️
0Read More

Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch Sizes

DEV Community: ai·Mayur Agnani·20 days ago
#Aakqp35q
#devtoai#devto#dev#state#batch

An attention layer undoes a rejected draft token by moving a pointer. A Mamba layer overwrites one fixed state every token, so the server saves a full copy per draft. At large batch sizes those copies erase the speed-up. Logging each token's small in

15s
0Read More

Technical question about Mamba Selective Scan kernel and FP16/FP32 precision

Reddit r/learnmachinelearning·u/Dry-Trouble4373·7 months ago
#kGw7XvYx
#kernel#fp16#mamba#fp32#precision

I'm trying to evaluate the model's accuracy when all internal operations are strictly limited to **FP16**. However, I noticed that the `selective_scan` CUDA kernel seems to use **FP32 accumulators** by default.…

15s