Menu

#Int4

3 posts

Feed·
3 of 3 posts
4-bit Naive Bayes: symmetric fails on AG News, zero-point rescues it
🖼️
0Read More

4-bit Naive Bayes: symmetric fails on AG News, zero-point rescues it

DEV Community: machinelearning·mycute·6 days ago
#RrQLZo9j

Two minimum-bit experiments: int4 NB keeps 93.2% on SMS at 160 bytes but collapses on AG News; TF-IDF coreset selection wins only at very small budgets.

15s
KV Cache Quantization for On-Device LLM Inference on Android
🖼️
0Read More

KV Cache Quantization for On-Device LLM Inference on Android

DEV Community·SoftwareDevs mvpfactory.io·6 months ago
#uHfXZPWY
#webdev#programming#software#coding#memory

Deep dive into KV cache memory management for on-device LLM inference — covering how quantizing key-value attention caches from FP16 to INT4 with group-wise scaling reduces memory footprint by 75%, implementing sliding window eviction policies that…

15s
How to Deploy Llama 3.2 405B with Quantization on a $60/Month DigitalOcean GPU Droplet: Enterprise Reasoning Without the $20K/Month API Bill
📰
0Read More

How to Deploy Llama 3.2 405B with Quantization on a $60/Month DigitalOcean GPU Droplet: Enterprise Reasoning Without the $20K/Month API Bill

DEV Community·RamosAI·7 months ago
#GDNNaul7
#part#programming#tutorial#ai#llama

From Dev.to - webdev: How to Deploy Llama 3.2 405B with Quantization on a $60/Month DigitalOcean GPU Droplet: Enterprise Reasoning Without the $20K/Month API Bill

15s