Menu

Hierarchical KV sharing reduces transformer memory usage
📰

Hierarchical KV sharing reduces transformer memory usage

DEV Community: machinelearning·Papers Mache·about 2 hours ago
#94oO5G7s
Reading 0:00
15s threshold

Hierarchical key‑value sharing cuts transformer memory use by about one third while preserving, even improving, downstream performance. The idea is simple: later layers reuse the KV pairs produced by earlier, stride‑aligned layers instead of materialising their own caches. This challenges the prevailing belief that deep models must keep a full per‑layer cache to maintain quality.

Before DepthGraft, cross‑layer connectivity was introduced mainly as an architectural novelty, often adding extra matrix multiplications or memory hops. Adjacent‑layer sharing and global‑anchor schemes have been explored, but they do not provide a systematic regularisation of the KV geometry nor a substantial compression of the cache. Practitioners therefore accepted the memory cost of deep inference as inevitable.

DepthGraft reduces the KV cache by roughly one third.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.