Hierarchical key‑value sharing cuts transformer memory use by about one third while preserving, even improving, downstream performance. The idea is simple: later layers reuse the KV pairs produced by earlier, stride‑aligned layers instead of materialising their own caches. This challenges the prevailing belief that deep models must keep a full per‑layer cache to maintain quality.
Before DepthGraft, cross‑layer connectivity was introduced mainly as an architectural novelty, often adding extra matrix multiplications or memory hops. Adjacent‑layer sharing and global‑anchor schemes have been explored, but they do not provide a systematic regularisation of the KV geometry nor a substantial compression of the cache. Practitioners therefore accepted the memory cost of deep inference as inevitable.
DepthGraft reduces the KV cache by roughly one third.…