KV, Prefix, and Session Caching in Co-located and Disaggregated LLM Serving

Sun, 11 Oct 2026 00:00:00 +0000

Splitting prefill and decode onto separate GPU pools turns the KV cache from local state into data that has to be routed, moved, and restored. Moving it once turns out to be cheap: in a worked example of a 70B-class coding agent on B200s, handing a 32K-token prompt’s KV from prefill to decode adds about 5% to time to first token over 400 Gb/s RDMA, even without overlap. What is hard is keeping KV reusable. Reuse pays off enormously, since session reuse alone cuts the agent’s prefill work about 24×, but stored KV competes for GPU memory, the resource that already limits decode. Disaggregation adds a second difficulty: prefix hits now depend on routing across two pools, and the newest copy of each session sits on the decode side, where pulling it back every turn costs more as the conversation grows. The upshot is that for any given set of GPUs, caching choices (session reuse, affinity, memory tiers, KV format) determine the traffic it can serve far more than the decision to disaggregate does.

Kv Cache on Li Cao's Blog

KV, Prefix, and Session Caching in Co-located and Disaggregated LLM Serving