1. Reminiscence fragmentation
Inner fragmentation
Programs pre-allocate a big chunk of reminiscence for every request, assuming the utmost potential output size (e.g., 2048 tokens). Nevertheless, if a request solely generates a brief output, a lot of that reserved reminiscence goes unused, resulting in important waste.
Exterior fragmentation
As a result of totally different requests reserve chunks of various sizes, the GPU reminiscence turns into scattered with unusable small gaps, making it laborious to suit new requests even when complete free reminiscence is obtainable. Our sources present that in current methods, solely 20.4% – 38.2% of KV cache reminiscence is definitely used to retailer token states, with the remaining being waste.
2. No reminiscence sharing
Superior decoding methods like parallel sampling or beam search usually generate a number of outputs from a single immediate, which means they might share components of the KV cache. Nevertheless, current methods can not simply share this reminiscence as a result of every sequence’s KV cache is in its personal separate, contiguous block.
