TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
Paper • 2404.11912 • Published • 17
Papers - Speculative Decoding - KV Cache
we recognize two memory bottlenecks: model weights and KV cache, and the latter gradually bottleneck(s) as context length increases