Hardware-Aware Dynamic Speculative Decoding
Cohere, Friday, July 10th, 2026
Cohere introduces a technique that adapts speculative decoding to GPU constraints for faster LLM inference.
This article introduces Hardware-Aware Dynamic Speculative Decoding (DSD), a technique that optimizes LLM inference by adaptively adjusting the number of draft tokens based on GPU hardware constraints. Unlike traditional speculative decoding with fixed token counts, DSD increases K when inference is memory-bandwidth-bound and decreases it when compute-bound, enabling faster performance across varying batch sizes.
Contributed to the vLLM inference framework, it demonstrates roughly 23% faster performance than fixed-K decoding at high batch sizes while remaining compatible with production optimizations like asynchronous scheduling and CUDA graphs.