144 points • 7 days ago • Article Link

推测式解码是对大型语言模型的一种强力优化，用以克服标准自回归解码的效率瓶颈。基线方法在循环中每次只生成并确认一个 token，而推测式解码将流程拆成两个阶段：先由一个轻量级草稿模块提出一串候选 token，随后目标模型在一次前向中验证这些候选项。如果目标模型接受草稿的 token，系统即可一次性提交多个输出，从而显著提升吞吐量。

文章介绍了 vLLM 当前支持的五种具体草稿方法：native Multi-Token Prediction (MTP) 、 Gemma 4 MTP 、 EAGLE-3 、 DFlash 和 DSpark 。这些方法主要在于如何利用目标模型的信息以及如何生成候选 token 存在差异。例如，native MTP 使用模型本身的辅助路径，而 DFlash 和 DSpark 则采用专门的、以目标为条件的网络并行预测整块未来 token 。相比之下，EAGLE-3 依赖一种自回归机制，结合了目标 Transformer 不同阶段的隐层状态。

在 AMD Instinct™ MI300X 和 MI355X GPU 上的实验表明，推测式解码的效果高度依赖具体的模型家族、草稿检查点和目标工作负载。某些配置的吞吐量提升超过 2 倍，但并非所有数据集和模型都能获得一致的性能提高。推测 token 的数量（即 proposal length）是关键调参项：随着草稿 token 数量增加，吞吐量通常会上升，但到达某一点后会趋于平缓甚至下降，因为额外的草稿开销开始抵消验证成功带来的收益。

在实际落地时，推测式解码需要细致的观测和调优。文章建议监控吞吐量、平均被接受长度和各位置的接受率等指标，以确定最优配置。由于不同负载的 token 可预测性差异显著，通常没有一刀切的设置。推荐的流程是从已知配置出发，针对不同的 proposal length 进行扫描，找到最适合当前任务的平衡点。

最后，为新的目标模型训练定制的 speculator 是进一步提升性能的可行途径。该过程包括收集目标模型的代表性隐层状态来训练草稿组件，确保投机器与目标的内部表征对齐。通过让草稿模型的架构和训练数据与目标领域（例如数学或代码生成）相匹配，用户可以获得更高的接受率，从而在生产环境中实现更高的服务吞吐量。 Speculative decoding serves as a powerful optimization for large language models, addressing the efficiency limitations of standard autoregressive decoding. While the baseline approach generates and commits one token at a time in a repetitive loop, speculative decoding separates the process into two distinct phases. A lightweight draft component first proposes a sequence of candidate tokens, and the original target model then verifies these candidates in a single pass. If the target model accepts the draft tokens, the system commits multiple outputs simultaneously, significantly increasing throughput.

The article details five specific drafting methods currently supported in vLLM: native Multi-Token Prediction (MTP), Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. These approaches differ primarily in how they leverage information from the target model and how they generate candidate tokens. For instance, native MTP uses a model-native auxiliary path, while DFlash and DSpark utilize dedicated, target-conditioned networks to predict entire blocks of future tokens in parallel. EAGLE-3, by contrast, relies on an autoregressive mechanism that incorporates hidden states from various stages of the target Transformer.

Experimental results on AMD Instinct™ MI300X and MI355X GPUs demonstrate that the impact of speculative decoding is highly dependent on the specific model family, draft checkpoint, and target workload. While some configurations showed throughput gains exceeding 2x, performance improvements were not uniform across all datasets and models. The number of speculative tokens, or the proposal length, proved to be a critical tuning variable. Throughput generally improved as more tokens were drafted, up to a point, after which it hit a plateau or declined because the overhead of additional drafting work began to outweigh the benefits of successful verification.

Practical implementation of speculative decoding requires careful observability and tuning. The article suggests that users monitor signals such as throughput, mean accepted length, and per-position acceptance rates to identify the optimal configuration. Because different workloads exhibit different token predictability, a one-size-fits-all setting is rarely sufficient. A recommended workflow involves starting from a known configuration and performing a sweep of different proposal lengths to find the balance that best suits the specific task at hand.

Finally, training a custom speculator for a new target model is a viable path for further performance gains. This process involves collecting representative hidden states from the target model to train the draft component, ensuring that the speculator is well-aligned with the target's internal representations. By matching the draft model's architecture and training data to the target's expected domain, such as mathematics or code generation, users can achieve better acceptance rates and, consequently, higher serving throughput in their production environments.